<html><body><h2 style="text-align:justify;">1. Introduction</h2>Elasticsearch is a distributed search engine that is built on top of the Apache Lucene library. It scales horizontally as more data is added, making it a highly flexible and scalable platform. It is an open-source technology that is designed to handle large volumes of data, making it an ideal choice for companies that need to analyze vast amounts of information.Kibana is an open-source data visualization and exploration tool used to analyze and visualize data stored in Elasticsearch. It provides a user-friendly interface to create real-time dashboards, charts, and graphs to visualize and analyze complex data. Kibana is often used in conjunction with Elasticsearch and Logstash to form the ELK stack (Elasticsearch, Logstash, Kibana).Key features of Elasticsearch:- Handle structured and unstructured data.- Its powerful search capabilities: perform complex search queries on large volumes of data in near-real-time.- Log analysis: ingest and index log data from servers, applications, and other systems in near-real-time.<h2 style="text-align:justify;">2. Key concepts</h2><img src="https://api.careers.saigontechnology.com/storage/ck-editor/image-omVs5S7RH3Ado6tJ91Kq90V2H2R0XI4DUEVZTt4G.png">There are 8 key concepts of Elasticsearch that we need to understand<ul><li style="text-align:justify;">Documents: stores data as documents, which can be thought of as rows in a database table. </li><li style="text-align:justify;">Indices: A collection of documents that have similar characteristics, such as data type or purpose. </li><li style="text-align:justify;">Shards: Elasticsearch breaks indices into smaller pieces called shards, which can be distributed across multiple nodes in a cluster. </li><li style="text-align:justify;">Replicas: Elasticsearch creates replicas of shards to provide redundancy and fault tolerance. </li><li style="text-align:justify;">Nodes: Elasticsearch operates as a distributed system, with each node in the cluster performing a specific function. </li><li style="text-align:justify;">Mapping: Elasticsearch uses mapping to define how data is indexed and searched.</li><li style="text-align:justify;">Queries: Elasticsearch provides a powerful querying system that allows for complex searches. </li><li style="text-align:justify;">Aggregations: Aggregations allow for data analysis and reporting, allowing users to extract insights from their data.</li></ul>In this blog, I want to introduce the usage of Elasticsearch with Python using the library Python Elasticsearch Client. Beside that, I use Kibana as a monitoring tool for Elasticsearch clusters.<h2 style="text-align:justify;">3. Installation</h2>This topic uses Python version 3.- Install Elasticsearch through this <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html">link</a>. Start Elasticsearch on port 9200 by default- Install Kibana through this <a href="https://www.elastic.co/guide/en/kibana/current/install.html">link</a>. Start Kibana on port 5600 by default<img src="https://api.careers.saigontechnology.com/storage/ck-editor/image-Nq3U7H79328IaIPpW5O76pTINqmICDOvTUljMprk.png">- Install Python Elasticsearch Client by following this <a href="https://elasticsearch-py.readthedocs.io/en/v8.7.0/">link</a><h2 style="text-align:justify;">4. Implementation</h2>- Once we have installed the Python Elasticsearch Client, the first thing we need to do is connect to Elasticsearch. We can do this using the Elasticsearch() class.<pre><code class="language-plaintext">import json
from urllib.request import Request, urlopen
from elasticsearch import Elasticsearch, exceptions
# connect to Elasticsearch
es = Elasticsearch([{"host": "localhost", "port": 9200}])</code></pre>- Create an index to store the documents in data structures mapping with the data type of fields. You can define a mapping type, which determines how the document will be indexed and fields in your data. Each index has one mapping type.For example:<pre><code class="language-plaintext">mappings = {
 "properties": {
    "abilities": {
      "properties": {
        "ability": {
          "properties": {
            "name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256,
                }
              },
            },
            "url": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256,
                }
              },
            },
          }
        },
        "is_hidden": {"type": "boolean"},
        "slot": {"type": "long"},
      }
    },
    "base_experience": {"type": "long"},
  }}</code></pre>This mapping based on data structure that we’re using. Example mapping above for data at https://pokeapi.co/api/v2/pokemon/1/.Then create index:<pre><code class="language-plaintext">es.indices.create(index="pokemon_index", mappings=mappings)</code></pre>If no mapping type is defined, Elasticsearch will automatically detect mapping type and add fields, which is called dynamic mapping.<pre><code class="language-plaintext">es.indices.create(index="pokemon_index")</code></pre>- Load data and start indexing. After that, we can perform action search with custom query corresponding to query types.<pre><code class="language-plaintext"># pokemon json data
url = "https://pokeapi.co/api/v2/pokemon/1/"
req = Request(url=url, headers={"User-Agent": "Mozilla/5.0"})
# Decode the bytes to string, assuming the remote file is encoded in UTF-8
decoded_data = urlopen(req).read().decode("utf-8")
# Parse the JSON data into object
data = json.loads(decoded_data)
# indexing data
es.index(index="pokemon_index", id=1, document=data)</code></pre>- Access Kibana on port 5600 with url: `/app/management/data/index_management/` to check indices list and mapping on each indices. This is mapping for data <a href="https://pokeapi.co/api/v2/pokemon/1/">https://pokeapi.co/api/v2/pokemon/1/</a> that used above.<img src="https://api.careers.saigontechnology.com/storage/ck-editor/image-laPWMreDQWYruTgRcBNnf6ApStNJH22aenv4tdyW.png">- Now is when we perform action search using many types of query. You can find Query DSL (Domain Specific Language) <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html">here</a>.<ul><li style="text-align:justify;">Leaf query clauses: look for a particular value in a particular field. For example: match query, term query, range query</li><li style="text-align:justify;">Compound query clauses: combine multiple queries in a logical fashion (bool, dis_max)</li><li style="text-align:justify;">Allow expensive queries: </li></ul><h3 style="text-align:justify;">Using the Elasticsearch Python library:</h3><ul><li style="text-align:justify;">Let’s try with the simple search through the “match” query which matches documents containing the searched term in any field.</li></ul><pre><code class="language-plaintext">query = {
  {
    "_source": ["name", "height", "weight"],
    "size": 1,
    “query": {"match": {"name": "bulbasaur"}},
  }
}
result = es.search(index="pokemon_index", body=query)
# result
# {
# "took": 5,
# "timed_out": False,
# "_shards": {"total": 1, "successful": 1, "skipped": 0, "failed": 0},
# "hits": {
# "total": {"value": 1, "relation": "eq"},
# "max_score": 0.2876821,
# "hits": [
# {
# "_index": "pokemon_index",
# "_type": "_doc",
# "_id": "1",
# "_score": 0.2876821,
# "_source": {"name": "bulbasaur", "weight": 69, "height": 7},
# }
# ],
# },
# }</code></pre><ul><li style="text-align:justify;">_source: specify fields should be included in the search results</li><li style="text-align:justify;">size: limit number of results returned</li></ul><h3 style="text-align:justify;">Use Kibana DevTools:</h3><ul><li style="text-align:justify;">Navigate to the DevTools tab</li><li style="text-align:justify;">Enter search query in the console and perform search action</li></ul><h2 style="text-align:justify;">5. Conclusion</h2>Thanks for reading and scrolling here. This blog is just an introduction about the simple usage of Elasticsearch. There are many deeper things related to Elasticsearch if you want to harness more power of it. Based on it, you can improve your search functionality and provide your users with better search results. This is a valuable asset for any organization looking to improve its search capabilities. Whether you're working with a small data set or a massive one, Elasticsearch can help you harness the power of your data and get the most out of your search efforts.If you have any questions, feel free to let me know via email: <a href="mailto:quang.ngdb@saigontechnology.com">quang.ngdb@saigontechnology.com</a>Have a good day! References<ul><li style="text-align:justify;"><a href="https://www.elastic.co/guide/index.html">https://www.elastic.co/guide/index.htm</a>l</li><li style="text-align:justify;">https://elasticsearch-py.readthedocs.io/en/v8.7.0/</li><li style="text-align:justify;">https://www.elastic.co/guide/en/kibana/current/install.html</li></ul>Source:<a href="https://saigontechnology.com/blog/harnessing-the-power-of-elasticsearch-boosting-your-search-capabilities"> https://saigontechnology.com/blog/harnessing-the-power-of-elasticsearch-boosting-your-search-capabilities</a>  </body></html>

In today's data-driven world, companies have access to vast amounts of data, but the ability to analyze and make sense of that data can be a challenge. This is where Elasticsearch comes in. Elasticsearch is a powerful search and analytics engine that is designed to handle large volumes of unstructured data, making it an ideal tool for companies that need to analyze vast amounts of information.

Harnessing the Power of Elasticsearch: boosting your search capabilities