TL;DR Using parts of Elasticsearch data in custom visualizations is possible via Elasticsearch’s REST-api. To search and get results one needs to use Lucene’s Query Syntax. Using advanced functions like field filters, time ranges, size parameters and sorting is possible via operators in the query string. This empowers users to retrieve the data needed and makes it possible to load them into applications and visualizing then using custom chart frameworks.

The Goal

Elasticsearch data visualized in a custom application using a custom chart framework.

What is Elasticsearch?

Basically, Elasticsearch is an open-source search and analysis engine built on Apache Lucene. It stores data and search results in a NoSQL format and makes it accessible via a RESTful web-interface. Elasticsearch is part of the well-known ELK-Stack, where it is joined together with Logstash and Kibana. This trio enables users to run log analysis for systems on open-sourced software for free.

The ELK-Stack and its usage

Every single server produces log messages and protocols while running. Imagine an administrator has to manage many decentralized systems, each of those produce a remarkable amount of log and protocol data. The greater the amount of systems to monitor, the harder it gets to analyze the log statements in an efficient and effective way using the built-in tools. Hundreds of servers running different applications produce lots of data that needs to be analyzed, especially when problems occur or to get a deep dive in understanding what goes on. The ability to add metrics gives an even better overview in comparison to logs only. By combining the metrics and the logs, things are visualized in a good way. One can see information on what is happening now, and also about the past and how software pieces were behaving. The ELK Stack provides a good toolset to get over these problems by combining the three mentioned tools:
  • Elasticsearch A highly performant search engine and NoSQL database based on Apache Lucene which is a full-text search engine. Its functionality can be used via a REST-interface. Elasticsearch is the place where all the data is stored and from where it is fetched. It is responsible for providing all the search and analysis results.
  • Logstash A pipeline-tool that accepts input data from various sources. It transforms input data via several available filters and other plugins (e.g. field aggregations, validation, alignment). It is responsible for normalizing and structuring incoming data and sending it to Elasticsearch.
  • Kibana A frontend visualization tool that allows users to create different views/graphs on the data and combining these views in dashboards.

Using Elasticsearch data in custom visualizations

Although Kibana as frontend provides a variety of great ways to create views for data analysis, there can also be other use-cases where the Elasticsearch data should be visualized in custom applications. Imagine you want to visualize parts of the data for public users on your website, e.g. non-sensivite metrics like the amount of energy the in-house solar power system produces in real-time or over time. Or – in our case – visualizing data for internal purposes like our OMM Stayfit Challenge at work. Creating custom graphics apart from the Kibana dashboards/views could be required because of corporate identity, etc. Fortunately getting search results from Elasticsearch programmatically is possible via its REST-api!

Querying Elasticsearch

Elasticsearch provides a REST-api where users can easily query and retrieve search results from the database. To tell Elasticsearch which specific entries we want to receive, the Lucene Query Syntax is used. We can build Queries with
  • a specific time range
  • specified result set size
  • sorted results
  • filtered results
  • timeouts
  • …..
https://elasticsearch/_your_index?q=@timestamp:[now/w TO now]&_source_exclude=headers,@version,host&size=1000&sort=@timestamp:desc
  • @timestamp=[now/w TO now] Search only after entries created in the actual week
  • _source_exclude=headers, @version, host Exclude those parameters and do not return them in the result set objects.
  • size=1000 Maximum size of the result set – 1000 objects
  • sort=@timestamp:desc Sort the result set after the creation date in descending order. The most recent entry is the first in the result set.
To get a more detailed overview on all functions and how to use them, check the Elastic Query Docs and the Request URI docs

Tip – Use Kibana Devtools to test/build queries

To quickly build your queries, the built-in Kibana devtools application can be used. It allows to execute custom queries against Elasticsearch and immediately displays the results in prettified JSON format. Testing queries this way is faster than setting up a whole Http-Service and allows to use the established connection between Kibana and Elasticsearch. Therefore one does not need to deal with things like authentication, webserver configurations etc at first. Give it a go and get more out of your Elasticsearch data!   Related links:

Leave a Reply