In the previous blog in this ELK stack tutorial series, we reviewed parsing our logs with Logstash and indexing them to Elasticsearch.
With this post, we move on and focus on visualizing the parsed log details using Kibana 4.3, as well as finding useful patterns and information residing in them. We will begin by creating a few basic visualizations.
Kibana 4.3.0 Setup and Installation
Kibana setup and installation are simple. Just download the appropriate zip file from here and extract it to the location you wish. Then browse into the extracted folder and open the folder named “bin” where you can see a script file named “kibana” (for Windows users it would be “kibana.bat”). Double click it or run via the terminal, and you are done.
Another option would be to use the hosted Kibana solution provided by Qbox.io. This avoids the hassles of installation and provides a much more convenient platform. Kibana is configured to run by default on the port number 5601. Take your browser and type in “localhost:5601” to start Kibana.
Note that this is the latest version of Kibana, and it requires Elasticsearch 2.1.0 or above to function. If you haven’t updated to Elasticsearch 2.1.0, please do so.
The typical welcome screen for Kibana is shown below:
There are four primary tabs or components in Kibana: “Settings,” “Discover,” “Visualize,” and “Dashboard,” and we will cover each one in detail in the following sections.
As you can see, we are launched to the “settings” tab of Kibana. Here we are asked to provide the index name (in this case, “logstash-test-01”) and the time field name in our index (which is “@timestamp”). After you provide these details, press the create button, and you will be redirected to a page where you can see the basic fields and and their mapping information.
We now move on to the “Discover” tab. After you press the “Discover” tab, you will be directed to the following page:
You can see in the figure that there are “No results found” for our index. This occurs because the default time range taken by Kibana is only 15 minutes. Change this by clicking on the blue box toward the upper right hand corner, marked as 1 in the figure, where you can select the time range of your choice. In this case I have selected previous month because the logs we have indexed in the previous blog are from the past month. It will vary, depending on the time ranges in the logs you have selected. After changing the “timefilter” value to previous month, the resulting page shows a bar graph with time on the x axis and document count on y axis.
Here we can see the field names on the left navigation bar and individual documents on the main page below the graphs. Also, in the search bar provided, we can make searches on terms in our documents.
Now comes the visualization panel, which is the heart of Kibana. Click the “Visualize” tab to be directed to the following page:
The visualization section is divided into two sections:
- Create new visualization: This section provides us with a wide range of visualization techniques ranging from simple area charts to location-based tile maps. We use these techniques to create visualizations on our data and make them easily understandable. Kibana also allows us to save these visualizations for future use. Unlike the previous versions of Kibana, this helps fit together visualizations from different indices to create a single dashboard. We will see the making of visualizations in the coming sections.
- Open a saved visualization: Here we can open a saved visualization to edit the properties.
The final component is the Dashboard panel. The dashboard panel allows us to place the saved visualizations in the order we prefer. We can also customize their lengths and heights for perfect fit.
Clicking on the Dashboard panel would open up a window like the one given in the screenshot.
Here, as marked in the blue boxes, we have various options to load/save dashboards, create new dashboard, load saved visualization, and share the dashboards we have created.
Now that we are familiar with the components of Kibana, we can move on to create a basic visualization, look up how to save it, and load it into dashboard. We will create a simple pie chart for practice.
Metrics is the simplest of all visualizations available in Kibana because it provides us with a single number. It can be the number of total hits in the index or the sum/average of a specified numerical field.
First, click on the visualization tab. A window with various visualization options will pop up. From there, click on the “Metrics” tab, which will lead you to the visualization settings window. The typical Kibana visualization window has two sections:
- The visualization settings panel appears on the left.
- The visualization rendering panel appears on the right.
In this example, we will go with the display of total hits. In the visualization settings panel, give the following settings as shown in the screenshot below, and press the green play button. Just above the panel and on the right section, there appears a number that represents the total hits in the index.
In the above picture you can see a few numbered red boxes. Let us explore what they are.
Because we set the default metrics to be that of the hits count by setting up the “aggregation” as “count,” we get the total number of hits as a big font visualization. If we want any other numerical field to be analyzed, we click on the add metrics tab, select the appropriate aggregation and the field, and then press the play button (indicated by the red box).
- The result of the new metrics will appear adjacent to the hits count in the visualization window.
- Save the visualization by using the save icon numbered 3 in the red box, which will produce a sliding input bar with the name “title” where we will name the visualization. We named this example visualization “metrics visualization” and clicked the “save” button below it.
- Up arrow button below the visualization panel. This button is indicated as red box numbered 4 in the above screenshot. After we press this button, the visualization rolls up, and we are shown 4 tabs named table, request, response, and statistics as shown in the screenshot below. The table tab shows the visualization rendered in a table format. The request tab gives us the query fired to get the results for this visualization. The response tab gives the response for the request in the json format. The statistics tab shows the statistics such as the index name on which the operation was performed, the query, and the response time with the total number of hits.
- Options. There is another tab indicated by the red box number 5, and it is an important setting. Clicking on this tab gives us many more options regarding the style of visualizations rendered. The other tab, “data,” deals with the manipulation of data, and the “options” tab deals with the customization to the visualizations. In the case of metrics visualization, we have other options, including changing the font color.
The next visualization example we are dealing with is “data tables,” which enables us to create tables with only fields of interest and their corresponding values. Let us look on how to implement the same. As in the above example, click the top visualization panel, and then select the “data table” option from the resulting tabs.
Now apply the following settings in the left panel, and press the green play button to get the visualization.
As you can see from the above image, I chose the metric to be “count” and the buckets to be “split rows,” where the “date histogram” was given against the time field. The resulting table shows the time values in one column and the corresponding hits count in the second column.
We can add sub buckets to this by clicking the “add sub buckets” option shown in the red box numbered 1. If we give another split rows bucket type and a term aggregation, the new field will appear as one of the columns in the table.
The tables generated display by default only 10 rows per page, but we can change this in the “options” settings to the value of our choice.
We then then saved it as a visualization under the name “data table visualization.”
In this blog we discussed the setup and installation of Kibana 4.3.0. We also became familiar with the components of Kibana, and we then created and saved a few sample visualizations (metrics and data tables).
In the next article of this series we will create some advanced visualizations such as multi-layered pie charts, bar charts, area charts, tile maps, and line and bubble graphs.