How to Visualize and Analyze Topbeat Data using Kibana 4
Posted by Vineeth Mohan May 19, 2016In the previous post of our ‘Beats’ series, we configured and setup Topbeat. We also explained how to index system data to Elasticsearch via Logstash. In this final post, we will see how the indexed data can be utilized to create useful visualizations as well as a helpful dashboard.
Setup
This is a continuation from our previous post on topbeat, and we use the same data as in that post. Please click the previous link if you are checking these posts for the first time.
Running Kibana
Since we have already indexed the data to Elasticsearch, we are ready to start Kibana and select the topbeat index (in our case logstash-topbeats-test-01
).
Visualizations using Kibana
Let us start by creating individual visualizations. Remember that after each visualization, we need to save it for creating the dashboard later. Since we are looking into the system processes in the data we have, we will render the types of processes, the process names, the statistics of free memory available, and the total process count against time.
Learn How to Install Ghost Node JS Blog with Docker on Supergiant, Our New Open Source Software.
Pie Chart Visualizations
Types of Activities
First and foremost, analysis on the server level will be the type of processes running. For this, use the pie chart visualization. Select the pie-chart
option from the visualizations
tab and apply the following settings:
In the above picture you can see that we have selected the type.raw
field for the analysis process. We have also selected the size of the aggregation as 5. We can infer from the resulting visualization that we are clear on the share of types of processes in the system.
Process Name
Create another visualization for the individual process names in the system that are running under processes. Create a similar pie chart as we have done above and then select the proc.name.raw
in the Field
section of Kibana.
Process Name and CPU Start Time
Suppose we want to know the start time against each process. For this visualization, opt for a stacked pie chart. Set the Field
to proc.name.raw
. In the sub-aggregations sections, select the Field
to proc.cpu.start_time.raw
, as shown in the figure below:
Histogram
Free Memory
It is good practice to track the free memory quanta in our system. In order to do so, use a histogram graph from the visualization tab and perform the following set up:
As you can see from the above diagram, I have set the interval value to 10 MB. Otherwise, the x-axis values would be too close and cluttered to read. The field in which the data is picked, in this case, is mem.free
.
CPU Idle
Now let us take the CPU idle time for analysis. This parameter is represented in milliseconds by default. We can use the histogram graph as we did in the previous free memory visualization. Set up the histogram graph and set the field to cpu.idle
and check the results. In order to avoid clutter in the graphs, use a 100000 millisecond interval.
Date Histogram
Total Activity Count
How about a time-based monitoring of the total activities of the system? This allows a good indication of system up and down times. In order to do that, select a date histogram
visualization from the visualization tab and apply the following settings:
Dashboard Creation
A consolidated dashboard can be prepared by using the individual visualization components we have created so far. As shown in our packetbeats visualization post, this can be done by clicking on the dashboard
section in the header and on the Add visualization
button. There you can see the individual visualizations in a dropdown, and select one-by-one to create a basic dashboard.
Resize the individual graphs to convenient sizes and finalize the dashboard by saving it. You do this by using the save
button in the top bar. I have resized and arranged the visualizations in the dashboard to look like this:
Notes
I have mentioned this in the packetbeats post, but in case you have missed it, note the following points:
Mappings
By default, the mapping for the topbeat index will be defined in topbeat.template.json
which is located in the /etc/topbeat/topbeat.template.json
file. In the example we have created, we named the index with the prefix logstash
. Therefore, it will apply logstash’s default mapping to the index. For a custom mapping, remove the logstash prefix and make the appropriate changes in the topbeat.template.json
file.
Naming the Indices
To get chronological naming, such as topbeat-YYY-MM-DD
, the output
settings in the logstash.conf
file have the following value for the index
field:
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
Summary
In this final post of our Filebeat series, we have seen the visualization of data from Topbeat using the ELK stack. We hope that you have enjoyed this series. Drop us a comment below!