How to Visualize and Analyze Topbeat Data using Kibana 4

In the previous post of our ‘Beats’ series, we configured and setup Topbeat. We also explained how to index system data to Elasticsearch via Logstash. In this final post, we will see how the indexed data can be utilized to create useful visualizations as well as a helpful dashboard.

Setup

This is a continuation from our previous post on topbeat, and we use the same data as in that post. Please click the previous link if you are checking these posts for the first time.

Running Kibana

Since we have already indexed the data to Elasticsearch, we are ready to start Kibana and select the topbeat index (in our case logstash-topbeats-test-01).

Visualizations using Kibana

Let us start by creating individual visualizations. Remember that after each visualization, we need to save it for creating the dashboard later. Since we are looking into the system processes in the data we have, we will render the types of processes, the process names, the statistics of free memory available, and the total process count against time.

Learn How to Install Ghost Node JS Blog with Docker on Supergiant, Our New Open Source Software.

Pie Chart Visualizations

Types of Activities

First and foremost, analysis on the server level will be the type of processes running. For this, use the pie chart visualization. Select the pie-chart option from the visualizations tab and apply the following settings:

In the above picture you can see that we have selected the type.raw field for the analysis process. We have also selected the size of the aggregation as 5. We can infer from the resulting visualization that we are clear on the share of types of processes in the system.

Process Name

Create another visualization for the individual process names in the system that are running under processes. Create a similar pie chart as we have done above and then select the proc.name.raw in the Field section of Kibana.

Process Name and CPU Start Time

Suppose we want to know the start time against each process. For this visualization, opt for a stacked pie chart. Set the Field to proc.name.raw . In the sub-aggregations sections, select the Field to proc.cpu.start_time.raw, as shown in the figure below:

Histogram

Free Memory

It is good practice to track the free memory quanta in our system. In order to do so, use a histogram graph from the visualization tab and perform the following set up:

As you can see from the above diagram, I have set the interval value to 10 MB. Otherwise, the x-axis values would be too close and cluttered to read. The field in which the data is picked, in this case, is mem.free.

CPU Idle

Now let us take the CPU idle time for analysis. This parameter is represented in milliseconds by default. We can use the histogram graph as we did in the previous free memory visualization. Set up the histogram graph and set the field to cpu.idle and check the results. In order to avoid clutter in the graphs, use a 100000 millisecond interval.

Date Histogram

Total Activity Count

How about a time-based monitoring of the total activities of the system? This allows a good indication of system up and down times. In order to do that, select a date histogramvisualization from the visualization tab and apply the following settings:

Dashboard Creation

A consolidated dashboard can be prepared by using the individual visualization components we have created so far. As shown in our packetbeats visualization post, this can be done by clicking on the dashboard section in the header and on the Add visualization button. There you can see the individual visualizations in a dropdown, and select one-by-one to create a basic dashboard.

Resize the individual graphs to convenient sizes and finalize the dashboard by saving it. You do this by using the save button in the top bar. I have resized and arranged the visualizations in the dashboard to look like this:

Notes

I have mentioned this in the packetbeats post, but in case you have missed it, note the following points:

Mappings

By default, the mapping for the topbeat index will be defined in topbeat.template.json which is located in the /etc/topbeat/topbeat.template.json file. In the example we have created, we named the index with the prefix logstash. Therefore, it will apply logstash’s default mapping to the index. For a custom mapping, remove the logstash prefix and make the appropriate changes in the topbeat.template.json file.

Naming the Indices

To get chronological naming, such as topbeat-YYY-MM-DD, the output settings in the logstash.conf file have the following value for the index field:

index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

Summary

In this final post of our Filebeat series, we have seen the visualization of data from Topbeat using the ELK stack. We hope that you have enjoyed this series. Drop us a comment below!