How to Capture System Statistics using Elasticsearch Topbeat
Posted by Vineeth Mohan May 18, 2016In the previous posts of filebeat series, we have seen the network monitoring capabilities using packetbeat. We have also used Kibana to visualize the data provided by packetbeats.
In this article we show how to monitor the system performance using another filebeat component, topbeat.
Topbeat
The topbeat component in filebeats will help us to monitor various parameter related to system monitoring such as CPU usage, memory consumption, and other per-process and system-wide data. Topbeat collects the system data using the top command. It then formats the data and sends it to elasticsearch for indexing.
Setup and Installation
The installation for top beats in ubuntu can be done by typing in the following commands in the terminal:
curl -L -O https://download.elastic.co/beats/topbeat/topbeat_1.1.2_amd64.deb sudo dpkg -i topbeat_1.1.2_amd64.deb
For more information on how to install topbeat in platforms other than ubuntu, you can refer here. In case you haven’t installed filebeats, you can refer to the earlier post on filebeats to do so.
Suggested Architecture
The suggested architecture for topbeat installation is the same as that of packetbeat. In a similar manner, it would be ideal if it is installed in a dedicated server.
Each topbeat instance collects the top beat information from the respective machine and sends that information to logstash. We will parse and enrich the data there. This data is sent to Elasticsearch, and from there we use Kibana for visualization.
The suggested architectural model is shown in the figure below for topbeat:
Configuration
A. Topbeat Configuration
The configuration file for topbeat is in the /etc/topbeat
folder, called topbeat.yml
.
The input section of this file, for this tutorial, is configured below:
As you can see from the above picture, the period
attribute defines the interval in which the data should be collected from the system. We can set value for this field depending on the monitoring we need. Here we have given it 10, which is in seconds.
The procs
in the field select the processes to be monitored. By default, it would take all the processes for monitoring.
The output section of the tobbeat.yml
file is to be configured just like that of the packebeat. Comment out the elasticsearch
and host
fields. Uncomment the logstash
field. Finally, enable the host
as localhost:5044
.
B. Filebeat Configuration
The filebeat configuration file, located in /etc/filebeat/filebeat.yml
, is similar to the above process. Change the output from elasticsearch to logstash, as well as the port number.
C. Logstash.conf
The configuration file for logstash looks like:
Indexing the Data
Follow the steps below to index the data:
- start filebeats
sudo /etc/init.d/filebeat start
- start topbeat
sudo /etc/init.d/topbeat start
- start logstash
sudo bin/logstash -f logstash.conf
After these process, topbeat will start to acquire and index the data. We can check this via the elasticsearch head plugin or by simply posting a curl request like below:
curl -XPOST 'localhost:9200/logstash-topbeats-test-01/_search?pretty=true' -d '{}'
Sample Data
Here is an example of what our indexed documents will look like:
{ "_index": "logstash-topbeats-test-01", "_type": "system", "_id": "AVPGl2M0aKy38zc9Iytc", "_score": 1, "_source": { "@timestamp": "2016-03-30T08:15:06.010Z", "beat": { "hostname": "ubuntu", "name": "ubuntu" }, "count": 1, "cpu": { "user": 680888, "user_p": 0.48, "nice": 2260, "system": 163954, "system_p": 0.05, "idle": 7294511, "iowait": 652844, "irq": 11, "softirq": 3674, "steal": 0 }, "load": { "load1": 3.25, "load5": 4.03, "load15": 2.95 }, "mem": { "total": 3234353152, "used": 3021340672, "free": 213012480, "used_p": 0.93, "actual_used": 2703441920, "actual_free": 530911232, "actual_used_p": 0.84 }, "swap": { "total": 3199995904, "used": 1505382400, "free": 1694613504, "used_p": 0.47 }, "type": "system", "@version": "1" } }
Conclusion
In this post we have seen in detail how to collect system information using the topbeat tool and to index to Elasticsearch via Logstash.
In the final installment to this series we will visualize the data with Kibana.