In this blog series, we introduce the Beats platform for Elasticsearch.

Beats platform is a complete package. You can collect various operational data and index to Elasticsearch via logstash and visualize using Kibana.

In this post we will concentrate more on packetbeats component of the beats platform.

Beats Platform

As said above, the beats platform offers us an all in one solution for data shipping to Elasticsearch. Beats platform, when used with the ELK stack, would provide us the enriched visualization of the analytics for the operational data indexed.

Here are the main components of the Beats platform.

  1. Packetbeat: Packetbeat is a distributed packet monitoring system which can monitor in real-time the network traffic for application level protocols like HTTP, MySQL etc. Packetbeat agents sniffs the network traffic generated and then parse it based on the protocol and maps the messages to transactions and for each such action a record is generated and indexed to Elasticsearch.
  2. Topbeat: While packetbeat is used for network monitoring, top beat is the beats service providing system-wide and per-process statistics along with a disk usage overview. It is widely used with ELK stack for monitoring server statistics.
  3. Filebeat: Filebeat is used to pass the logs output from packetbeat of topbeat to the input of logstash to parse the logs the way we want.

Packetbeat — Suggested Architecture

Installation of packetbeats can be done in either of the two ways:

  1. Within the application server
  2. On dedicated servers

The first approach can be adopted in the case where there is no much load in the application server, but as the load increases in the application server, it might become difficult in managing both.

The second approach would be the ideal solution because it doesn’t interfere with the processes in application servers and would be easy to manage and configure.

Here the main limitation coming in to play might be the cost involved in setting up separate servers for packetbeats.The suggested architecture is as shown in the block diagram given below:

 

packetbeat1.png#asset:966

As you can see from the above figure, there are two separated networks which have packetbeats installed. The output from each of the packetbeats instance is forwarded to a logstash event. This enables us to enrich the data by parsing it using logstash and convert it to the formats we require. The next step involves the passing of the logstash output to Elasticsearch to index it. Now the last and final step is to visualize the indexed data in Elasticsearch using Kibana.

Learn about our new open source product Supergiant: The first datacenter total control system that makes it easy to save up to 60% on your AWS bill.

Setup and Install

We can install packetbeats in linux by typing in the following commands in the terminal:

sudo apt-get install libpcap0.8
curl -L -O https://download.elastic.co/beats/packetbeat/packetbeat_1.1.2_amd64.deb
sudo dpkg -i packetbeat_1.1.2_amd64.deb

For platforms other than Linux, you can refer to the documentation here. As the next step we need to install filebeats, which is to act as an interface between packetbeats and Elasticsearch. It can be installed as below:

curl -L -O https://download.elastic.co/beats/filebeat/filebeat_1.1.2_amd64.deb
sudo dpkg -i filebeat_1.1.2_amd64.deb

5. Configuration

Packetbeats config
After finishing the packetbeats installation, browse to the folder where the packetbeats is installed, which is /etc/packetbeat.

Open the file packetbeat.yml in an editor with administrator permission and then make the following changes in the output section.

First of all, since we are going to ship our data to logstash first, we need to disable the data forwarding to Elasticsearch first in the packetbeat setting. To do that, under the outputsection, the default output would be specified as elasticsearch and the port as localhost:9200.

Comment out those options like below:Packetbeat2
In the above figure you can see I have commented out, the elasticsearch and the hosts by adding # in front of it.

Now scrolling down the file after this, uncomment the logstash and the hosts section. Now change the port for the hosts to localhost:5044, which in general is the port given to logstash, like in the picture below:

logstash as output

Filebeat config
Change the output section in the filebeat configuration yml file ( located in /etc/filebeat/filebeat.yml) from elasticsearch to logstash as shown below:
filebeat config

Logstash config
Let us see what the setup changes in the logstash config file (you can see the setup and installation of logstash in this qbox blog) are. Since the input is from beats, we need to specify that. The logstash.conf file would like below:

logstash config

Indexing the Data

Now that we have configured the setup, we will see how to start indexing the data. For that we need to start all three services. Here are the commands to start all of them:

  1. File beat sudo /etc/init.d/filebeat start
  2. PacketBeat sudo /etc/init.d/packetbeat start
  3. Logstash sudo bin/logstash -f logstash.conf

If you take the head plugin installed in your browser and check for the indexlogstash-packetbeat-test-01 in Elasticsearch head, you can see there would be numerous documents indexed in that index.

Sample Output

Let us have a look at a sample document indexed via packetbeat:

{
  "_index": "logstash-packetbeat-test-01",
  "_type": "http",
  "_id": "AVOzGRZ8O2uvv56Gwoe6",
  "_score": 1,
  "_source": {
    "@timestamp": "2016-03-26T13:24:38.109Z",
    "beat": {
      "hostname": "ubuntu",
      "name": "ubuntu"
    },
    "bytes_in": 228,
    "bytes_out": 136,
    "client_ip": "192.168.1.54",
    "client_port": 33911,
    "client_proc": "",
    "client_server": "",
    "count": 1,
    "direction": "out",
    "http": {
      "code": 304,
      "content_length": 0,
      "phrase": "Modified"
    },
    "ip": "91.189.95.36",
    "method": "GET",
    "params": "",
    "path": "/meta-release-lts",
    "port": 80,
    "proc": "",
    "query": "GET /meta-release-lts",
    "responsetime": 180,
    "server": "",
    "status": "OK",
    "type": "http"
  }
}

Conclusion

In this post we have seen the introduction to the beats platform of Elasticsearch, how to set up packetbeat, and how to connect it with logstash to index data to Elasticsearch.

In the next post we will see how to visualize the packetbeat data using Kibana, as well as the necessary changes in set up for achieving the same.