Logstash is a layer in the Elasticsearch ELK stack that allows you to parse and process logs from many sources in your infrastructure and store them in a centralized location. It began as an independently developed, open source project, but now exists as a part of the official Elasticsearch family of products, bolstering development and support and retaining its Apache license. For a brief introduction to Logstash, take a look at our “Welcome to the ELK Stack” post and the official getting started guide. If you’re looking for a more in-depth overview, check out the Logstash book.

Running Logstash involves configuring plugins that control how your instance of Logstash receives, parses and forwards logs. The default installation contains many useful plugins that provide the ability to interact with a wide array of services. In this example we will be using the Elasticsearch output plugin. For more information, the docs contain an overview of all provided plugins.

As a part of the ELK stack, Logstash is designed to output logs to Elasticsearch for viewing and searching. A common way to set this up is to run a full stack on a single server, configuring Logstash to output to the local instance of Elasticsearch. However, this quickly becomes limiting and complicated when moving to production at scale. This is where Qbox comes in, providing quick deployments and long-term maintenance of multi-node Elasticsearch clusters. Once we’ve launched an instance with Qbox, all that’s needed is to install and configure Logstash in our environment to output to the Elasticsearch endpoint(s) provided.

To get started, boot up and connect to your Logstash box. In this example we’ll be using the hashicorp/precise64 Vagrant box as a reference. This sets up 64-bit Ubuntu 12.04 LTS box for us that will need a few packages installed. Let’s assume that we’ve already downloaded and booted our box.

First we need to install Logstash’s only requirement, a Java runtime. Logstash is compatible with open source versions of Java and we’ll be installing the default provided by Ubuntu: OpenJDK.

vagrant@precise64:~$ sudo apt-get install default-jre
vagrant@precise64:~$ java -version
java version "1.6.0_33"
OpenJDK Runtime Environment (IcedTea6 1.13.5) (6b33-1.13.5-1ubuntu0.12.04)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)

For the sake of simplicity, I'll install Logstash by using the example provided in the getting started docs. If you’re interested in adding the official Elasticsearch repositories to apt, check out this example on DigitalOcean.

vagrant@precise64:~$ curl -O https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz
vagrant@precise64:~$ tar zxvf logstash-1.4.2.tar.gz

For testing purposes, write a simple configuration file:

#qbox.conf
input {
  stdin { }
}
output {
  stdout {
    codec => rubydebug
  }
}

We've added a rubydebug codec to our output; this will display Logstash's output with Ruby's awesome_print gem. To ensure everything is correctly installed, let’s run Logstash with the simple conf file and send a message to stdin.

vagrant@precise64:~$ logstash-1.4.2/bin/logstash agent -f qbox.conf
hello world
{
  "message" => "hello world",
  "@timestamp" => "2014-10-28T16:25:43.483Z",
  "@version" => "1",
  "host" => "precise64"
}

The -f flag is used to specify a location for our configuration file. If you install Logstash with apt, you’ll write to this file in /etc/logstash/conf.d and have access to logs in /var/log/logstash.

In order for our Logstash instance to accept incoming data from Heroku, we’ll need to install Heroku’s toolbelt and Logstash’s contrib plugins. Let’s install the toolbelt, login and check our apps.

vagrant@precise64:~$ wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh
vagrant@precise64:~$ heroku login
vagrant@precise64:~$ heroku apps
=== My Apps
your-app-0001
your-app-0002

The contrib plugins are a separate repository of Logstash plugins that exist outside of the core of common plugins. These plugins provide Logstash with the functionality to interact with many different services, including Heroku. Logstash contains an automated script to easily install them:

vagrant@precise64:~$ logstash-1.4.2/bin/plugin install contrib

Warning: If you installed Logstash with apt, the automated plugin install in the current version of Logstash, 1.4.2, is unfortunately broken. However, the users participating in the GitHub issue have provided a simple workaround.

Let’s get back to our configuration file. We’ll need an input for Heroku, an output for our Qbox cluster and a filter. I’m using the pattern recommended by the Logstash team for the filter.

input {
  heroku { 
    app => "your-app-0001" 
  }
}
filter {
  grok {
    pattern => "^%{TIMESTAMP_ISO8601:timestamp} %{WORD:component}\[%{WORD:process}(?:\.%{INT:instance:int})?\]: %{DATA:message}$"
  }
  date { timestamp => ISO8601 }
}
output {
  stdout { }
  elasticsearch {
    host => "your-cluster-subdomain.qbox.io"
    port => "80"
    protocol => "http"
  }
}

Logstash is good to go! When you start it up, Logstash it should output some warnings about plugins and your parsed Heroku logs to stdout. To make sure your data is reaching your Qbox cluster, check Kibana for events.

comments powered by Disqus