Monitor Logstash 5.0 with the New Monitoring APIs

Posted by Adam Vanderbush March 9, 2017

A common use case that comes up when we use any product is how can we get metrics from it? How can we monitor it? Elasticsearch, since its early release, has always provided a way to monitor it using the _cat/stats API. However, for Logstash there wasn’t a way to gather metrics and monitor it until recently. With the release of Logstash 5.0+, Logstash has introduced a set of APIs to monitor Logstash. In this article we explore the monitoring APIs exposed by Logstash, which includes the Node Info API, the Plugins API, the Node Stats API, and the Hot Threads API.

Tutorial

For this post, we will be using hosted Elasticsearch on Qbox.io. You can sign up or launch your cluster here, or click “Get Started” in the header navigation. If you need help setting up, refer to “Provisioning a Qbox Elasticsearch Cluster.“

For monitoring, Logstash exposes a set of 4 API’s: node info, node stats, plugins and hot threads api. When the Logstash is started, the monitoring API’s bind to the port 9600 by default. We can override the port by specifying the flag –“http.port” or by specifying the value “http.port” in “logstash.yml” file found under config folder.

Let’s create a basic Logstash pipeline and run Logstash with monitoring api bound to 9601.

# custom.conf
input {
stdin{
}
}
output {
stdout{
codec => json
}
}
#starting logstash using the above configuration.
C:\qbox\Logstash -5.2.0\bin>Logstash  -f custom.conf --http.port 9601

Once the Logstash is started, we can verify the general information about Logstash instance by invoking the root resource (/) using either curl or using a web browser like below.

curl –XGET http://localhost:9601/

logstash-1.png#asset:1199

Node Info API

Information related to the underlying operating system, JVM (info like JVM version, memory initialized/allocated etc) and pipeline (info like batch size, workers, etc.) can be obtained using the nodes api. The rest api or the resource url for obtaining nodes info is “_node”. To fetch information specific to pipeline or OS or JVM, we can append “type(s)” to the nodes api:

curl –XGET<a href="http://localhost:9601/_node"> http://localhost:9601/_node</a>

logstash2_170307_144240.png#asset:1200

If you want to retrieve node information with operating system and JVM details use this command:

curl –XGET<a href="http://localhost:9601/_node/os,jvm"> http://localhost:9601/_node/os,jvm</a>

Plugins API

Logstash pipeline is made of input, filter, and output plugins. Logstash ships with many default plugins and provides the ability to install hosted plugins or build and install custom plugins. The richness of Logstash is due to the huge number of plugins it supports. If you want to know the list of plugins the current Logstash instance is running with use the resource “_node/plugins”.

curl –XGET<a href="http://localhost:9601/_node/plugins"> http://localhost:9601/_node/plugins</a>

logstash3_170307_144241.png#asset:1201

Node Stats API

Node API provides a basic information about the node. However, if you are interested in gathering a detailed runtime metrics about the node make use of the node stats api. The node stats api provides detailed information about:

JVM – including stats about threads, memory usage, garbage collectors, and uptime.
Process – including stats about file descriptors, memory consumption, and cpu usage.
Pipeline – runtime stats about the Logstash pipeline.
Reloads – stats about config reload successes and failures.
OS – runtime stats about cgroups when Logstash is running in a container.

The rest api or the resource url for obtaining nodes info is “_node/stats”. You can append the “type(s)”[comma separated] to the resource/rest url to get specific type runtime metrics.

curl –XGET http://localhost:9601/_node/stats
curl –XGET http://localhost:9601/_node/stats/pipeline,reload

logstash4.png#asset:1202

Note: By passing the flag “--config.reload.automatic” when starting Logstash, automatic configuration reloading can be enabled.

Hot Threads API

A hot thread is a Java thread that has high CPU usage and executes for a longer than normal period of time. Logstash provides hot threads api to retrieve the same. The rest api or the resource url for obtaining hot threads info is “_node/hot_threads”. By default it returns 3 threads. By passing the threads parameter to the rest api, you can customize the returned number of threads, which is shown in the below example.

curl –XGET http://localhost:9601/_node/hot_threads?threads=4

logstash5.png#asset:1203

After going through the monitoring API’s you may wonder how to push these metrics to elasticsearch and monitor Logstash via Kibana. Is there an out of the box configuration to push the metrics to ES directly? No. However, you can use a combination of http_poller input plugin and elasticsearch output plugin to achieve the same.

Logtash API is still experimental in nature and the APIs might change. Some of the APIs are still incomplete. For example when we execute “_node/stats/pipeline”, the stats api returns an empty array of input plugins. All the APIs support additional parameter “pretty=true”, but only “hot_threads” api supports “human=true” parameter. Still, this should be a good place to start exploring Logstash monitoring API’s. Questions/Comments? Drop us a line below.

Related Helpful Resources

Give It a Whirl!

It’s easy to spin up a standard hosted Elasticsearch cluster on any of our 47 Rackspace, Softlayer, Amazon or Microsoft Azure data centers. And you can now provision a replicated cluster.

Questions? Drop us a note, and we’ll get you a prompt response.

Not yet enjoying the benefits of a hosted ELK-stack enterprise search on Qbox? We invite you to create an account today and discover how easy it is to manage and scale your Elasticsearch environment in our cloud hosting service.

Categorized under: