Monitor Logstash 5.0 with the New Monitoring APIs
Posted by Adam Vanderbush March 9, 2017A common use case that comes up when we use any product is how can we get metrics from it? How can we monitor it? Elasticsearch, since its early release, has always provided a way to monitor it using the _cat/stats API. However, for Logstash there wasn’t a way to gather metrics and monitor it until recently. With the release of Logstash 5.0+, Logstash has introduced a set of APIs to monitor Logstash. In this article we explore the monitoring APIs exposed by Logstash, which includes the Node Info API, the Plugins API, the Node Stats API, and the Hot Threads API.
Tutorial
For this post, we will be using hosted Elasticsearch on Qbox.io. You can sign up or launch your cluster here, or click “Get Started” in the header navigation. If you need help setting up, refer to “Provisioning a Qbox Elasticsearch Cluster.“
For monitoring, Logstash exposes a set of 4 API’s: node info, node stats, plugins and hot threads api. When the Logstash is started, the monitoring API’s bind to the port 9600 by default. We can override the port by specifying the flag –“http.port”
or by specifying the value “
http.port”
in “
logstash.yml”
file found under config
folder.
Let’s create a basic Logstash pipeline and run Logstash with monitoring api bound to 9601.
# custom.conf input { stdin{ } } output { stdout{ codec => json } } #starting logstash using the above configuration. C:\qbox\Logstash -5.2.0\bin>Logstash -f custom.conf --http.port 9601
Once the Logstash is started, we can verify the general information about Logstash instance by invoking the root resource (/)
using either curl or using a web browser like below.
curl –XGET http://localhost:9601/
Node Info API
Information related to the underlying operating system, JVM (info like JVM version, memory initialized/allocated etc) and pipeline (info like batch size, workers, etc.) can be obtained using the nodes api. The rest api or the resource url for obtaining nodes info is “_node”
. To fetch information specific to pipeline or OS or JVM, we can append “type(s)”
to the nodes api:
curl –XGET<a href="http://localhost:9601/_node"> http://localhost:9601/_node</a>
If you want to retrieve node information with operating system and JVM details use this command:
curl –XGET<a href="http://localhost:9601/_node/os,jvm"> http://localhost:9601/_node/os,jvm</a>
Plugins API
Logstash pipeline is made of input, filter, and output plugins. Logstash ships with many default plugins and provides the ability to install hosted plugins or build and install custom plugins. The richness of Logstash is due to the huge number of plugins it supports. If you want to know the list of plugins the current Logstash instance is running with use the resource “_node/plugins”
.
curl –XGET<a href="http://localhost:9601/_node/plugins"> http://localhost:9601/_node/plugins</a>
Node Stats API
Node API provides a basic information about the node. However, if you are interested in gathering a detailed runtime metrics about the node make use of the node stats api. The node stats api provides detailed information about:
- JVM – including stats about threads, memory usage, garbage collectors, and uptime.
- Process – including stats about file descriptors, memory consumption, and cpu usage.
- Pipeline – runtime stats about the Logstash pipeline.
- Reloads – stats about config reload successes and failures.
- OS – runtime stats about cgroups when Logstash is running in a container.
The rest api or the resource url for obtaining nodes info is “_node/stats”
. You can append the “type(s)”[comma separated]
to the resource/rest url to get specific type runtime metrics.
curl –XGET http://localhost:9601/_node/stats curl –XGET http://localhost:9601/_node/stats/pipeline,reload
Note: By passing the flag “
--config.reload.automatic”
when starting Logstash, automatic configuration reloading can be enabled.
Hot Threads API
A hot thread is a Java thread that has high CPU usage and executes for a longer than normal period of time. Logstash provides hot threads api to retrieve the same. The rest api or the resource url for obtaining hot threads info is “_node/hot_threads”
. By default it returns 3 threads. By passing the threads parameter to the rest api, you can customize the returned number of threads, which is shown in the below example.
curl –XGET http://localhost:9601/_node/hot_threads?threads=4
After going through the monitoring API’s you may wonder how to push these metrics to elasticsearch and monitor Logstash via Kibana. Is there an out of the box configuration to push the metrics to ES directly? No. However, you can use a combination of http_poller
input plugin and elasticsearch
output plugin to achieve the same.
Logtash API is still experimental in nature and the APIs might change. Some of the APIs are still incomplete. For example when we execute “_node/stats/pipeline”
, the stats api returns an empty array of input plugins. All the APIs support additional parameter “
pretty=true”
, but only “
hot_threads”
api supports “
human=true”
parameter. Still, this should be a good place to start exploring Logstash monitoring API’s. Questions/Comments? Drop us a line below.
Related Helpful Resources
- Indexing Heroku Logs into Elasticsearch with Logstash
- Parsing Logs Using Logstash
- Migrating MySql Data Into Elasticsearch Using Logstash
- How to Import from CSV into Elasticsearch via Logstash and Sincedb
- Introduction to the Logstash Translate Filter
Give It a Whirl!
It’s easy to spin up a standard hosted Elasticsearch cluster on any of our 47 Rackspace, Softlayer, Amazon or Microsoft Azure data centers. And you can now provision a replicated cluster.
Questions? Drop us a note, and we’ll get you a prompt response.
Not yet enjoying the benefits of a hosted ELK-stack enterprise search on Qbox? We invite you to create an account today and discover how easy it is to manage and scale your Elasticsearch environment in our cloud hosting service.