A comprehensive log management and analysis strategy is mission critical, enabling organizations to understand the relationship between operational, security, and change management events and to maintain a comprehensive understanding of their infrastructure. Log files from web servers, applications, and operating systems also provide valuable data, although in different formats, and in a random and distributed fashion.

As with any web server, the task of logging NGINX is somewhat of a challenge. NGINX access and error logs can produce thousands of log lines every second, and this data, if monitored properly, can provide you with valuable information not only on what has already transpired but also on what is about to happen. But how do you extract actionable insights from this information? How do you effectively monitor such a large amount of data?

NGINX access logs contain a wealth of information including client requests and currently active client connections that, if monitored efficiently, can provide a clear picture of how the web server and the application that it serves is behaving. This tutorial describes how Qbox can be used to overcome this challenge by monitoring NGINX access logs with Qbox provisioned Elasticsearch Stack.

Keep reading

A common use case that comes up when we use any product is how can we get metrics from it? How can we monitor it? Elasticsearch, since its early release, has always provided a way to monitor it using the _cat/stats API. However, for Logstash there wasn’t a way to gather metrics and monitor it until recently. With the release of Logstash 5.0+, Logstash has introduced a set of APIs to monitor Logstash.  In this article we explore the monitoring APIs exposed by Logstash, which includes the Node Info API, the Plugins API, the Node Stats API, and the Hot Threads API. 

Keep reading

A question that we answer quite often is: What's the best way to monitor key performance metrics in Elasticsearch—such as response time? There are several open source projects for #Elasticsearch monitoring tools, and one very good commercial solution. In this article, we invite you to take three minutes our of your day to acquaint yourself with the best monitoring tools.

Keep reading

From Day One, Qbox has offered a number of monitoring plugins on our Elasticsearch cluster dashboard.

However, Marvel also has the same limitation of the other monitoring plugins: an engineer or tech must monitor the metrics manually to get event notifications. Qbox closes this gap—and establishes a lead-in to other upcoming features—with our new active cluster alerting feature.

We now have a system component that continuously checks clusters for potential problems, then creates an event and notifies users of issues. Continue reading to learn about the types of cluster alerts that we send.

Keep reading

During our recent trip to Boston for an Elasticsearch meetup, we heard a great talk by Xiao Yu on the challenges presented by large Elasticsearch clusters. If you've used a distributed data store, you probably can relate to the difficulty of managing several nodes. Xiao (@HypertextRanch), a Boston dev and blogger, knows all about those challenges -- and he created a new Elasticsearch plugin to deal with them.

Keep reading

Elasticsearch Marvel Released

Posted by Michael Lussier January 28, 2014

Elasticsearch has released Marvel, a cluster management and monitoring product built by members of the team itself. Marvel is an incredibly detailed system to monitor your cluster events providing deep insight into real-time metrics on your cluster, node, and indices. Marvel keeps you ten steps ahead allowing you to see where problems will arise. Giving you the heads up on every event and exactly what was affected. Even when you need to travel back in time, Marvel has you fully prepared, storing all historical metrics for you to view anytime. This allows for more time developing and less time dealing with cluster headaches.

Keep reading