A common use case that comes up when we use any product is how can we get metrics from it? How can we monitor it? Elasticsearch, since its early release, has always provided a way to monitor it using the _cat/stats API. However, for Logstash there wasn’t a way to gather metrics and monitor it until recently. With the release of Logstash 5.0+, Logstash has introduced a set of APIs to monitor Logstash.  In this article we explore the monitoring APIs exposed by Logstash, which includes the Node Info API, the Plugins API, the Node Stats API, and the Hot Threads API. 

Keep reading

“ChatOps” has recently become a buzzword in places that are aiming for continuous delivery. It is based on chat clients like Slack and Hipchat and is plugged in with chatbots for real-time communication and task execution among members of development and IT operations teams.

Chat has become an integral part of the “better” delivery models. With huge amounts of data flowing within the system, wouldn’t it be nice if we could put it into an analysis tool and churn out some results that might improve the business?

In this article, we explore how to integrate Slack with Elasticsearch and perform basic data analyses for examples.

Keep reading

Having the ability to deploy Elasticsearch, Logstash and Kibana (ELK) from a single command is a wonderous thing. Together, in this post, we shall build the Ansible playbook to do just that.

There are some prerequisites. This Ansible playbook is made for Ubuntu Server and executed on Ubuntu Server 16.04. A basic system of 2 CPU cores and 4GB of RAM will be enough. The specs of the machine are entirely up to the situation and the volume of data.

This blog post is an alternative to using the ELK stack on Qbox. To easily deploy and run your own ELK setup on Qbox, simply sign up or launch your cluster here, and refer to the tutorial "Provisioning a Qbox Elasticsearch Cluster."

Keep reading

If you are interested in networking or information security then you are likely familiar with the port scanning tool nmap. Network Mapper is a free and open source (license) utility for network discovery and security auditing. Many systems and network administrators also find it useful for tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime. 

If you're unaware, I warn you that using nmap to port scan IP addresses of infrastructure that you don’t own is most likely illegal in your country. To be safe, scan only your own infrastructure, or get permission to do so. This article assumes that you know how to use nmap.

Keep reading

In many cases we want to use inputs from different databases which are not natively supported by Elasticsearch. In this post we show how to migrate data from a MySql database to Elasticsearch via Logstash.

Keep reading

This article explains how to use Logstash to import CSV data into Elasticsearch. We make use of the file input, CSV filter, and Elasticsearch output components of Logstash. Importing CSV into Elasticsearch using Logstash is a pretty simple and straightforward task, but several aspects of this process can make importing a CSV into Elasticsearch complicated quickly. I'm going to teach you some concepts that are important in this context. Some of these concepts will be useful for working with Logstash and Elasticsearch in general.

Keep reading