A comprehensive log management and analysis strategy is mission critical, enabling organizations to understand the relationship between operational, security, and change management events and to maintain a comprehensive understanding of their infrastructure. Log files from web servers, applications, and operating systems also provide valuable data, although in different formats, and in a random and distributed fashion.

As with any web server, the task of logging NGINX is somewhat of a challenge. NGINX access and error logs can produce thousands of log lines every second, and this data, if monitored properly, can provide you with valuable information not only on what has already transpired but also on what is about to happen. But how do you extract actionable insights from this information? How do you effectively monitor such a large amount of data?

NGINX access logs contain a wealth of information including client requests and currently active client connections that, if monitored efficiently, can provide a clear picture of how the web server and the application that it serves is behaving. This tutorial describes how Qbox can be used to overcome this challenge by monitoring NGINX access logs with Qbox provisioned Elasticsearch Stack.

Keep reading

A comprehensive log management and analysis strategy is mission critical, enabling organizations to understand the relationship between operational, security, and change management events and to maintain a comprehensive understanding of their infrastructure. Log files from web servers, applications, and operating systems also provide valuable data, although in different formats, and in a random and distributed fashion.

Logs are a crucial part of any system because they give you insight into what a system is doing as well what happened. Virtually every process running on a system generates logs in some form or another. These logs are usually written to files on local disks. When your system grows to multiple hosts, managing the logs and accessing them can get complicated. 

Searching for a particular error across hundreds of log files on hundreds of servers is difficult without good tools. A common approach to this problem is to set up a centralized logging solution so that multiple logs can be aggregated in a central location. To effectively consolidate, manage, and analyze these different logs, many customers choose to implement centralized logging solutions using Elasticsearch, Logstash, and Kibana, popularly known as ELK Stack.

Keep reading

A common use case that comes up when we use any product is how can we get metrics from it? How can we monitor it? Elasticsearch, since its early release, has always provided a way to monitor it using the _cat/stats API. However, for Logstash there wasn’t a way to gather metrics and monitor it until recently. With the release of Logstash 5.0+, Logstash has introduced a set of APIs to monitor Logstash.  In this article we explore the monitoring APIs exposed by Logstash, which includes the Node Info API, the Plugins API, the Node Stats API, and the Hot Threads API. 

Keep reading

“ChatOps” has recently become a buzzword in places that are aiming for continuous delivery. It is based on chat clients like Slack and Hipchat and is plugged in with chatbots for real-time communication and task execution among members of development and IT operations teams.

Chat has become an integral part of the “better” delivery models. With huge amounts of data flowing within the system, wouldn’t it be nice if we could put it into an analysis tool and churn out some results that might improve the business?

In this article, we explore how to integrate Slack with Elasticsearch and perform basic data analyses for examples.

Keep reading

Having the ability to deploy Elasticsearch, Logstash and Kibana (ELK) from a single command is a wonderous thing. Together, in this post, we shall build the Ansible playbook to do just that.

There are some prerequisites. This Ansible playbook is made for Ubuntu Server and executed on Ubuntu Server 16.04. A basic system of 2 CPU cores and 4GB of RAM will be enough. The specs of the machine are entirely up to the situation and the volume of data.

This blog post is an alternative to using the ELK stack on Qbox. To easily deploy and run your own ELK setup on Qbox, simply sign up or launch your cluster here, and refer to the tutorial "Provisioning a Qbox Elasticsearch Cluster."

Keep reading

If you are interested in networking or information security then you are likely familiar with the port scanning tool nmap. Network Mapper is a free and open source (license) utility for network discovery and security auditing. Many systems and network administrators also find it useful for tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime. 

If you're unaware, I warn you that using nmap to port scan IP addresses of infrastructure that you don’t own is most likely illegal in your country. To be safe, scan only your own infrastructure, or get permission to do so. This article assumes that you know how to use nmap.

Keep reading