So you have moved all your applications to Docker and have begun enjoying all the fruits of lightweight and fast-to-deploy containers. 

That's great, but once you have multiple containers spread across multiple nodes, you'll need to find a way to track their health, storage, CPU, and memory usage, network load, etc. 

To track these metrics, you need an efficient monitoring solution and some backend store to keep your container data for subsequent analysis and processing. Managing thousands of Docker containers in production made our team here at Qbox quickly realize that Docker container monitoring is a valuable addition to our cluster management process. 

In a previous article, we discussed how to use Metricbeat to ship metrics from Kubernetes. Now, it's time to share our experience of using Metricbeat to monitor bare Docker containers and shipping container data to Elasticsearch and Kibana. This knowledge may be useful for developers and administrators who manage Docker containers without orchestration. Let's get started!

Keep reading

Kubernetes is a popular container orchestration and container management platform for automating deployment, scheduling, and update of your containerized workloads in distributed compute environments. It goes without saying that managing multiple nodes and applications in Kubernetes requires an efficient monitoring system. You need to have a real-time picture of events happening in your cluster to get actionable insights for optimization and improving performance. 

Kubernetes ships with some default monitoring and metrics solutions like Heapster and Kubernetes Metrics Server. However, in order to apply analytics, do data discovery, and visualize metrics data flowing from your cluster, you'll be better off using solutions designed specifically for such type of tasks. One popular option for log and metrics monitoring and analysis is the ELK stack (Elasticsearch, Logstash, Kibana) used in pair with Elastic Beats log shippers. 

In this article, we introduce you to monitoring Kubernetes with ELK and Elastic Beats. In particular, we'll show how to send Kubernetes metrics to Elasticsearch indexes using Metricbeat and access them in your Kibana dashboard for subsequent processing. Let's get started!

Keep reading

In Part I of this overview, we'll explore the Qbox plugins for morphological and phonetic analysis, tokenization and concatenation, and native scripting, among others. By the end of this review, you'll have a better understanding of what plugins you might wish to install on your Qbox-hosted cluster.

Keep reading

While a search request returns a single “page” of results, the scroll API can be used to retrieve large numbers of results (or even all results) from a single search request, in much the same way as you would use a cursor on a traditional database. Scrolling is not intended for real time user requests, but rather for processing large amounts of data, e.g. in order to reindex the contents of one index into a new index with a different configuration.

The results that are returned from a scroll request reflect the state of the index at the time that the initial search request was made, like a snapshot in time. Subsequent changes to documents (index, update or delete) will only affect later search requests.

Keep reading

Redis, the popular open source in-memory data store, has been used as a persistent on-disk database that supports a variety of data structures such as lists, sets, sorted sets (with range queries), strings, geospatial indexes (with radius queries), bitmaps, hashes, and Hyper Logs. The in-memory store is used to solve various problems in areas such as real-time messaging, caching, and statistic calculation.

Provisioning an Elasticsearch cluster in Qbox is easy. In this article, we walk you through the initial steps to start and configure your cluster. We then setup and configure logstash to ship the logs to elasticsearch in order to monitor Redis performance. Redis performance logs shipped to elasticsearch can then be visualized and analyzed via Kibana dashboards.

Keep reading

A comprehensive log management and analysis strategy is vital, enabling organizations to understand the relationship between operational, security, and change management events and maintain a comprehensive understanding of their infrastructure. Log files from web servers, applications, and operating systems also provide valuable data, though in different formats, and in a random and distributed fashion.

No real-world web application can exist without a data storage backend, and most applications today use relational database management systems (RDBMS) for storing and managing data. The most commonly used database is MySQL, which is an open-source RDBMS that is the ‘M’ in the open-source enterprise LAMP Stack (Linux, Apache, MySQL and PHP).

Middle and large-sized applications send multiple database queries per second, and slow queries are often the cause of slow page loading and even crashes. The task of analyzing query performance is critical to determine the root cause of these bottlenecks, and most databases come with built-in profiling tools to help us.

Provisioning an Elasticsearch cluster in Qbox is easy. In this article, we walk you through the initial steps and show you how simple it is to start and configure your cluster. We then install and configure logstash to ship our MySQL or MariaDB/Galera logs to Elasticsearch. MySQL logs shipped to elasticsearch can then be visualized and analyzed via Kibana dashboards.

Keep reading