Scaling Elasticsearch is not an easy task. In this article, we go over different methods to make a High-Availability Logstash Indexing Solution using Qbox Hosted Elasticsearch.

Logstash Indexer is the component that indexes events and sends them to Elasticsearch for faster searches. We will use multiple logstash indexers with the exact same configuration. Having multiple indexers with the same configuration opens up different possibilities to make a highly available logstash solution for your ELK stack. These indexer nodes with identical configuration can easily be created using configuration management tools like Puppet or Chef.

Keep reading

We have already discussed about indexing parent-child relationships in elasticsearch. We gave realised that the parent-child functionality allows us to associate one document type with another, in a one-to-many relationship—one parent to many children.

For this post, we will be using hosted Elasticsearch on Qbox.io. You can sign up or launch your cluster here, or click "Get Started" in the header navigation. If you need help setting up, refer to "Provisioning a Qbox Elasticsearch Cluster."

The advantages that parent-child has over nested objects are as follows:

  • The parent document can be updated without reindexing the children.

  • Child documents can be added, changed, or deleted without affecting either the parent or other children. This is especially useful when child documents are large in number and need to be added or changed frequently.

  • Child documents can be returned as the results of a search request.

Keep reading

Not yet enjoying the benefits of a hosted ELK-stack enterprise search on Qbox? Discover how easy it is to manage and scale your Elasticsearch environment.

Get Started 5 minutes to get started

Effective log management involves a possibility to instantly draw useful insights from millions of log entries, identify issues as they arise, and visualize/communicate patterns that emerge out of your application logs. Fortunately, ELK stack (Elasticsearch, Logstash, and Kibana) makes it easy to ship logs from your application to ES collections for storage and analysis. 

Recently, Elastic infrastructure was extended by useful tools for shipping logs called Beats. Filebeat is a part of Beats tool set that can be configured to send log events either to Logstash (and from there to Elasticsearch), or even directly to the Elasticsearch. The tool turns your logs into searchable and filterable ES documents with fields and properties that can be easily visualized and analyzed.

In a previous post, we discussed how to use Filebeat to ship Linux system logs. Now, it's time to show how to ship logs from your MySQL database via Filebeat transport to your Elasticsearch cluster. Making MySQL general and slow logs accessible via Kibana and Logstash will radically improve your database management, log analysis and pattern discovery leveraging the full potential of ELK stack.

Keep reading

We have been discussing extensively on Handling Relationships and Data Modeling in our series so far. The need to bridge the gap between flat mapping and the real world has made us focus on the following techniques.

  • Application-side joins

  • Data denormalization

  • Nested objects

Keep reading

Elasticsearch, by default, return the results sorted by relevance with the most relevant docs first. In order to sort by relevance, we need to represent relevance as a value. The relevance score of each document is represented by a positive floating-point number called the _score. The higher the _score, the more relevant the document.

Keep reading

In the previous tutorial, we have discussed how to use elasticsearch.js, the official Node.js client for Elasticsearch, to index, add documents, and search them using simple queries and Query DSL. In this tutorial, we're going to dive deeper into elasticsearch.js describing more advanced methods and concepts like scrolling, aggregations, and analyzers.

As always, we will be using hosted Elasticsearch on Qbox.io. We assume that you have installed the latest version of Node.js, downloaded the elasticsearch.js module into your Node.js application and connected it to your Elasticsearch cluster as described in the previous tutorial.

Keep reading