Integrating an application with Elasticsearch can be achieved two ways, one using REST APIs, and the other using Native clients. In the article “REST Calls Made Easy - A New Elasticsearch Java Rest Client”, we covered extensively the new Java REST Client API to integrate easily with Elasticsearch.

Keep reading

This tutorial explains how to configure alerting using ElastAlert with the popular proprietary issue tracking product JIRA.

ElastAlert is now available on Qbox provisioned Elasticsearch clusters and can be easily configured. Implementing ElastAlert is easy on Qbox. When you provision a cluster, there is a configuration box where you can input your Alert rules.  If you’re unclear how to structure rules in YAML, be sure to consult the ElastAlert Documentation.

Keep reading

Not yet enjoying the benefits of a hosted ELK-stack enterprise search on Qbox? Discover how easy it is to manage and scale your Elasticsearch environment.

Get Started 5 minutes to get started

In the previous tutorial in ElastAlert Series, we implemented new_term, change and spike rules for ElastAlert alerting via Slack. We will next be looking into configuring and setting up alerting using ElastAlert on to the popular cloud-based team collaboration tool HipChat.

Many organisations use Elasticsearch to rapidly prototype and launch new search applications, and moving quickly at scale raises challenges. In particular, we often encounter difficulty making changes to query logic without impacting users, as well as finding client library bugs, problems with multi-tenancy, and general reliability issues. As the number of queries grow, the Search Infrastructure faces difficulty in supporting the multitude of ways queries are being sent to Elasticsearch cluster. The infrastructure designed for a single team to communicate with a single cluster does not scale to tens of teams and tens of clusters.

Indexing in large volumes require instantaneous alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch. If you have data being written into Elasticsearch in near real time and want to be alerted when that data matches certain patterns, ElastAlert is the tool for you.

Keep reading

In the previous tutorial in ElastAlert Series, we implemented flatline, frequency and blacklist rules for ElastAlert alerting via Email. We will be next looking into configuring and setting up alerting using ElastAlert on to the popular cloud-based team collaboration tool Slack.

ElastAlert was developed to automatically query and analyze the log data in Elasticsearch clusters and generate alerts based on easy-to-write rules. The initial goal was to create a comprehensive log management system for the data. It is easy to configure a few basic alerts such as “Send us an email if a user fails login X times in a day” or “Send a Sensu alert if the number of error messages spikes.” But, the usual requirement is a generic architecture which could suit almost any alerting scenario needed across any organisation using Elasticsearch. ElastAlert takes a set of “rules”, each of which has a pattern that matches data and a specific alert action it will take when triggered. For each rule, ElastAlert will query Elasticsearch periodically to grab relevant data in near real time.

ElastAlert is now available on Qbox provisioned Elasticsearch clusters and can be easily configured. Implementing ElastAlert is easy on Qbox. When you provision a cluster, there is a configuration box where you can input your Alert rules.  If you’re unclear how to structure rules in YAML, be sure to consult the ElastAlert Documentation.

Keep reading

ELK scales well and helps with incident response, comparing metrics, tracking bugs, etc. However, as the number of dashboards and amount of data grow, we have a need for automation. Unless someone is actively looking at a dashboard or searching for the right thing, we miss a lot.

We need a way to monitor the data we have in Elasticsearch in near real time. We want a generic way to look for certain patterns in our data, without duplicating our data somewhere or spinning up a heavyweight service. The final requirement is the data should be accessible to engineers and operations from every team across the organization for quick resolution.

ElastAlert, developed by Yelp, is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch. Several organisations use Elasticsearch, Logstash and Kibana for managing their ever increasing amount of data and logs. Kibana is great for visualizing and querying data, but a companion tool is needed for alerting on inconsistencies in the data. Out of this need, ElastAlert was created. If you have data being written into Elasticsearch in near real time and want to be alerted when that data matches certain patterns, ElastAlert is the tool for you.

Keep reading

Before setting up elasticsearch to fulfill entity extraction, it is worth checking out how it became such an easy task. There is a lot of buzz around the new Ingest API shipped with elasticsearch 5.x.

The Ingest API allows data manipulation and enrichment by defining a pipeline through which every document is subject to pass. This pipeline is created with a set of processors - each of which do specific tasks that enrich our data. A typical example of the processor is a grok processor, which allows you to modify and structure your unstructured log using pattern matching. Elasticsearch 5 ships many built-in processors about which you can read here.

Keep reading