Scaling Elasticsearch is not an easy task. In this article, we go over different methods to make a High-Availability Logstash Indexing Solution using Qbox Hosted Elasticsearch.

Logstash Indexer is the component that indexes events and sends them to Elasticsearch for faster searches. We will use multiple logstash indexers with the exact same configuration. Having multiple indexers with the same configuration opens up different possibilities to make a highly available logstash solution for your ELK stack. These indexer nodes with identical configuration can easily be created using configuration management tools like Puppet or Chef.

The figure show below is one of the methods to scale your ELK stack with multiple indexers:

Indexing-Logstash.png#asset:1408

The Logstash-Forwarder / Filebeat shown in the above diagram is actually the log forwarder agents that will reside on your servers. These will forward logs to the IP Address or Domain Name of your indexers.

Since we have multiple indexers here, there are multiple approaches that we can use. Let’s imagine you have 1000 servers in your environment, and need to forward events to the indexer. In that case, there are a couple possibilities.

  1. Use the first indexer “IP address/domain name” across 500 servers, and use the second indexer “IP address/domain name” across the next set of 500 servers. This can be easily achieved using configuration management tools like Chef or Puppet.
  2. The other alternative is to use a DNS based round robin load balancing.  In this method, you add IP addresses of both the indexers behind the same dns name. For example, indexer.example.com is pointing to 10.12.2.1 & 10.12.2.2, where 10.12.2.1 is the first indexer & 10.12.2.2 is the second indexer. DNS will, by default, do round robin if you have multiple addresses added to a single name. You then simply use the same indexer dns name across your server fleet.

Keep in mind that we have used only two logstash indexers in the above example, just to keep things simple. You can scale it up to whatever number of nodes you want, and spread the load across them using different IP’s or DNS based load balancing.

Tutorial

For this post, we will be using hosted Elasticsearch on Qbox.io. You can sign up or launch your cluster here, or click "Get Started" in the header navigation. If you need help setting up, refer to "Provisioning a Qbox Elasticsearch Cluster."

Filebeat can use either of the above methods in its configuration inside the OUTPUT section.

Filebeat actually comes with a “load balance” option. You can also place the IP addresses of all your indexers right into Ffilebeat running on all servers.

Let’s consider an example filebeat.yml file, as shown below:

cat /etc/filebeat/filebeat.yml
filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/*.log
  input_type: log
  document_type: syslog
  registry: /var/lib/filebeat/registry
output.logstash:
  hosts: ["10.12.2.1:6000", "10.12.2.2:6000"]
  loadbalance: true
  index: filebeat

The hosts array shown in the above example contains all the logstash indexer nodes in our environment.

The loadbalance: true option will make Filebeat send events to all the hosts (10.12.2.1 & 10.12.2.2 in our example above) in a load balanced manner.

This will send events to all the hosts mentioned in a “one after the other method”.  Filebeat will stop sending events to the indexer if it fails to respond with an ACK. Which means if one of your indexer goes down, you still have the other one to take care.

As mentioned in the beginning, we will be using a Qbox.io elasticsearch cluster. Our logstash indexer nodes will have Qbox elasticsearch endpoint and credentials.

A typical logstash.conf on the indexer will look like below:

input {
 beats {
   port => 6000
 }
syslog {
   type => syslog
   port => 5514
  }
}
filter {
 if [type] == "syslog" {
   grok {
    .
    .
    .
   }
 }
}
output {
  stdout { }
  elasticsearch { 
     host =>"eb745857.qb0x.com"
     port => 443
     protocol => "http"
     use_ssl => true
     user => "ec184833427234735hf08345d3"
     password => "ejkgh6g1e0"
  }
}

The example output section shown above contains credentials to access qbox cluster. You can add your application specific filters in the filter section as usual.

Haproxy

Instead of using this filebeat method to send events in a distributed manner, we can use popular load balancing tools like Haproxy. Haproxy can sit in front, and balance load across all indexer nodes.  In this case, your output section in filebeat will not be a list of servers, but the Haproxy IP address/DNS name.


logstash2_170804_143027.png#asset:1409

The load balancer shown above can be either Haproxy, or a standard Elastic Load Balancer from Amazon. This load balancer will sit and redirect requests coming from the nodes on the left side to the indexers.

An example configuration of Haproxy is shown below:

listen logstash-5001 10.2.2.1:5001
    mode tcp
    balance leastconn
    option tcplog
    server logstash-indexer1 10.2.2.2:5001 check
    server logstash-indexer2 10.2.2.3:5001 check

The above configuration will load balance the traffic coming to 10.2.2.2 and 10.2.2.3.

All logstash forwarders/filebeat can use the IP address OR DNS name of the Haproxy node.

Note:  You can use the same logstash indexer example configuration with qbox elasticsearch output in this case as well.

In the load balanced scenario we just saw, there is a single point of failure. The single point of failure is the load balancer itself. This can be solved by using something called keepalived.

You can solve this problem by introducing another Haproxy server, which in total is two Haproxy servers with identical configuration. The idea is to have only one Haproxy act as the ACTIVE one and the other as standby. The standby will become ACTIVE if the other goes down.

A single IP address keeps on floating between two Haproxy nodes. If the active one is not available, the standby will assign the ip address to itself, so that requests from client will automatically be routed to it.

We basically need two Haproxy servers( with identical configuration - the configuration that we saw above) and both of them should have keepalived configured.

You can install keepalived using apt-get install keepalived. The configuration file for keepalived is located at /etc/keepalived/keepalived.conf.

vrrp_script chk_haproxy {           
        script "killall -0 haproxy"     
        interval 2                      
        weight 2                        
}
vrrp_instance VI_1 {
        interface eth0
        state MASTER
        virtual_router_id 51
        priority 101                    
        virtual_ipaddress {
            10.2.2.1
        }
        track_script {
            chk_haproxy
        }
}

The second Haproxy can have the same keepalived configuration with one single change. The change is of the line priority. Basically ACTIVE one will have a priority of higher value, and passive one with a lesser value. That is the only difference.  For example, priority 101 will become ACTIVE and priority 100 will become the PASSIVE.

Apart from this, you can also take another approach of DNS based load balancing by having two Haproxy nodes. However it is not reliable, because DNS won't know if an Haproxy node is up and running. It will simply return both the addresses in round robin fashion.

Being said that, if you have a feature where the DNS can do some sort of health checks on the IP addresses and respond with only healthy and active nodes, then taking the DNS based approach is also quite possible.

Conclusion

We have different methods available to scale logstash indexers which do the heavy lifting of filtering logs and sending it to elasticsearch. One is to use the forwarder side techniques like “filebeat” load balancing.  The second approach is to use multiple indexers, and use them evenly across the server fleet. The third approach is to use DNS based load balancing. The fourth is to use the standard open source load balancing softwares like Haproxy with keepalived to make it highly available. You can also use multiple Haproxy nodes with DNS round robin, provided you have health check feature in your DNS.

Other Helpful Tutorials

Give It a Whirl!

It's easy to spin up a standard hosted Elasticsearch cluster on any of our 47 Rackspace, Softlayer, Amazon, or Microsoft Azure data centers. And you can now provision your own AWS Credits on Qbox Private Hosted Elasticsearch

Questions? Drop us a note, and we'll get you a prompt response.

Not yet enjoying the benefits of a hosted ELK stack enterprise search on Qbox? We invite you to create an account today and discover how easy it is to manage and scale your Elasticsearch environment in our cloud hosting service.

comments powered by Disqus