In terms of security, the internet has become a very hostile place. All kinds of bad guys from criminals to government agencies are out to hack you, and they must be taken seriously.

One thing about being a developer is that you build things with deadlines and you don't always have time to plan the security aspects of whatever it is you are developing. Attacking developers directly is becoming very popular as we can see from the hacking team hack, for example.

Elasticsearch is an extremely powerful tool not only for developers but also for system administrators. Some people log almost everything to Elasticsearch, including sensitive data. It is therefore very important that you restrict who is able to access your Elasticsearch clusters, as well as what you are logging to them.

If you are using Elasticsearch in a developer environment and you can trust the other people on your network, then it is probably okay to not worry that much about security, as long as you aren’t storing anything sensitive in Elasticsearch. If you want to expose your Elasticsearch cluster to the public internet, however, you need to lock it down extensively, as well as perform regular testing of your security.

Elasticsearch is a datastore in the way that a relational database is a datastore. It is just usually used to store different kinds of data. The data is just as important, if not more.

"If money could be said to exist anywhere in a network, it exists on a database server." - The Database Hacker's Handbook Defending Database Servers, David Litchfield and Chris Anley

With relational databases there is the concept of access control where you can lock down access to certain databases, columns, or tables, based on users or groups. Elasticsearch has no such concept or such finer-grained access control. 

Let’s say, for example, you wrote a PHP application that was using Elasticsearch to store data of all your different clients. If one of those clients could find a bug in the application that gave them direct access to Elasticsearch, they would be able to access data in all indexes and access the data that belongs to other clients. If this application was written using a relational database such as MySQL or Postgresql, then you would have been able to lock down access by giving each client access only to certain columns and tables in a database. The commercial Elasticsearch plugin, SHIELD, is supposed to address this problem.

Introducing our new open source datacenter total control system that makes it easy to save up to 60% on your AWS bill. Supergiant.io

In a way, this feature of Elasticsearch makes sense from an operations context because system administrators usually have access to everything. The only people who should have direct access to Elasticsearch would be the system administrators. This works in an ideal world, but these days developers (not only system administrators) are also using Elasticsearch.

Finding Public-Facing Elasticsearch Clusters

You can find public-facing Elasticsearch clusters with Shodan as follows.

https://www.shodan.io/search?query=elasticsearch

If you want to automate this process, Shodan has an API. However, beware that interacting with other people’s Elasticsearch clusters is likely illegal in your country, even if they are wide open for you to access.

Elasticsearch has a very powerful REST API, and as with any API that you are considering publishing to the public internet, you will have to add authentication either in the form of API keys or username and password authentication.

If you are not working on a tight budget and you want a commercial solution that will help with security and restricting access to the ELK stack, then you should consider the paid plugin for Elasticsearch called SHIELD. Paying for this plugin seems worthwhile because you can pay for support for this plugin. With SHIELD you can do all kinds of things such as user authentication, role-based access control, and even auditing of Elasticsearch security-related Logs.

If you don't have money to pitch out but you do have time and effort to invest, then look at using the following. A common setup that many people use is to only allow access to Elasticsearch from localhost and to restrict access to Kibana by using Nginx as a reverse proxy to Kibana running on localhost. You can then either use HTTP authentication to restrict access to Kibana, or you can use Nginx's ability to do LDAP authentication. This is useful, for example, if you would like to authenticate against active directory.

It is a good idea to add authentication to Elasticsearch, too. The below example just shows how to give users access authenticated access to Kibana only.

logstash -> elasticsearch -> kibana (Running on 127.0.0.1:5601) -> Nginx Reverse Proxy Serving us Kibana but on a public IP.

You can look online for examples on configuring Kibana to use Nginx with HTTP auth or LDAP authentication. Remember to use SSL, too. There is no use adding authentication when you are sending sensitive credentials in cleartext over a network.

Adding Authentication to Elasticsearch

You can add authentication to Elasticsearch’s REST API using Nginx. You can add different kinds of authentication such as LDAP or HTTP auth. Make sure to also use SSL. (This example does not use SSL.)

This is how our setup will work:

Logstash (Optional) -> elasticsearch (Running on 127.0.0.1:9200) -> Nginx Reverse Proxy Serving us Elasticsearch but on a public IP with authentication

This allows us to access Elasticsearch on http://our_domain.com on port 80. If you are using SSL, then it will be served on port 443.

This is what our nginx configuration should look like. Remember to create a .htpasswd file.

# My Nginx conf
server {
  listen                *:80 ;
 server_name           server;
  access_log            /var/log/nginx/es.log;
  error_log            /var/log/nginx/es.log;
 location / {
    proxy_pass http://localhost:9200;
    proxy_set_header Host $host;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    auth_basic "Restricted";
    auth_basic_user_file /etc/nginx/conf.d/es.htpasswd;
  }
}

Interacting with Authenticated Elasticsearch via Python Client API

In this example I added HTTP authentication to Elasticsearch, but I didn’t add SSL. I’d like to use this article to get you thinking -- but not to create a tutorial from which you can copy/paste. If you are going to add HTTP authentication, then please also use SSL.

If you are using the Python client API, then please consider the following: “By default SSL certificates won’t be verified, pass in verify_certs=True to make sure your certificates will get verified” (from https://elasticsearch-py.readthedocs.org/en/master/). Please look at the examples of using SSL with the Python Elasticsearch client API’s documentation because the client API does not ship with any CA certificates, but the examples show you how to point the client to the CA certs on your server.

This is how we would interact with our server that is running Elasticsearch. Elasticsearch was configured to only run on localhost. We are connecting to Elasticsearch which is running on localhost, thanks to Nginx that is acting as a reverse proxy.

Here is an example of interacting with Elasticsearch from my desktop computer. Note that Elasticsearch is running on a server which is on the same network, but not on my desktop.

#!/usr/bin/env python
import requests
from elasticsearch import Elasticsearch
import json
es = Elasticsearch(["http://timo:secretpassword@172.20.0.152:80"])
res = es.search(index="nginx_json_elk_example", body={ 
      "query": {
    "filtered": {
      "query": {
        "query_string": {
          "query": "*",
          "analyze_wildcard": "true"
        }
      },
      "filter": {
        "bool": {
          "must": [
            {
              "range": {
                "@timestamp": {
                  "gte": "now-12M",
                  "lte": "now",
                  "format": "epoch_millis"
                }
              }
            }
          ],
          "must_not": []
        }
      }
   }
  },
  "size": 0,
  "aggs": {
    "blah": {
      "terms": {
        "field": "remote_ip.raw",
        "size": 1000,
        "order": {
          "_count": "desc"
        }
      }
    }
  } })
#re
for f in res['aggregations']['blah']['buckets']:
    print('Request from IP %s' % f['key'])

Other Security Considerations

Set up your firewall. In the case of Linux, you will need to set up IPTables or use UFW, which is just a wrapper for Iptables. Also, check that none of the services are running on an IP rather than on localhost. You can do that with:

# lsof -i
COMMAND   PID          USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
dhclient  723          root    5u  IPv4  11546      0t0  UDP *:bootpc 
dhclient  723          root   20u  IPv4  11512      0t0  UDP *:44172 
dhclient  723          root   21u  IPv6  11513      0t0  UDP *:38515 
sshd      950          root    3u  IPv4  11728      0t0  TCP *:ssh (LISTEN)
sshd      950          root    4u  IPv6  11730      0t0  TCP *:ssh (LISTEN)
sshd     1117          root    3u  IPv4  11804      0t0  TCP es.lan:ssh->computer.lan:43280 (ESTABLISHED)
sshd     1165          timo    3u  IPv4  11804      0t0  TCP es.lan:ssh->computer.lan:43280 (ESTABLISHED)
java     1409 elasticsearch  118u  IPv6  10973      0t0  TCP localhost:9300 (LISTEN)
java     1409 elasticsearch  120u  IPv6  12489      0t0  TCP localhost:9300 (LISTEN)
java     1409 elasticsearch  149u  IPv6  12522      0t0  TCP localhost:9200 (LISTEN)
java     1409 elasticsearch  151u  IPv6  12523      0t0  TCP localhost:9200 (LISTEN)

Testing for Elasticsearch on your Network

We do an nmap scan for Elasticsearch. This is what Elasticsearch running on localhost on a host looks like when the host is port scanned with nmap.

$ nmap -PN -p9200 172.20.0.152
Starting Nmap 6.47 ( http://nmap.org ) at 2016-04-22 19:43 SAST
Nmap scan report for 172.20.0.152
Host is up (0.00046s latency).
PORT     STATE  SERVICE
9200/tcp closed wap-wsp
Nmap done: 1 IP address (1 host up) scanned in 13.08 seconds

Now let’s look at what happens if we change the Elasticsearch config to allow anyone on the network to talk to Elasticsearch.

We change the Elasticsearch config, which is located on Ubuntu at:

# vim /etc/elasticsearch/elasticsearch.yml

Add the following to your config file:

# This is what should be in your /etc/elasticsearch/elasticsearch.yml 
# Change this line so that Elasticsearch will be listening on this IP. 
network.host: 172.20.0.152

Now let's restart the Elasticsearch service and then run our port scan again:

# service elasticsearch restart
 * Starting Elasticsearch Server
   ...done.

Now let's run out portscan again:

$ nmap -PN -p9200 172.20.0.152
Starting Nmap 6.47 ( http://nmap.org ) at 2016-04-22 19:49 SAST
Nmap scan report for 172.20.0.152
Host is up (0.00042s latency).
PORT     STATE SERVICE
9200/tcp open  wap-wsp

Seeing results from a security-related scanning tool is never good enough. You need to confirm your findings somehow. Security scanning tools can show false positives or even false negatives.

Let's see if we can interact with the host with is supposedly running Elasticsearch.

$ curl -XGET http://172.20.0.152:9200/_cat/...
yellow open nginx_json_elk_example 5 1 51462 0 17.5mb 17.5mb 
yellow open search                 5 1     0 0   795b   795b 
yellow open .kibana                1 1    13 0 49.6kb 49.6kb

This is pretty dangerous. Consider that nmap can do network wide scans, so unlike in our example where we knew what host we were going to scan, you can rather specify an IP range and scan that. Finding developers running open Elasticsearch clusters on an internal network is really easy. If you were at a coffee shop that had open wifi with no network isolation in an area rich with software developers, chances are you might find somebody running Elasticsearch on their laptop. My motivation for pointing out these things is not to encourage you to do illegal things and attack other developers but rather to make you more security conscious.

$ nmap -PN -p9200 172.20.0.1-200
....
Nmap scan report for 172.20.0.151
Host is up.
PORT     STATE    SERVICE
9200/tcp filtered wap-wsp
Nmap scan report for 172.20.0.152
Host is up (0.00028s latency).
PORT     STATE SERVICE
9200/tcp open  wap-wsp
Nmap scan report for 172.20.0.153
Host is up.
PORT     STATE    SERVICE
9200/tcp filtered wap-wsp
...

Conclusion

The point of this article is not to make you paranoid or to show you ways to hack other people. My goal is to make you more aware of vulnerabilities. Elasticsearch is a very powerful tool, and if you use it, you should take note of the possible security precautions that are needed to use it in a production environment. 

Be careful when using Elasticsearch in development, but also make sure that the person doing the production deployment of your Elasticsearch clusters is security conscious.

comments powered by Disqus