A Vagrant Virtual Machine for Elasticsearch 2.x

Posted by Sloan Ahrens February 2, 2016

Elasticsearch 2.0.0 introduced a number of breaking changes. When I set out to install ES 2.0 to do some local testing, I found that the techniques that I had been using to set up virtual machines for local development (for instance, here) were no longer adequate. So I set out to discover what the “proper” method should be, and along the way I ran into a few problems. I’ll outline those issues here, hopefully saving some other people a little bit of trouble.

The instructions that follow will assume you are using OSX. It should be straightforward to adapt them to other operating systems, but I will not address those considerations here.

Vagrant

In earlier posts I used full Ubuntu virtual machines (with UI), as this was my preferred method of development. Recently I have more often used headless VMs with Vagrant, and for many purposes I find this to be more useful. I find that running Elasticsearch in a Vagrant VM works nicely for local development, allowing me to sandbox the Elasticsearch installation, but port-forwarding means I can easily access ES as if it were running in my host OS.

To follow along, you will need Vagrant installed, which can be found here. Vagrant requires a provider, and I find VirtualBox to be convenient. You can find VirtualBox here.

Vagrantfile

The Vagrantfile we’ll use for our Elasticsearch VM is quite simple:

Vagrantfile:
Vagrant.configure(2) do |config|
  config.vm.box = "ubuntu/trusty64"
  config.vm.provision :shell, path: "bootstrap.sh"
  config.vm.network :forwarded_port, host: 9200, guest: 9200
  config.vm.provider "virtualbox" do |v|
    v.memory = 2048
    v.cpus = 1
    v.name = "ES-2.x"
  end
end

A few things to note here:

This VM uses the “ubuntu/trusty64” box, which runs Ubuntu Server 14.04 LTS. If you want to use a different box, you can find some here, or create your own.
The “bootstrap.sh” file is where our commands for installing Elasticsearch will reside; we’ll get to that in a minute.
We are forwarding port 9200 from the guest OS to the host OS, so we can access Elasticsearch at localhost:9200 just like we could if it were running in the host OS.
This is not a particularly beefy virtual machine, with a single CPU and 2GB of RAM. Depending on your application you may want to change these values.
The VM name “ES-2.x” is arbitrary; feel free to change it to whatever you want.

Next we’ll take a look at the bootstrap script. I’ll explain it piece by piece, but I’ve included the full file below for convenience.

Installing Elasticsearch

Elasticsearch requires at least Java 7. I like to use OpenJDK, mainly because I can install it without any prompts, which is convenient for a Vagrant setup.

Here are the commands we’ll use to install Java 7:

# install openjdk-7 
sudo apt-get purge openjdk*
sudo apt-get -y install openjdk-7-jdk

The Elasticsearch documentation provides a number of different ways to install Elasticsearch, and run it as a service. You can take a look here and here.

We will use the debian/ubuntu package. To do so we’ll need to add the correct repository, as described here.

This command adds the Elasticsearch repository Public Signing Key:

wget -qO - <a href="https://packages.elastic.co/GPG-KEY-elasticsearch">https://packages.elastic.co/GPG-KEY-elasticsearch</a> | sudo apt-key add -

This command adds the repository (see the explanation here for why we need to use the echo method):

echo "deb <a href="http://packages.elastic.co/elasticsearch/2.x/debian">http://packages.elastic.co/elasticsearch/2.x/debia...</a> stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list

Then we can update Aptitude and install Elasticsearch with:

sudo apt-get update && sudo apt-get install elasticsearch

We want to run Elasticsearch as a service, which we can do by setting these defaults:

sudo update-rc.d elasticsearch defaults 95 10

And finally, to start Elasticsearch for the first time (it will automatically start on reboot), we can use:

sudo /etc/init.d/elasticsearch start

Tweaking a Few Settings

There are a few settings we need to change to get Elasticsearch up and running. The first is necessary to use ES from the host OS. The others are optional; I find them very helpful for local development, but be careful using them in a production deployment; read the docs!

Network Binding

In order to connect to ES from the host OS, we will need to configure the network binding. The networking defaults changed in 2.0, and so now, in order to connect to ES at localhost:9200 from the host operating system, we will need (at least) one of the following two lines in our bootstrap.sh file:

# either of the next two lines is needed to be able to access "localhost:9200" from the host os
sudo echo "network.bind_host: 0" >> /etc/elasticsearch/elasticsearch.yml
sudo echo "network.host: 0.0.0.0" >> /etc/elasticsearch/elasticsearch.yml

Dynamic Scripting

I sometimes like to use dynamic scripting, and while it can be a security risk if your Elasticsearch cluster is not secured behind an adequate authorization system, if used carefully scripting can be a powerful tool. The settings needed to enable dynamic scripting also changed in ES 2.0. Here are the settings we will need to enable dynamic scripting in our Elasticsearch VM:

# enable dynamic scripting
sudo echo "script.inline: on" >> /etc/elasticsearch/elasticsearch.yml
sudo echo "script.indexed: on" >> /etc/elasticsearch/elasticsearch.yml

Enable CORS

I do a lot of work with JavaScript, and so I often need my Elasticsearch cluster to have CORS enabled. The first thing to do is to allow CORS, which we will set in our bootstrap.sh file as follows:

sudo echo "http.cors.enabled: true" >> /etc/elasticsearch/elasticsearch.yml

We also need to set allowed-origin. The default setting changed in ES 2.0, so we will need to set that. For a local development VM like we are building here, it’s probably okay to allow any origin, but in production you would want to be much more careful.

Note that the example in the ES docs sets:
http.cors.allow-origin: /https?:\/\/localhost(:[0-9]+)?/
which only allows JavaScript connections from localhost.

You almost certainly don’t want to do this in production:

sudo echo "http.cors.allow-origin: /https?:\/\/.*/" >> /etc/elasticsearch/elasticsearch.yml

After changing all these settings, we will need to restart the cluster:

sudo /etc/init.d/elasticsearch restart

Putting It All Together

Here is the full bootstrap file:

bootstrap.sh:
#!/usr/bin/env bash
sudo apt-get update
sudo apt-get upgrade
# install openjdk-7 
sudo apt-get purge openjdk*
sudo apt-get -y install openjdk-7-jdk
# install ES
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
sudo apt-get update && sudo apt-get install elasticsearch
sudo update-rc.d elasticsearch defaults 95 10
sudo /etc/init.d/elasticsearch start
# either of the next two lines is needed to be able to access "localhost:9200" from the host os
sudo echo "network.bind_host: 0" >> /etc/elasticsearch/elasticsearch.yml
sudo echo "network.host: 0.0.0.0" >> /etc/elasticsearch/elasticsearch.yml
# enable dynamic scripting
sudo echo "script.inline: on" >> /etc/elasticsearch/elasticsearch.yml
sudo echo "script.indexed: on" >> /etc/elasticsearch/elasticsearch.yml
# enable cors (to be able to use Sense)
sudo echo "http.cors.enabled: true" >> /etc/elasticsearch/elasticsearch.yml
sudo echo "http.cors.allow-origin: /https?:\/\/.*/" >> /etc/elasticsearch/elasticsearch.yml
sudo /etc/init.d/elasticsearch restart

Setting Up

To get the VM up and running, make sure you have VirtualBox and Vagrant installed, then simply save the two files presented here, Vagrantfile and bootstrap.sh, to an otherwise empty directory, then go to that directory in a terminal and type:

vagrant up

Enjoy!

Categorized under: