A Vagrant Virtual Machine for Elasticsearch 2.x
Posted by Sloan Ahrens February 2, 2016Elasticsearch 2.0.0 introduced a number of breaking changes. When I set out to install ES 2.0 to do some local testing, I found that the techniques that I had been using to set up virtual machines for local development (for instance, here) were no longer adequate. So I set out to discover what the “proper” method should be, and along the way I ran into a few problems. I’ll outline those issues here, hopefully saving some other people a little bit of trouble.
The instructions that follow will assume you are using OSX. It should be straightforward to adapt them to other operating systems, but I will not address those considerations here.
Vagrant
In earlier posts I used full Ubuntu virtual machines (with UI), as this was my preferred method of development. Recently I have more often used headless VMs with Vagrant, and for many purposes I find this to be more useful. I find that running Elasticsearch in a Vagrant VM works nicely for local development, allowing me to sandbox the Elasticsearch installation, but port-forwarding means I can easily access ES as if it were running in my host OS.
To follow along, you will need Vagrant installed, which can be found here. Vagrant requires a provider, and I find VirtualBox to be convenient. You can find VirtualBox here.
Vagrantfile
The Vagrantfile we’ll use for our Elasticsearch VM is quite simple:
Vagrantfile: Vagrant.configure(2) do |config| config.vm.box = "ubuntu/trusty64" config.vm.provision :shell, path: "bootstrap.sh" config.vm.network :forwarded_port, host: 9200, guest: 9200 config.vm.provider "virtualbox" do |v| v.memory = 2048 v.cpus = 1 v.name = "ES-2.x" end end
A few things to note here:
- This VM uses the “
ubuntu/trusty64
” box, which runs Ubuntu Server 14.04 LTS. If you want to use a different box, you can find some here, or create your own. - The “
bootstrap.sh
” file is where our commands for installing Elasticsearch will reside; we’ll get to that in a minute. - We are forwarding port
9200
from the guest OS to the host OS, so we can access Elasticsearch atlocalhost:9200
just like we could if it were running in the host OS. - This is not a particularly beefy virtual machine, with a single CPU and 2GB of RAM. Depending on your application you may want to change these values.
- The VM name “
ES-2.x
” is arbitrary; feel free to change it to whatever you want.
Next we’ll take a look at the bootstrap script. I’ll explain it piece by piece, but I’ve included the full file below for convenience.
Installing Elasticsearch
Elasticsearch requires at least Java 7. I like to use OpenJDK, mainly because I can install it without any prompts, which is convenient for a Vagrant setup.
Here are the commands we’ll use to install Java 7:
# install openjdk-7 sudo apt-get purge openjdk* sudo apt-get -y install openjdk-7-jdk
The Elasticsearch documentation provides a number of different ways to install Elasticsearch, and run it as a service. You can take a look here and here.
We will use the debian/ubuntu package. To do so we’ll need to add the correct repository, as described here.
This command adds the Elasticsearch repository Public Signing Key:
wget -qO - <a href="https://packages.elastic.co/GPG-KEY-elasticsearch">https://packages.elastic.co/GPG-KEY-elasticsearch</a> | sudo apt-key add -
This command adds the repository (see the explanation here for why we need to use the echo method):
echo "deb <a href="http://packages.elastic.co/elasticsearch/2.x/debian">http://packages.elastic.co/elasticsearch/2.x/debia...</a> stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
Then we can update Aptitude and install Elasticsearch with:
sudo apt-get update && sudo apt-get install elasticsearch
We want to run Elasticsearch as a service, which we can do by setting these defaults:
sudo update-rc.d elasticsearch defaults 95 10
And finally, to start Elasticsearch for the first time (it will automatically start on reboot), we can use:
sudo /etc/init.d/elasticsearch start
Tweaking a Few Settings
There are a few settings we need to change to get Elasticsearch up and running. The first is necessary to use ES from the host OS. The others are optional; I find them very helpful for local development, but be careful using them in a production deployment; read the docs!
Network Binding
In order to connect to ES from the host OS, we will need to configure the network binding. The networking defaults changed in 2.0, and so now, in order to connect to ES at localhost:9200
from the host operating system, we will need (at least) one of the following two lines in our bootstrap.sh file
:
# either of the next two lines is needed to be able to access "localhost:9200" from the host os sudo echo "network.bind_host: 0" >> /etc/elasticsearch/elasticsearch.yml sudo echo "network.host: 0.0.0.0" >> /etc/elasticsearch/elasticsearch.yml
Dynamic Scripting
I sometimes like to use dynamic scripting, and while it can be a security risk if your Elasticsearch cluster is not secured behind an adequate authorization system, if used carefully scripting can be a powerful tool. The settings needed to enable dynamic scripting also changed in ES 2.0. Here are the settings we will need to enable dynamic scripting in our Elasticsearch VM:
# enable dynamic scripting sudo echo "script.inline: on" >> /etc/elasticsearch/elasticsearch.yml sudo echo "script.indexed: on" >> /etc/elasticsearch/elasticsearch.yml
Enable CORS
I do a lot of work with JavaScript, and so I often need my Elasticsearch cluster to have CORS enabled. The first thing to do is to allow CORS, which we will set in our bootstrap.sh
file as follows:
sudo echo "http.cors.enabled: true" >> /etc/elasticsearch/elasticsearch.yml
We also need to set allowed-origin
. The default setting changed in ES 2.0, so we will need to set that. For a local development VM like we are building here, it’s probably okay to allow any origin, but in production you would want to be much more careful.
Note that the example in the ES docs sets:
http.cors.allow-origin: /https?:\/\/localhost(:[0-9]+)?/
which only allows JavaScript connections from localhost
.
You almost certainly don’t want to do this in production:
sudo echo "http.cors.allow-origin: /https?:\/\/.*/" >> /etc/elasticsearch/elasticsearch.yml
After changing all these settings, we will need to restart the cluster:
sudo /etc/init.d/elasticsearch restart
Putting It All Together
Here is the full bootstrap file:
bootstrap.sh: #!/usr/bin/env bash sudo apt-get update sudo apt-get upgrade # install openjdk-7 sudo apt-get purge openjdk* sudo apt-get -y install openjdk-7-jdk # install ES wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list sudo apt-get update && sudo apt-get install elasticsearch sudo update-rc.d elasticsearch defaults 95 10 sudo /etc/init.d/elasticsearch start # either of the next two lines is needed to be able to access "localhost:9200" from the host os sudo echo "network.bind_host: 0" >> /etc/elasticsearch/elasticsearch.yml sudo echo "network.host: 0.0.0.0" >> /etc/elasticsearch/elasticsearch.yml # enable dynamic scripting sudo echo "script.inline: on" >> /etc/elasticsearch/elasticsearch.yml sudo echo "script.indexed: on" >> /etc/elasticsearch/elasticsearch.yml # enable cors (to be able to use Sense) sudo echo "http.cors.enabled: true" >> /etc/elasticsearch/elasticsearch.yml sudo echo "http.cors.allow-origin: /https?:\/\/.*/" >> /etc/elasticsearch/elasticsearch.yml sudo /etc/init.d/elasticsearch restart
Setting Up
To get the VM up and running, make sure you have VirtualBox and Vagrant installed, then simply save the two files presented here, Vagrantfile
and bootstrap.sh
, to an otherwise empty directory, then go to that directory in a terminal and type:
vagrant up
Enjoy!