This tutorial introduces Moloch and how to use it in conjunction with Elasticsearch. Moloch is an open source piece of software that can be used to index very large PCAP files into Elasticsearch. Moloch is a project which began at AOL. You can find the source code here: https://github.com/aol/moloch.

Moloch consists of four different parts: A web interface or viewer, a capture application which was written in C, a datastore which is Elasticsearch, and a REST API. The web interface is used to view the PCAP files or network traffic indexed into Elasticsearch. Moloch was designed, with performance in mind, to be able to handle very large sets of data. Moloch is fast and can scale upwards, which is helpful if you have many server resources to allocate to a Moloch cluster.

Installing Moloch

To use Moloch, start by cloning it from github.

$ git clone https://github.com/aol/moloch

Now, install the dependencies for Moloch. I prefer to run this before installing anything new on Debian or Ubuntu.

$ sudo apt-get update && sudo apt-get upgrade

Now that everything is up to date, install the dependencies.

$ sudo apt-get install wget curl libpcre3-dev uuid-dev libmagic-dev pkg-config g++ flex bison zlib1g-dev libffi-dev gettext libgeoip-dev make libjson-perl libbz2-dev libwww-perl libpng-dev xz-utils libffi-dev

Now, install Elasticsearch. Moloch works with the latest stable version of Elasticsearch, which at the time of writing is Elasticsearch version 2.3.3.

I usually take the .deb file from the official Elasticsearch website. Remember to verify the checksum, not just to see that your download hasn’t been corrupted, but also for security reasons. Yes, the download happens over SSL, but remember that security bests works in layers.

$ wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.3.3/elasticsearch-2.3.3.deb<a href="https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.3.3/elasticsearch-2.3.3.deb"></a>

Now, verify the download. Download the SHA1, too, so you can view it on the server.

$ wget <a href="https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.3.3/elasticsearch-2.3.3.deb.sha1"></a><a href="https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.3.3/elasticsearch-2.3.3.deb.sha1"></a>https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.3.3/elasticsearch-2.3.3.deb.sha1

Now, verify the SHA1 value:

$ cat elasticsearch-2.3.3.deb.sha1 
8385dda7aa3870a3b13fc1df91d8b750d940700e

Now you can verify the .deb file that was just downloaded:

$ sha1sum elasticsearch-2.3.3.deb
8385dda7aa3870a3b13fc1df91d8b750d940700e  elasticsearch-2.3.3.deb

Now the SHA1 hash of the file matches what we see on the Elasticsearch website. Go ahead and install it.

$ sudo dpkg -i elasticsearch-2.3.3.deb

If you started the Elasticsearch service, stop it quickly. The Elasticsearch service should be stopped in order to install Moloch.

$ sudo  service elasticsearch stop

It will help to use screen or tmux during the install so you can do things in another terminal while the install script is running. For this example I am using screen.

Open up a new screen session with the following. Remember to open several screens in your session. We can start a new screen session with:

$ screen -S molochsession

For this tutorial, I assume that you are going to install Moloch on a single host, in other words not in clustering mode. If you look in the code for Moloch that you just cloned from github, you will see there is a script used to configure and install Moloch on a single host. Run the following script:

$ sudo ./easybutton-singlehost.sh

During the install, the script will prompt you with a few important questions. The most important question is regarding the amount of memory to allocate to Elasticsearch. You can give it half the amount of memory that you have on the box. If the box has 32GB of memory available then tell the script to give Elasticsearch 16GB of memory to use.

The script will also ask you which interface Moloch should be listening on. You can use Moloch to intercept traffic, index, and analyze the traffic live. You don’t need to only use PCAP’s, although I mostly use Moloch to index PCAP files.

At some point during the install Moloch will try to connect to Elasticsearch. In another screen you can start Elasticsearch while the script is running with:

$ sudo service elasticsearch start

Moloch will start once you have run the install script. At boot, Moloch does not come with a script to automatically start it. You will have to create your own Moloch service script. I found one online that I slightly adapted. This is the startscript that I use:

#!/bin/bash
cd /data/moloch/bin
/bin/rm -f /data/moloch/capture.log.old
/bin/mv /data/moloch/logs/capture.log /data/moloch/logs/capture.log.old
/data/moloch/bin/moloch-capture -c /data/moloch/etc/config.ini > /data/moloch/logs/capture.log 2>&1 &
sleep 25
# Start moloch viewer process
cd /data/moloch/viewer
/bin/rm -f /data/moloch/logs/viewer.log.old
/bin/mv /data/moloch/logs/viewer.log /data/moloch/logs/viewer.log.old
export NODE_ENV=production
exec /data/moloch/bin/node /data/moloch/viewer/viewer.js -c /data/moloch/etc/config.ini > /data/moloch/logs/viewer.log 2>&1 &

You can test this start script by rebooting your server and starting Elasticsearch.

$ sudo service elasticsearch start

I then run the start script:

$ sudo ./start_moloch.sh

Make sure that Moloch started. You can do this by checking if something is running on port 8005. I don’t recommend running Moloch on the public internet, so if you plan on running it on a VPS then make sure to lock it down and restrict access to Moloch only on localhost.

$ sudo lsof -i | grep 8005
node     1761          daemon   11u  IPv4  11125      0t0  TCP *:8005 (LISTEN)

To make this script run at every startup, add it to your cronjobs and make the script run at @reboot. There are other ways you can make this start script run at boot too, but I’m not going to go into that now.

We now want to access the Moloch web interface or viewer. I like to run Moloch on localhost only since I won’t be doing any capturing from an interface, and to restrict access to the web interface only from localhost.

I then use SSH local forwarding and forward port 8005 from my remote server to localhost 8005 on my personal computer. You can do the SSH forwarding with:

$ ssh -f -N -L 8005:127.0.0.1:8005 timo@172.20.0.203

Go ahead and open up the web interface at https://localhost:8005/. Now you should accept the self signed SSL certificate before you continue. The username: admin and password: admin will help you get into the web interface.

Learn about Qbox’s new open source software Supergiant.io

Acquire a publicly available PCAP file that you can import and play around with. Also, create a directory to hold the PCAP files on your server to help stay organized. Moloch allows you to import pcap by pcap or even a whole directory of PCAP’s at once.

$ mkdir ~/pcaps
$ cd ~/pcaps

You can find a big list of pcaps that are available to the public to download here: http://www.netresec.com/?page=PcapFiles#iscx

Download a few PCAP files. Let’s start with this PCAP that I found from an infosec CTF competition:

$ wget http://shell-storm.org/repo/CTF/CSAW-2011/Networking/Networking101%20-%20100%20Points/capture.pcap

I already have quite a few PCAP files on my server, but for this example I only want to index the capture from the CSAW 2011 CTF into Elasticsearch, so I am going index only one PCAP, and not the whole directory. It is usually not a good idea, from a security point of view, to use sudo su. You can use sudo and the command instead, but just for demonstration purposes I will switch to root and run everything in root.

$ sudo su
# cd /data/moloch/bin
# ./moloch-capture -r /home/timo/pcaps/capture.pcap

Open up Moloch’s viewers. You should see something on the graph. You can access the viewer and view all data at: https://localhost:8005/?date=-1

Perhaps you have a directory full of PCAP’s that you would like to index. This is how you would index per directory, the -R in below command is for “recursive”.

# ./moloch-capture -R /home/timo/pcaps/

At this point I hope you are using screen or tmux. Open another screen and run top or htop. It is interesting to see the CPU and memory usage of the server when indexing large data sets. I don’t have a very big server, but this is what it looks like on my server when I run htop while indexing a directory with some large PCAP’s.

htop.png#asset:1037

Moloch offers the ability to look at the data from the PCAP files in different ways. One interesting feature is a view that shows the data on a map, which maps the IP’s to physical location.

map_view_final.png#asset:1040

Moloch also allows you to see the relationship between different IP’s, even on an internal network level, which is extremely interesting. This feature is useful in different contexts. For example, you can use this to see which hosts are making the same type of connection to a known malicious host.

moloch_relationship_view.png#asset:1038

Another great feature is the search engine functionality. Moloch uses Elasticsearch as a datastore which allows you to quickly search over data. This is very useful for security related investigations, for example, if you are looking at PCAP files of botnet related traffic on a network, or maybe you would like to search for dns traffic that fits a certain criteria.

Supergiant Tutorial: How to Install Supergiant on AWS EC2

Look for requests to port 53 that only send one packet by searching for: “port.dst == 53 && packets == 1”. This could yield interesting results, however, don’t think that all traffic will be dns because it is destined for port 53.

53_single_packet.png#asset:1036

Moloch has some built in functionality in the viewer to help you filter over different types of network traffic, and to filter by specific properties. For example, you can filter network traffic by type “http” and then filter by “URL”. Have a look at the “SPI View” in Moloch’s viewer. Make use of the “snap to” functionality when selecting a date in the SPI View. The SPI View is resource intensive and won’t work if you view “All” your data at once. In this example I am filtering http traffic by the user agent.

SPI_VIEW_User_Agent.png#asset:1039

Conclusion

Enjoy using Moloch and use it responsibly; it is a very powerful tool. Make sure to change the default password on Moloch and to add non admin users. Also read over the documentation on the project’s wiki on github and make sure to lock down your Moloch even if you are not going to expose it to the public internet.