According to Sense’s README on github, Sense is: “A JSON aware developer’s interface to Elasticsearch. Comes with handy machinery such as syntax highlighting, autocomplete, formatting and code folding.”

Sense is a web application used to interact with Elasticsearch’s REST API, and it makes interacting with the Elasticsearch API much easier. Sense has features such as a shorthand notation that makes writing queries for Elasticsearch much easier. It also has autocomplete built-in syntax validation and json collapsing that allow you to fold away parts of a JSON query when the query gets too big.

Sense is useful because you can have multiple queries typed out in Sense, but you can choose to only run one query at a time, as opposed to using curl on the command line where viewing your old queries isn’t that simple especially if the queries are long. Developers like tools that make their job easier, and Sense is definitely a tool aimed at developers.

What  are your alternatives to using Sense? You could use curl on the command line, your browser, or even a manual request editor such as Burp Suite or fiddler, but I would suggest using curl or Sense.

How to Install Sense?

Sense has gone through several phases in development, from being a for-purchase piece of software to where it is today an open-sourced Kibana plugin.  You will need Kibana installed to be able to install Sense.

Navigate to where you installed Kibana, and find the script and run:

$ ./bin/kibana plugin --install elastic/sense

You also need to start Kibana in order to get Sense running:

$ ./bin/kibana

If you are on a network that only allows connections to the internet via an http proxy, then you will need to download the plugin manually and install it with kibana manually.

First download sense, make sure https_proxy is set:

$ echo $https_proxy
http://172.20.1.12:8080
$ wget https://download.elasticsearch.org/elastic/sense/sense-latest.tar.gz

Now install it manually. In this example we downloaded the plugin to the /tmp directory

$ bin/kibana plugin -i sense -u file:///tmp/sense-latest.tar.gz

Accessing Sense on a Remote Server

Usually Sense only runs on localhost at: http://localhost:5601/app/sense. However, you can make it run on any IP. If you are running Elasticsearch on a public facing server, then you might only want to run it on localhost until you’ve locked down access to your machine.

You can still access the web application that is running on localhost on the remote server via your browser. You just have to SSH into the machine and request local forwarding. You can do so with:

$ ssh -f -N -L an unused port on your local pc:127.0.0.1:5601 [user]@[ip of your server]
$ ssh -f -N -L 5601:127.0.0.1:5601 sysadmin@123.123.123.123

You can now access sense in your browser by navigating to:

http://localhost:5601/app/sense

What is the Elasticsearch REST API?

Almost all interaction with Elasticsearch happens via the Elasticsearch REST API. Even if you are making use of Kibana, you are indirectly interacting with the Elasticsearch REST API.

Sense Shorthand Notation

Before we get into the details of working with the REST full API, have a look at the shorthand notation that is used by sense. If you are used to working with Elasticsearch, then you might be familiar with using curl to interact with Elasticsearch’s REST API.

To check if Elasticsearch is running healthy, run the following:

$ curl 'http://localhost:9200?pretty=true'
{
  "name": "Asmodeus",
  "cluster_name": "elasticsearch",
  "version": {
    "number": "2.3.1",
    "build_hash": "bd980929010aef404e7cb0843e61d0665269fc39",
    "build_timestamp": "2016-04-04T12:25:05Z",
    "build_snapshot": false,
    "lucene_version": "5.5.0"
  },
  "tagline": "You Know, for Search"
}

Add this to your copy/paste buffer, then click on the Sense UI on the left where user input is accepted. If you click paste, then it shows the shorthand notation of the above curl command.

The equivalent shorthand notation for this in Sense is:

GET /

HTTP Verbs and Meaning

Make yourself familiar with all the different HTTP verbs and what they are used for. It is also a good idea to make yourself familiar with curl and advanced curl usage such as loading JSON from a file and then sending the JSON from file in a request.

It is worth it to read more about the different HTTP verbs available, in the context of Elasticsearch the following HTTP verbs are used: PUTDELETEPOST and GET. PUT is mostly used to create or update indices, where DELETE is used to DELETE. The difference between POST and GET in the context of Elasticsearch is not always so obvious.

Something Useful with Sense

Let’s start out by creating an index.

PUT /random_index?pretty=true
{
  "acknowledged": true
}

Elasticsearch accepts the optional argument ?pretty with the optional value of true. Pretty defaults to true, so the output of passing the argument without the true value will give the same output.

PUT /random_index?pretty

If you ran this command again, it would give you an error because you can’t create the same index twice. So let’s delete the index so we can create it again.

DELETE /random_index?pretty
{
  "acknowledged": true
}

This gives the output which is now beautified, because of the pretty argument:

{
 "acknowledged": true
}

Now we create the index again with:

PUT /random_index?pretty
{
  "acknowledged": true
}

Now let’s index some actual data. American politics interests me, so let’s use election candidates as our example data.

PUT /candidate_index/candidate/1?pretty=true
{
 "name" : "Donald John Trump",
 "affiliation" : "Republican",
 "age" : "69",
 "occupation" : "businessman",
 "twitter" : "https://twitter.com/realDonaldTrump",
 "website" : "http://www.donaldjtrump.com"
}
{
  "_index": "candidate_index",
  "_type": "candidate",
  "_id": "1",
  "_version": 1,
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "created": true
}

This table describes similarities between Elasticsearch and relational databases.

Database    Table      Row          Column
Index       Type       Documents    Field

From the previous example we created an index candidate_index in which the type name was candidate and our candidate ID was one. So we created candidate 1. Let’s verify everything.

List the indexes. On Sense you sometimes won’t see that much of a difference when using pretty=true. Try it when using curl on the command line, though, and you will see a big difference in the formatting of your output.

GET _cat/indices?pretty=true
yellow open .kibana         1 1 1 0 3.1kb 3.1kb 
yellow open random_index    5 1 0 0  260b  260b 
yellow open candidate_index 5 1 2 0 9.1kb 9.1kb

We can have a look at candidate 1 that we just created by running:

GET /candidate_index/candidate/1?pretty=true
{
  "_index": "candidate_index",
  "_type": "candidate",
  "_id": "1",
  "_version": 3,
  "found": true,
  "_source": {
    "name": "Donald John Trump",
    "affiliation": "Republican",
    "age": "69",
    "occupation": "businessman",
    "twitter": "https://twitter.com/realDonaldTrump"
    "website": "http://www.donaldjtrump.com"   
 }
}

Now let’s create candidate 2.

PUT /candidate_index/candidate/2?pretty=true
{
 "name" : "Hillary Clinton",
 "affiliation" : "Democratic",
 "age" : "68",
 "occupation" : "politician",
 "twitter" : "https://twitter.com/HillaryClinton",
 "website" : "https://www.hillaryclinton.com"
}
{
  "_index": "candidate_index",
  "_type": "candidate",
  "_id": "2",
  "_version": 1,
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "created": true
}

We only created two candidates, but imagine that we created 1000 candidates. We suddenly feel like searching candidates that have experience in the world of politics. This is what our query would look like:

GET /candidate_index/_search?q=politician&pretty=true
{
  "took": 71,
  "timed_out": false, 
 "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 0.095891505,
    "hits": [
      {
        "_index": "candidate_index",
        "_type": "candidate",
        "_id": "2",
        "_score": 0.095891505,
        "_source": {
          "name": "Hillary Clinton",
          "affiliation": "Democratic",
          "age": "68",
          "occupation": "politician",
          "twitter": "https://twitter.com/HillaryClinton",
          "website": "https://www.hillaryclinton.com"
        }
      }
    ]
  }
}

Or maybe we would like to search candidates by their political affiliation to find all the Democratic candidates.

This query searches in the candidate_index index and searches for the keyword “democratic,” we then tell Elasticsearch to beautify the JSON response.

GET /candidate_index/_search?q=democratic&pretty=true
{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
  "total": 1,
  "max_score": 0.095891505,
  "hits": [
    {
     "_index": "candidate_index",
     "_type": "candidate",
     "_id": "2",
     "_score": 0.095891505,
     "_source": {
      "name": "Hillary Clinton",
      "affiliation": "Democratic",
      "age": "68",
      "occupation": "politician",
      "twitter": "https://twitter.com/HillaryClinton",
      "website": "https://www.hillaryclinton.com"
     }
    }
   ]
 }
}

Conclusion

I only used two candidates in my examples. You can use more candidates to make your experience using Sense and the Elasticsearch REST API more interesting. You can make up your own examples if you want to understand Elasticsearch even better. You should start using Sense as soon as possible. It will tremendously improve your experience of working with the Elasticsearch REST API.