This post is Part 3 of a 3-part series about tuning Elasticsearch Search. Part 1 can be found here, and Part 2 can be found here. The aim of this tutorial is to further talk about some Search Tuning techniques, strategies, and recommendations specific to Elasticsearch 5.0 or onward.

Elasticsearch 5.0.0 had really been a major release after Elasticsearch 2.x version, and it does have something for everyone. It is a part of a wider release of the Elastic Stack that lines up version numbers of all the stack products. Kibana, Logstash, Beats, Elasticsearch are all version 5.0 now. It is the fastest, safest, most resilient, easiest to use version of Elasticsearch ever, and it comes with a boatload of enhancements and new features.

For this post, we will be using hosted Elasticsearch on Qbox.io. You can sign up or launch your cluster here, or click "Get Started" in the header navigation. If you need help setting up, refer to "Provisioning a Qbox Elasticsearch Cluster."

We have already discussed the “The Authoritative Guide to Elasticsearch Performance Tuning” in a three-part tutorial series to introduce some general tips and methods for performance tuning, explaining at each step the most relevant system configuration settings and metrics. We have also discussed “How to Maximize Elasticsearch Indexing Performance” in a three part tutorial series to introduce some general tips and methods to achieve maximum indexing throughput and reduce monitoring and management load.  

Let us talk further about some Search Tuning techniques, strategies and recommendations specific to Elasticsearch 5.0 or onwards.

Search Rounded Dates

Queries on date fields that use now are typically not cacheable because the range that is being matched changes all the time. However, switching to a rounded date is often acceptable in terms of user experience, and it has the benefit of making better use of the query cache.

For instance the below query:

curl -XPUT 'ES_HOST:ES_PORT/index/type/1?pretty' -H 'Content-Type: application/json' -d '{
 "my_date": "2016-05-11T16:30:55.328Z"
}'
curl -XGET 'ES_HOST:ES_PORT/index/_search?pretty' -H 'Content-Type: application/json' -d '{
 "query": {
   "constant_score": {
     "filter": {
       "range": {
         "my_date": {
           "gte": "now-1h",
           "lte": "now"
         }
       }
     }
   }
 }
}'

could be replaced with the following query:

curl -XGET 'ES_HOST:ES_PORT/index/_search?pretty' -H 'Content-Type: application/json' -d '{
 "query": {
   "constant_score": {
     "filter": {
       "range": {
         "my_date": {
           "gte": "now-1h/m",
           "lte": "now/m"
         }
       }
     }
   }
 }
}'

In that case we rounded to the minute, so if the current time is 16:31:29, the range query will match everything whose value of the my_date field is between 15:31:00 and 16:31:59. And if several users run a query that contains this range in the same minute, the query cache could help speed things up a bit. The longer the interval that is used for rounding, the more the query cache can help, but beware that too aggressive rounding might also hurt user experience.

It might be tempting to split ranges into a large cacheable part and smaller not-cacheable parts to be able to leverage the query cache, as shown below:

curl -XGET 'ES_HOST:ES_PORT/index/_search?pretty' -H 'Content-Type: application/json' -d '{
 "query": {
   "constant_score": {
     "filter": {
       "bool": {
         "should": [
           {
             "range": {
               "my_date": {
                 "gte": "now-1h",
                 "lte": "now-1h/m"
               }
             }
           },
           {
             "range": {
               "my_date": {
                 "gt": "now-1h/m",
                 "lt": "now/m"
               }
             }
           },
           {
             "range": {
               "my_date": {
                 "gte": "now/m",
                 "lte": "now"
               }
             }
           }
         ]
       }
     }
   }
 }
}'

However, such practice might make the query run slower in some cases because the overhead introduced by the bool query may defeat the savings from better leveraging the query cache.

Warm Up Global Ordinals

The default behavior of Elasticsearch is to load in-memory fielddata lazily. The first time Elasticsearch encounters a query that needs fielddata for a particular field, it will load that entire field into memory for each segment in the index.

For small segments, this requires a negligible amount of time. But if you have a few 5 GB segments and need to load 10 GB of fielddata into memory, this process could take tens of seconds. Users accustomed to subsecond response times would suddenly be hit by an apparently unresponsive website.

There are three methods to combat this latency spike:

  • Eagerly load fielddata

  • Eagerly load global ordinals

  • Prepopulate caches with warmers

All are variations on the same concept: preload the fielddata so that there is no latency spike when the user needs to execute a search.

Note: Ordinals are only built and used for strings. Numerical data (integers, geopoints, dates, etc) doesn’t need an ordinal mapping, since the value itself acts as an intrinsic ordinal mapping. Therefore, you can only enable eager global ordinals for string fields.

Global ordinals are a data structures used to run terms aggregations on keyword fields. They are loaded lazily in memory because Elasticsearch does not know which fields will be used in terms aggregations and which fields won’t. We can tell Elasticsearch to load global ordinals eagerly at refresh time by configuring mappings as described below:

curl -XPUT 'ES_HOST:ES_PORT/index?pretty' -H 'Content-Type: application/json' -d '{
 "mappings": {
   "type": {
     "properties": {
       "foo": {
         "type": "keyword",
         "eager_global_ordinals": true
       }
     }
   }
 }
}'

Eager building of global ordinals can have an impact on the real-time aspect of our data. For very high cardinality fields, building global ordinals can delay a refresh by several seconds. The choice is between paying the cost on each refresh, or on the first query after a refresh. If you index often and query seldom, it is probably better to pay the price at query time instead of on every refresh.

We can make our global ordinals pay for themselves. If you have very high cardinality fields that take seconds to rebuild, increase the refresh_interval so that global ordinals remain valid for longer. This will also reduce CPU usage, as you will need to rebuild global ordinals less often.

Warm Up the Filesystem Cache

Elasticsearch, by default, completely relies on the operating system file system cache for caching I/O operations. It is possible to set index.store.preload in order to tell the operating system to load the content of hot index files into memory upon opening. This setting accept a comma-separated list of files extensions: all files whose extension is in the list will be pre-loaded upon opening. This can be useful to improve search performance of an index, especially when the host operating system is restarted because this causes the file system cache to be trashed. Note, however, that this may slow down the opening of indices because they will become available only after data have been loaded into physical memory.

This setting is best-effort only and may not work at all depending on the store type and host operating system.

The index.store.preload is a static setting that can either be set in the config/elasticsearch.yml:

index.store.preload: ["nvd", "dvd"]

or in the index settings at index creation time:

curl -XPUT 'ES_HOST:ES_PORT/my_index' -d ‘{
  "settings": {
    "index.store.preload": ["nvd", "dvd"]
  }
}’

The default value is the empty array, which means that nothing will be loaded into the file-system cache eagerly. For indices that are actively searched, you might want to set it to ["nvd", "dvd"], which will cause norms and doc values to be loaded eagerly into physical memory. These are the two first extensions to look at because  Elasticsearch performs random access on them.

If the machine running Elasticsearch is restarted, the filesystem cache will be empty, so it will take some time before the operating system loads hot regions of the index into memory so that search operations are fast. Thus, it makes sense to explicitly tell the operating system which files should be loaded into memory eagerly depending on the file extension using the index.store.preload setting.

Loading data into the filesystem cache eagerly on too many indices or too many files will make search slower if the filesystem cache is not large enough to hold all the data. This setting can be dangerous on indices that are larger than the size of the main memory of the host because it would cause the filesystem cache to be trashed upon reopens after large merges, which would make indexing and searching slower. Use with caution.

Related Helpful Resources

Give it a Whirl!

It's easy to spin up a standard hosted Elasticsearch cluster on any of our 47 Rackspace, Softlayer, or Amazon data centers. And you can now provision your own AWS Credits on Qbox Private Hosted Elasticsearch

Questions? Drop us a note, and we'll get you a prompt response.

Not yet enjoying the benefits of a hosted ELK-stack enterprise search on Qbox? We invite you to create an account today and discover how easy it is to manage and scale your Elasticsearch environment in our cloud hosting service.

comments powered by Disqus