Qbox to join forces with Instaclustr. Read about it on our blog post here.

TTL values (document expiration)

Document-level TTL (time to live) values in Elasticsearch are generally considered to be inefficient, especially on large datasets. This is due to Elasticsearch needing to continually check documents for expiration in the background. Elasticsearch version 2.x depricated TTL and it has been removed in version 5.x.

If you have an application in which document expiration is needed, our suggestion is to group data into indices with timestamps (i.e. “rolling” indices). Logstash, for example, creates a new index each day by default to group time series data into manageable chunks. This is achieved simply by indexing to a dynamic index name such as “logstash-%{+YYY.MM.dd}”, which is the Logstash default.

With a “rolling” index setup, queries can still be performed across the entire collection of indices, with the added benefit of being able to destroy whole indices past a certain age. Index deletion, in contrast to individual document TTL values, will cause far less strain on the cluster.

Elasticsearch Reference: Index API, TTL

Elasticsearch Reference: Mapping TTL Field