The new ELK stack 6.6.0 was officially released by Elasticsearch on January 29, 2019, and it offers a lot of groundbreaking features and enhancements for Elasticsearch, Kibana, Logstash, APM, and Beats.

We've already tested Elasticsearch 6.6.0 with the brand new Kibana and are excited to share our experience with such valuable features as Index Lifecycle Management and Remote Cluster management. In this article, we'll summarize these and other major new features for Elasticsearch, Kibana, and Elastic APM and will give you a glimpse of some cool stuff you can now do with your Elasticsearch indices in Kibana 6.6.0. Let's get started!

As always, a major ES release comes with multiple cool features and enhancements. Among those new features, one should point to the index lifecycle management (which we're about to describe), as well as new median absolute deviation aggregation, some new analyzers and dictionaries, new SQL functions, and ML tools. You can find the exhaustive list of all changes in the ES 6.6.0 release notes. In what follows, we'll focus on two important features: index lifecycle management and new agent metrics in Elastic APM.

Index Lifecycle Management

Hot-Warm architecture that allows allocating indexes between nodes depending on the index age, size, and query frequency has been a hot topic in the Elasticsearch community at least since ELK 5.0.

Essentially, the usefulness of the "hot-warm" architecture is motivated by the fact that certain indexes become extremely large in size, less popular, or outdated. It's therefore reasonable to divide an individual index lifecycle into several stages: hot, warm, and cold. Each stage reflects the trade-off of the index importance and costs users are willing to bear to preserve the index.

For example, ES users would like to place their most popular and new indexes on "hot" nodes that support the best read-write performance and replication. The hardware of this node is usually optimized for high performance at peak traffic times.

As the index gets older and grows in size, users would like to move it to the "warm" stage. At this stage, indexes are no longer writable and, hence, should be optimized for read-only operations. For example, users can shrink these indexes in size by reducing the number of shards and replicas or force-merge them down to a single segment for optimized storage and search.

The "cold" stage is essentially the index's winter because "cold" indexes are the ones queried less frequently. Therefore, users can allocate "cold" shards on less performant hardware. Because queries become slower, ES users can reduce the number of index replicas, too.

In the earlier ES versions, users had to do a lot of manual configuration to implement the "hot-warm" architecture. Designing a viable "hot-warm" architecture also required excellent expertise in ES API. Until very recently, the easy setup of the "hot-warm" architecture was only available for Elasticsearch Cloud users.

In the ELK 6.6.0, however, you can easily define index lifecycle policies using Kibana Management UI. In particular, users can specify the timing for each phase and configure how and when to shrink, replicate, force-merge, or delete indexes. We'll discuss this in more detail when we get to the new Kibana 6.6.0 features.

Elasticsearch is planning to extend the "hot-warm" architecture with the concept of "frozen" indices in future indexes. When you "freeze" an index, it's no longer open but remains available for queries in accordance with a special procedure. When the search targets the frozen index, the query fully opens, searches, and then closes the index. Because "frozen" indexes are opened only for the query time, they use fewer memory resources than open indexes. This makes it cheap to manage thousands of legacy indexes on a single node. The only trade-off of "freezing" is the search latency associated with the procedure of opening and closing "frozen" indices.

New Agent Metrics and Distributed Tracing in Elastic AMP

Elastic APM is an application performance monitoring system introduced in the ELK 6.0 release. The APM has the following features:

  • Monitors application performance in real time by collecting data on query response times, database queries, external HTTP requests, calls to caches, etc. Monitoring is executed by APM agents (open source libraries) integrated into the user applications and services.
  • Collects data about unhandled errors and exceptions from applications and services. The APM watches applications' stack traces and identifies errors as they emerge.
  • APM agents send the collected data to the APM server that aggregates them and saves to Elasticsearch indexes for subsequent processing and analysis (see the APM architecture in the image below).


Source: Elastic

The APM 6.6.0 release introduced new agent metrics. The latest version of Elastic agents can now automatically report system and process-level RAM and CPU metrics along with errors and traces. The metrics can be visualized in the new service Metrics tab inside the APM UI in Kibana.

Also, APM 6.6.0 ships with support for Distributed Tracing. All APM agents are now OpenTracing compliant, which means they can automatically trace requests all the way through different microservices inside the microservices applications. The latter are network-connected pieces of software communicating via REST API. APM 6.6.0 UI can visualize these microservices traces in one single view.

What's New in Kibana 6.6.0?

Kibana is the open source visualization and analytics platform designed to work with data stored in the Elasticsearch indexes. A new Kibana 6.6.0 release ships with many useful features and fixes that make its analytics and visualization capabilities even more powerful. You can find an exhaustive list of new Kibana 6.6.0 features here.

Solving Single Point-of-Failure Issue

In Kibana 6.6.0, users can define multiple Elasticsearch nodes that Kibana can access. By allowing requests to target multiple ES nodes in the cluster, this feature solves the challenge of having a single point of failure on the Kibana - Elasticsearch connections. Users can enable the feature by setting the elasticsearch.host in their kibana.yml file to the URLs of the Elasticsearch nodes they would like to use.

Integrating Index Lifecycle Management into Kibana UI (Beta)

Index lifecycle policies discussed above can now be defined in the Kibana dashboard. Users can access this functionality under Management -> Elasticsearch -> Index Lifecycle Management in their Kibana Dashboard (see the image below).

Here, you can activate a specific phase of your index lifecycle (hot, warm, cold) and define specific settings for the phase.

For example, you can define the maximum index size and the maximum age of indexes in the "hot" stage and decide how the "hot" index rolls over to the "warm" stage.

For the "warm" phase, you can shrink the index by reducing the shard size, decrease the number of replicas, or decide to do a force merge of index to reduce the number of segments in your shard. Enabling these policies can ensure faster searches for "warm" indexes.

For each stage after the "hot" one, you can also define whether it starts immediately on the previous stage's rollover or after a certain time after rollover.

After the policy is created, you can add it to an index on Management > Elasticsearch > Index Management. page. This page lists all Elasticsearch indices, which you can filter by lifecycle status and lifecycle phase.

To add a policy, choose the desired index and then select Manage > Add lifecycle policy. You’ll see the policy name, the current phase, the current action, and any errors that occurred while performing that action.

Among other cool Kibana 6.6.0 features to check, we are drawing your attention to the following:

  • Managing Remote clusters. Kibana 6.6.0 added the new UI for managing remote clusters. You can add and remove remote clusters, check their connectivity, and manage them with cross-cluster search and cross-cluster replication.
  • Auto-follow pattern. This feature helps you initiate and manage the remote replication process. You can follow an index pattern on your remote cluster for auto-discovery and then replicate new clusters matching this pattern.
  • Upgrade Assistant for ES 7.0. Kibana's new Upgrade Assistant can help you upgrade from Elasticsearch 6.x to Elasticsearch 7.0. In particular, it helps you identify issues in your cluster and address them before upgrading. You can access this feature under Management > Elasticsearch > Upgrade Assistant in your Kibana dashboard.

For the exhaustive list of all Kibana 6.6.0 features and enhancements, see the official release highlights.

Conclusion

To sum it up, the ELK 6.6.0 has a lot of cool features to offer, and most of these features are available for free. In this article, we only scratched the surface of the most groundbreaking innovations introduced in the ELK 6.6.0. In a subsequent article, we'll focus on the Index Lifecycle Management, showing all steps for creating and enabling the index lifecycle policy for your ES indices. Stay tuned to our blog to find out more!