Tutorial Series: How to Maximize Elasticsearch Indexing Performance
This series covers Elasticsearch Indexing. It specifically focuses on tuning Elasticsearch to achieve maximum indexing throughput and to reduce monitoring and management load.
This post is part 1 of a 3-part series about tuning Elasticsearch Indexing. This series focuses specifically on tuning Elasticsearch to achieve maximum indexing throughput and reduce monitoring and management load.
As a starting point, assume that you start Elasticsearch, create an index, and feed it with JSON documents without incorporating schemas. Elasticsearch will then iterate over each indexed field of the JSON document, estimate its field, and create a respective mapping. While this may seem ideal, Elasticsearch mappings are not always accurate. If, for example, the wrong field type is chosen, indexing errors will occur.
This post is part 2 of a 3-part series about tuning Elasticsearch Indexing. Part 1 can be found here.
The tutorial series focuses specifically on tuning elasticsearch to achieve maximum indexing throughput and reduce monitoring and management load. Elasticsearch is near-realtime, in the sense that when you index a document, you need to wait for the next refresh for that document to appear in search.
Refreshing is an expensive operation and that is why it’s made at a regular interval (default), instead of after each indexing operation. If you are planning to index a lot of documents and you don’t need the new information to be immediately available for search, you can optimize for indexing performance over search performance by decreasing refresh frequency until you are done indexing.
This tutorial series focuses specifically on tuning elasticsearch to achieve maximum indexing throughput and reduce monitoring and management load.
Elasticsearch provides sharding and replication as the recommended way for scaling and increasing availability of an index. A little over allocation is good but a bazillion shards is bad. It is difficult to define what constitutes too many shards, as it depends on their size and how they are being used. A hundred shards that are seldom used may be fine, while two shards experiencing very heavy usage could be too many. Monitor your nodes to ensure that they have enough spare capacity to deal with exceptional conditions.