Qbox to join forces with Instaclustr. Read about it on our blog post here.

As your journey with Elasticsearch progresses you should learn about shards, how they affect your cluster, and also how to work with them.

An index can be broken into pieces that can be spread across nodes on a cluster. This is handy if the size of your index, for example, exceeds the size of a hard drive on one of the nodes in your cluster.  This is conceptually similar to how Mongodb stores data in glusterfs by breaking it into smaller pieces and spreading it across nodes in the cluster.

Related Article: Optimizing Elasticsearch: How Many Shards per Index?

When creating an index, you can define how many pieces you would like to subdivide the index into across nodes. This obviously only makes sense in a multi node cluster environment. The number of shards for an index can be defined when you create an index, but cannot be changed once the index is created.

For this post, we are going to be using hosted Elasticsearch on Qbox.io. You can sign up or launch your cluster here, or click the “Get Started” button in the navigation. If you need help setting up, refer to our post Provisioning a Qbox Elasticsearch Cluster.

When it comes to shards, an important operation is to check the cat health API. The cat health API is a short form version of the cluster health API. You can get a quick overview of your cluster’s health and of the state the shards are in with this command:

$ curl -s 'http://localhost:9200/_cat/health?v=true' 
epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1472048060 16:14:20  elasticsearch yellow          1         1      7   7    0    0        7             0                  -                 50.0%

You can view similar information in a different format by running the following:

$ curl -s 'http://localhost:9200/_cluster/health?pretty=true'
 "cluster_name" : "elasticsearch",
 "status" : "yellow",
 "timed_out" : false,
 "number_of_nodes" : 1,
 "number_of_data_nodes" : 1,
 "active_primary_shards" : 7,
 "active_shards" : 7,
 "relocating_shards" : 0,
 "initializing_shards" : 0,
 "unassigned_shards" : 7,
 "delayed_unassigned_shards" : 0,
 "number_of_pending_tasks" : 0,
 "number_of_in_flight_fetch" : 0,
 "task_max_waiting_in_queue_millis" : 0,
 "active_shards_percent_as_number" : 50.0

You can view the cluster health on a shard level per index. Take note that I am beautifying the output with the command line tool jq.

$ curl -s 'http://localhost:9200/_cluster/health/aminno_member_email?level=shards' | jq '.'


Let’s look over how the shard count is defined during index creation.

When people start out with Elasticsearch they don’t bother to learn about shards and how to work with Shards. As your Elasticsearch journey progresses you will have to start taking note of how many shards and how many replicas your index will need. This is how you would define those values upon index creation:

PUT /food
  "settings" : {
     "number_of_shards" : 1,
     "number_of_replicas" : 1

This defines an index by the name of food, with one primary shard and one replica shard. Now we can view info on the index we just created with its predefined settings.

GET /_cat/shards/food?pretty=true

This is the output:

food 0 p STARTED    0 130b Deathlok 

As you can see in the output it shows the shard state, as well. There are different states that a shard can be in: Initializing, Unassigned, and Started.

Replica Shards

We mentioned earlier that replica shards can be defined when creating an index, but what are replica shards? Replica shards are copies of the primary, or original, shards. Replica shards have several purposes, but most importantly they serve the function of handling failure in your cluster and they also help to scale out when you need to increase your search capability.

Think of replica shards as a co-pilot sitting next to the pilot during a flight. The co-pilot can take over whenever the pilot is overloaded with work or if he is tired. The co-pilot does not have the same authority or same level of ability as the pilot, but the co-pilot can certainly save the day if things go wrong.

Case Study: How Qbox Saved 5 Figures per Month using Supergiant

As seen in the example, we need to define the number of primary shards when we create an index. We can also define the number of replica shards, although this is a setting that we can change at any time after index creation. Replica shards can be used to read data from, which allows them to be used to scale out your search capabilities, but not your data storage capabilities. You can scale out your search capabilities with shard replicas as long as you have hardware to add to your cluster.

Replica shards can be used as a failover mechanism when one of your primary shards fail or when a whole node in your cluster goes offline.  When a primary shard goes offline, Elasticsearch will replace that shard with a replica shard to keep the index in tact.

Remember, an index is built from shards, and replica shards are copies of your primary shards. Elasticsearch has a clever built-in capability to be self aware of primary shards that are taking strain, and then replace those failing shards with copies of themselves.

It is wise to assign more than one replica shard per primary shard for your mission critical indexes.  When it comes to high availability of your Elasticsearch cluster, the best way to be prepared for failure is to actually test or simulate failure in your cluster before it happens in real life.

Scaling An Index

People often ask what strategy should be used for scaling an index. The answer is to over-allocate shards, but this does not include replica shards. Replica shards can be under-allocated because they can be dynamically added or removed after the index has been created.

You can add your replica shards dynamically with:

$ curl  -XPUT 'localhost:9200/aminno_member_email/_settings' -d '{
   "index" : {
       "number_of_replicas" : 1,
       "refresh_interval" : "1s"

We can view the status of what is happening with our cluster in terms of shards with the following:

$ curl -XGET 'http://localhost:9200/_cat/shards?v'



Understanding shards and how to plan your cluster is an important part of implementing a successful Elasticsearch deployment for your organization. If you are not familiar with Shards and shard replicas, you will have problems scaling out your Elasticsearch cluster when it is too late. Invest some time to get familiar with shards and the concepts around using Shards in an Elasticsearch cluster. Questions/Comments? Drop us a line below.