We discussed about Data Denormalization in our previous post Denormalization and Concurrency Issues in Elasticsearch and had emulated a filesystem with directory trees in Elasticsearch, much like a filesystem on Linux: the root of the directory is /, and each directory can contain files and subdirectories. The problem comes when we want to allow more than one person to rename files or directories at the same time. We shall be discussing about Concurrency issues and various kinds of locking in Elasticsearch in this post.

For this post, we will be using hosted Elasticsearch on Qbox.io. You can sign up or launch your cluster here, or click “Get Started” in the header navigation. If you need help setting up, refer to “Provisioning a Qbox Elasticsearch Cluster.

Let’s consider that you rename the /qbox directory, which contains hundreds of thousands of files. Meanwhile, another user renames the single file /qbox/cloud/elasticsearch/elasticsearch.yml. That user’s change, although it started after yours, will probably finish more quickly.

One of two things will happen:

  • You have decided to use version numbers, in which case your mass rename will fail with a version conflict when it hits the renamed elasticsearch.yml.
  • You didn’t use versioning, and your changes will overwrite the changes from the other user.

The problem is that Elasticsearch does not support ACID transactions. Changes to individual documents are ACIDic, but not changes involving multiple documents.

If your main data store is a relational database, and Elasticsearch is simply being used as a search engine or as a way to improve performance, make your changes in the database first and replicate those changes to Elasticsearch after they have succeeded. This way, you benefit from the ACID transactions available in the database, and all changes to Elasticsearch happen in the right order. Concurrency is dealt with in the relational database.

If you are not using a relational store, these concurrency issues need to be dealt with at the Elasticsearch level. The following are three practical solutions using Elasticsearch, all of which involve some form of locking:

  • Global Locking
  • Document Locking
  • Tree Locking

Global Locking

We can avoid concurrency issues completely by allowing only one process to make changes at any time. Most changes will involve only a few files and will complete very quickly. A rename of a top-level directory may block all other changes for longer, but these are likely to be much less frequent.

Because document-level changes in Elasticsearch are ACIDic, we can use the existence or absence of a document as a global lock. In order to request a lock, we try to create the global-lock document:

curl -XPUT ‘ES_HOST:ES_PORT/filesystem/lock/global/_create -d '{
}'

If this create request fails with a conflict exception, another process has already been granted the global lock and we will have to try again later. If it succeeds, we are now the proud owners of the global lock and we can continue with our changes. Once we are finished, we must release the lock by deleting the global lock document:

curl -XDELETE 'ES_HOST:ES_PORT/filesystem/lock/global'

Depending on how frequent changes are, and how long they take, a global lock could restrict the performance of a system significantly. We can increase parallelism by making our locking more fine-grained.

Document Locking

Instead of locking the whole filesystem, we could lock individual documents by using the same technique as previously described. We can use a scrolled search to retrieve all documents that would be affected by the change and create a lock file for each one.

Here, the ID of the lock document would be the same as the ID of the file that should be locked and the process_id is a unique ID that represents the process that wants to perform the changes. If some files are already locked, parts of the bulk request will fail and we will have to try again.

curl -XPUT 'ES_HOST:ES_PORT/filesystem/lock/_bulk'
{ "create": { "_id": 1}}
{ "process_id": 123    }
{ "create": { "_id": 2}}
{ "process_id": 123 }

It is obvious that if we try to lock all of the files again, the create statements that we used previously will fail for any file that is already locked by us! Instead of a simple create statement, we need an update request with an upsert parameter and this script:

Here, process_id is a parameter that we pass into the script. assert false will throw an exception, causing the update to fail. Changing the op from update to noop prevents the update request from making any changes, but still returns success.

if ( ctx._source.process_id != process_id ) {
  assert false;
}
ctx.op = 'noop';

The full update request looks like this:

curl -XPOST 'ES_HOST:ES_PORT/filesystem/lock/1/_update' -d '{
  "upsert": { "process_id": 123 },
  "script": "if ( ctx._source.process_id != process_id )
  { assert false }; ctx.op = 'noop';"
  "params": {
    "process_id": 123
  }
}'

If the document doesn’t already exist, the upsert document is inserted, much the same as the previous create request. However, if the document does exist, the script looks at the process_id stored in the document. If the process_id matches, no update is performed (noop) but the script returns successfully. If it is different, assert false throws an exception and you know that the lock has failed.

Once all locks have been successfully created, you can proceed with your changes.

Afterward, you must release all of the locks, which you can do by retrieving all of the locked documents and performing a bulk delete:

The refresh call ensures that all lock documents are visible to the search request.

curl -XPOST 'ES_HOST:ES_PORT/filesystem/_refresh'

We can use a scroll query when you need to retrieve large numbers of results with a single search request.

curl -XGET 'ES_HOST:ES_PORT/filesystem/lock/_search?scroll=1m' -d '{
    "sort" : ["_doc"],
    "query": {
        "match" : {
            "process_id" : 123
        }
    }
}'
curl -XGET 'ES_HOST:ES_PORT/filesystem/lock/_bulk'
{ "delete": { "_id": 1}}
{ "delete": { "_id": 2}}

Document-level locking enables fine-grained access control, but creating lock files for millions of documents can be expensive. In some cases, you can achieve fine-grained locking with much less work, as shown in the following directory tree scenario.

Tree Locking

Rather than locking every involved document as in the previous example, we could lock just part of the directory tree. We will need exclusive access to the file or directory that we want to rename, which can be achieved with an exclusive lock document:

{ "lock_type": "exclusive" }

We need shared locks on any parent directories, with a shared lock document.Here, the lock_count records the number of processes that hold a shared lock

{
  "lock_type":  "shared",
  "lock_count": 1
}

A process that wants to rename /qbox/cloud/elasticsearch/elasticsearch.yml needs an exclusive lock on that file, and a shared lock on /qbox, /qbox/cloud, and /qbox/cloud/elasticsearch.

A simple create request will suffice for the exclusive lock, but the shared lock needs a scripted update to implement some extra logic. If the lock_type is exclusive, the assert statement will throw an exception, causing the update request to fail. Otherwise, we increment the lock_count.

if (ctx._source.lock_type == 'exclusive') {
  assert false; 
}
ctx._source.lock_count++

This script handles the case where the lock document already exists, but we will also need an upsert document to handle the case where it doesn’t exist yet. The full update request is as follows:

The ID of the document is /qbox, which is URL-encoded to %2fqbox

curl -XPOST 'ES_HOST:ES_PORT/filesystem/lock/%2Fqbox/_update' -d '{
  "upsert": {
    "lock_type":  "shared",
    "lock_count": 1
  },
  "script": "if (ctx._source.lock_type == 'exclusive')
  { assert false }; ctx._source.lock_count++"
}'

Once we succeed in gaining a shared lock on all of the parent directories, we try to create an exclusive lock on the file itself:

curl -XPUT 'ES_HOST:ES_PORT/filesystem/lock/%2Fqbox%2fcloud%2felasticsearch%2felasticsearch.yml/_create' -d '{ "lock_type": "exclusive" }'

Now, if somebody else wants to rename the /qbox directory, they would have to gain an exclusive lock on that path:

curl -XPUT 'ES_HOST:ES_PORT/filesystem/lock/%2Fqbox/_create -d '{ "lock_type": "exclusive" }'

This request would fail because a lock document with the same ID already exists. The other user would have to wait until our operation is done and we have released our locks. The exclusive lock can just be deleted:

curl -XDELETE 'ES_HOST:ES_PORT/filesystem/lock/%2Fqbox%2fcloud%2felasticsearch%2felasticsearch.yml'

The shared locks need another script that decrements the lock_count and, if the count drops to zero, deletes the lock document. Once the lock_count reaches 0, the ctx.op is changed from update to delete.

if (--ctx._source.lock_count == 0) {
  ctx.op = 'delete'
}

This update request would need to be run for each parent directory in reverse order, from longest to shortest:

curl -XPOST 'ES_HOST:ES_PORT/filesystem/lock/%2Fqbox%2fcloud%2felasticsearch/_update -d '{
  "script": "if (--ctx._source.lock_count == 0) { ctx.op = 'delete' }”
}'

Tree locking gives us fine-grained concurrency control with the minimum of effort. Of course, it is not applicable to every situation and the data model must have some sort of access path like the directory tree for it to work.

The global, document, or tree locking, however, doesn’t deals with the problem associated with locking: what happens if the process holding the lock dies? The unexpected death of a process leaves us with two problems:

  • How do we know that we can release the locks held by the dead process?
  • How do we clean up the change that the dead process did not manage to complete?

While denormalization is a good choice for many projects, the need for locking schemes can make for complicated implementations. Instead, Elasticsearch provides two models that help us deal with related entities: nested objects and parent-child relationships.

Give it a Whirl!

It’s easy to spin up a standard hosted Elasticsearch cluster on any of our 47 Rackspace, Softlayer, or Amazon data centers. And you can now provision your own AWS Credits on Qbox Private Hosted Elasticsearch.

Questions? Drop us a note, and we’ll get you a prompt response.

Not yet enjoying the benefits of a hosted ELK-stack enterprise search on Qbox? We invite you to create an account today and discover how easy it is to manage and scale your Elasticsearch environment in our cloud hosting service.