We get it. You want to easily support and scale your Elasticsearch cluster as your application load increases. Of course you want to reduce costs by absorbing that maintenance effort in-house. But have you given serious thought to all of the costs that you’ll incur?
It’s much more than saving money on a remote virtual machine. There are other critical factors to consider: reliability, scalability, security, and expert support.
Continue reading for a full treatment of these and other factors. If you haven’t run into issues with your locally-managed cluster, you can be sure that you will. We’ll be ready to help you when you need to upgrade.
It’s easy enough to acquire proficiency in running Elasticsearch because it ships with sensible defaults and hides complex search and distribution mechanics from beginners. It works quite well, right out of the box. With a short learning curve for grasping the basics, you can learn to be productive with indexing and searching in a short amount of time.
Before we continue, allow us to pose a question: How do maintain your environment—and keep it stable—when the software vendor (Elastic) continues a product release pace that has included 9 releases in the 6 months? An important function of a hosted Elasticsearch provider is to verify new bug fixes and major software releases into an infrastructure that is mature and robust. This is difficult to do when you are maintaining your own ES production environment.
There’s a better way. A hosted Elasticsearch service can help you manage all of these costs. You can start small and keep your initial costs way down. Then when you’re ready, you can easily grow your cluster in the cloud and incrementally limit your cost increases. This is a big help in managing the expectations of your team—and your boss. Continue reading for more details, and also learn how to reap an intangible benefit: peace of mind.
In this article we compare various aspects of Elasticsearch cluster management. We contrast the different types of effort, the significant difference in cost, and the major differences in time that you’ll spend doing it yourself in comparison with a hosted service.
1. Setting Up the Cluster
Hosted Elasticsearch: With a robust hosting provider, you get a simple dashboard with an array of options to configure your new ES cluster. You can easily add memory, nodes, and cores.
Do-it-yourself cluster management: Initial set up means editing the configuration files manually. Any future changes will also need to be done manually, and these edits are usually more prone to errors and mistakes. You’ll also need to spend time learning what changes might be necessary when you make the move to a production environment.
Hosted Elasticsearch: When you go with a ES hosting service, the provider takes responsibility for network security. Yes, you still own the security of your application—but the hosting service is likely to offer additional features. This is a critically important consideration because Elasticsearch doesn’t provide authentication and authorization. In addition to SSL and HTTP authentication, Qbox offers easy-to-configure IP whitelisting.
DIY cluster management: You need to protect your own network: build your own firewalls, configure and maintain security software, and ensure that your security configurations are solid.
3. Reliable Backup and Restore
Hosted Elasticsearch: A good hosting provider will schedule regular backups and make it easy to restore on demand. Qbox keeps a rolling backup for 7 days, and clusters can automatically recover.
DIY cluster management: It’s all on you to configure and verify a reliable backup system. Of course, you’ll need to acquire and connect separate hardware to store the backup data.
4. Automatic Monitoring
Hosted Elasticsearch: The ES hosting provider is likely to offer an number of management tools or plugins to help you monitor your cluster. Even if you don’t actively monitor your cluster, the hosting provider will perform basic monitoring and health checks. Qbox goes further and offers automatic cluster alerting for unresponsive nodes, low disk space, memory usage, shard allocation thresholds, and various other problems with your configuration.
DIY cluster management: There are a number of tasks for you to manage in regard to cluster monitoring. The most labor-intensive and high-risk approach for a production cluster is to attempt manual monitoring. If you realize that burden is too great, then you’ve got to invest effort evaluating automated solutions. And of course you need to perform the (usually tedious) job of configuring the alerts to trigger on specific conditions. Then there’s the balancing act: if you set a threshold too low, you’ll get too many alerts and your support team will eventually disregard them. If you set it too high, you may invite disaster.
5. Maximizing Scalability
Hosted Elasticsearch: A hosting provider is likely to have all the capacity you could possibly ever need, yet make it available to you in increments. The Qbox dashboard gives you a very simple interface so that you can easily scale both vertically and horizontally. When you’re ready, you can move up to a larger VM, add memory, or add more nodes to your existing cluster. All it takes is a few clicks.
DIY Cluster Management: In addition to your initial capital investment, any significant growth in demand means that you’ll need to acquire, install, and connect additional resources to your existing cluster. As you probably know, this is especially risky in your production environment.
6. Minimizing Downtime
Hosted Elasticsearch: Changes can often result in downtime that negatively impacts your production cluster, which can be very detrimental to end-user experience. Since it isn’t typically feasible to simply resize a cloud server, you’ve got to replace it. To maximize uptime, Qbox migrations will link the nodes on the existing cluster to other nodes that have a different size. This approach is very efficient for transferring data (shards) from the source nodes to the destination nodes. Also, Qbox replicated clusters give you even more flexibility and very little risk of downtime. You can easily configure the amount of replication, which gives your more throughput and redundancy. With properly configured replicated clusters, you can alter the system configuration, add or remove nodes, or migrate with zero downtime.
DIY cluster management: With homegrown clusters, there is a significant risk that any configuration changes will have serious impact that may bring the system down.
7. Enjoying Peace of Mind with Round-the-Clock Support
Hosted Elasticsearch: With a service, you get access to Elasticsearch experts. They can help you configure the application and work through any issues you experience. Qbox is a premier service that offers 24/7 support—ready to help you when you need it most.
DIY cluster management: You—or your team—will have to own this. In addition to development tasks, you’ve got to wear the hats of a system administrator and a support representative. You’ll have to do the research, which takes additional time and effort. These inefficiencies can have a major negative impact on user experience and the reputation of your team.
8. Careful Management of Software Upgrades
Hosted Elasticsearch: Elasticsearch upgrades and bug-fix releases become available at rate that averages more than once a month. A meticulous hosting provider will monitor all vendor releases, verify them, and then make them available as soon as is practicable. Premier providers such as Qbox will keep multiple versions of the software availabl and permit a wide range of versioning for their customer base. Qbox also provides 24/7 support to help users step easily through any difficulty they may encounter with upgrades.
DIY cluster management: With a homegrown cluster, you’ll need to retrieve and install the latest version of software. Ask yourself: How long will it take to validate a continual stream of new releases and updates? You’ll need to manage continuity in your production environment while installing the upgrade—and be ready to implement a contingency plan in case you need to fallback for some reason. And, of course, you’ll need to test your application with the new version and resolve any issues yourself. If you’re going it alone, you can’t lean on the expert assistance of an ES hosting provider in such critical tasks.
A Strong Case for Hosted Elasticsearch
In summary, let’s recap what we’ve been saying since our inception: when you host your ES cluster with a SaaS service, much of the tedium that comes with cluster management goes over to the hosting provider. When that happens, you’re free to spend the extra time on other important tasks.
When you consider both the hard costs and intangible costs, a hosted service will be a better value in the long run. Even if somehow you can build a homegrown setup that costs less than the initialization costs on a hosted service, you will quickly reclaim those costs in saved effort and time—and in the frustrations that you avoid.
You can greatly reduce the effort that you and your colleagues spend on all the tedious details of managing your Elasticsearch deployment. This leaves you free to focus on developing your application and optimizing your Elasticsearch cluster.