Read CEO Mark Brandon’s explanation of business continuity steps taken in the face of the global health and economic crisis.
As the global COVID-19 health crisis and its attendant economic fallout continue to evolve, there are understandable concerns about business continuity and disaster recovery in case our small business became a casualty along with so many others. This post illuminates the plans we have in place to protect your data recovery so your Qbox Elasticsearch instances are NOT one more thing you will have to worry about amid all the other challenges.
First, our company transitioned to being fully distributed last year. All employees work from home already so there is little chance that our work environment will result in mass infection and staff shortage. For security purposes, and without revealing too much, our employees’ encrypted laptops with certificates and a location range are required for them to access the AWS resources. Our ticket queue is staffed around the clock with backups on call. Of course, employees have standing instructions to take care of themselves and their families if they show symptoms, but fingers crossed, this has not affected anybody on the Qbox payroll as I write.
Second, since all infrastructure is housed on Amazon Web Services, we pay for and receive the highest level of service from them, backed by their own aggressive Service Level Agreements. This alone should be worth the markup that is our margin because it would cost many thousands of dollars per month at minimum for each of you to get this level of service separately from AWS.
Third, our architecture is meant for fault tolerance. We have Kubernetes clusters in the various data centers and replicate across availability zones. These architectural choices mean that you have fallback resources in the case of overuse and higher network throughput than is available on comparable virtual machines. For most of your instances, it would require more than one availability zone to go off line for extended periods for data loss to occur. Given AWS’ uptime guarantee of 4 nines and actual performance across their history of many more nines, this is an unlikely event. I should make a disclaimer that many customers pay for a sandbox cluster, meaning that it is fewer than the best-practice 3 nodes required of a production-worthy machine. We do this for the many customers that want a cheap sandbox for test and dev purposes. If you are trying to run this type of topography in production, you should upgrade your infrastructure to get the full high availability and disaster recovery benefits. Fill out a support ticket, and we can talk you through it.
Fourth, since the beginning, we have taken daily snapshots of your cluster and protected them on AWS S3 buckets, which itself has uptime and availability in the multiple nines. In the worst case, restoring this snapshot would result in something less than 24 hours of data loss. However, this scenario is presently a trivial risk.
Fifth, at any time, you can ask our support personnel to perform a snapshot and provide it to you between this daily window. You are not locked in to our service. If you need to bring your cluster in house, this is how you do it. Spin up your own cluster, restore from our snapshot, and as long as you know how to self-host Elasticsearch, you can be up and running within a few minutes or more, depending on the size of your dataset. It is important to note that, with few exceptions, our employees support only the infrastructure within our service.
Sixth, while maintaining our own privacy, I would just like to state that our financial condition is not precarious. We have been cash flow positive for some time and have a cushion against adverse economic conditions. Obviously, this could change along with the pain that our customers endure. I ask that, if you are considering making changes to your relationship with us, please give us a chance to show you how we can solve your problems.
Finally, I would like to extend an offer to you if you have a measure of certainty. The best thing we can do to cut costs — which can be passed on to customers — is to commit to forward pricing from AWS. This requires an annual commitment. Any customer that projects a spend of more than $7000 annually can get a discount from us because the commitment allows us to procure infrastructure with less cost from AWS. If this is you, please fill out a support ticket, and we will be glad to show you the options. For those with certainty, this can help you profit from the turmoil. For those without certainty, this can help you weather the storm.
Wishing you a healthy 2020,
Mark Brandon, CEO, Qbox, Inc.