Our customers who utilize Rackspace both for their own infrastructure and the location of their Qbox Elasticsearch deployments have been rightfully alarmed about a possible cloud re-boot.
In October, a widespread vulnerability in the Xen Hypervisor resulted in an abrupt re-boot across all three of our infrastructure partners, resulting in a very unpleasant weekend for many, especially those of us at Qbox. Over the last few days, they have been sending messages to customers hinting at another re-boot.
Our team here at Qbox immediately started preparing.
During the last round, a minority of our servers (although a larger percentage than would be expected) just failed to reboot at all. This state could not be more adverse for a clustered application like Elasticsearch. After the last round, we refined our contingency plans, had everything backed up, and ready in case similar events affected us in the future.
Rackspace today sent a message indicating that their recently-discovered issue will require less disruptive measures to fix — so that possibility of a re-boot is off.
The following is from an unnamed Racker:
“Our team is focused on keeping your environment secure. We will continue to communicate with you as circumstances dictate to assure that you have the information needed to respond to risk in a timely and effective manner.”
So we can all rest easy for now.