Several months ago, shortly after a fundraising round enabled us to finally undertake long-term thinking, I tasked our team to come up with an infrastructure model that was better optimized, lower cost, more stable, more performant, and ultimately the highest, best use for our favorite data exploration and analytics platform.
Our team suggested a containerization route, with Docker being the most appealing. However, we found that Docker can be cumbersome with stateful applications like Elasticsearch, Postgres, MySQL, etc. Perusing the case study history, we also found that using Docker in a SaaS environment came with its share of performance and “noisy neighbor” problems.
We knew that other companies in our space were taking this approach, but we also knew that their performance was grossly sub-par compared to what we knew we could provide. In order to get the benefits of Docker without the problems, we would have to rethink stateful applications on Docker and rethink multi-tenancy for a very compute-intensive, distributed application like Elasticsearch.
Kubernetes went part of the way, and our hand-rolled Supergiant did the rest. Over the past several months, we’ve fused Kubernetes with the last 3 years’ worth of our cloud experience, and produced an open source solution with:
- Automated server management / capacity control
- Sharable load balancers
- Volume management, resizing, backups
- Extensible deployments
- Resource monitoring
- Seriously cool and spacey UI
Supergiant is an active work in progress, so that list will be growing quickly over the next few months. But it’s already being used in production, with a major impact.
Supergiant is now available on Github, freely downloadable, and usable under an Apache 2 license. For now, Supergiant works for Amazon Web Services. We have Google Cloud, Rackspace, and Openstack on the road map.