Shipping MyNames with Docker Published Tue Jul 15 14:00:00 SAST 2014
At MyNames, we ship our entire application as a pre-tested, pre-configured container image that can be deployed to any production box in seconds.
How we Streamlined Deployment and Escaped Dependency Hell
Over the past twelve months, my co-founder Pierre and I, have been building a .CO.ZA domain management service with a focus on usability and high-availability, dubbed MyNames.
Built in 15k lines of Python and CoffeeScript and deployed to AWS EC2, ease of deployment was a priority from day one. We started off using Puppet for provisioning, but quickly graduated to the new remote configuration tool on the block, Saltstack.
Dependency Hell
Without rigorous state management, deploying applications to mutable infrastructure quickly descends into dependency hell. Virtualisation and automation tools like Chef and Salt help manage dependencies, but spinning up a fresh VM can be costly and take minutes to boot.
As much as we enjoyed remote configuration with Salt, we spent far too much time fighting Linux, Python and mySQL dependencies with our YAML-based Salt State Files. That is, until we discovered Docker.
What is Docker?
Docker is a Linux container engine that solves the dependency hell problem by shipping applications in containers that are isolated at the network and file system level. Containers are easy to package, ship and deploy to virtually any infrastructure.
Within two days we migrated our deployment flow to a Docker-based container architecture. Our build process now packages a ~700MB Docker image artifact which represents all our application services. Within minutes, any LXC-compatible Linux machine can pull this image and run our entire infrastructure with no dependency hell or configuration.
We're still using Salt for orchestration to trigger Docker image updates and service restarts, but we've significantly cut back on our configuration script size.
Key Benefits
My favourite part of this Dockerised deployment flow is that all of our tests run in an isolated, pre-packaged container before it is deployed. The image is only tagged as "tested" and pushed to production if all tests pass. Since the container fully emulates our production environment, we never have any surprises in production due to missing dependencies.
Our Deployment Flow
- New code changes are noticed by TeamCity and fetched by a build agent.
- A Docker build kicks off on the agent using the latest
Dockerfile, which describes all our Python, mySQL and Flask dependencies[^1]. - The latest source code is added to the newly built image.
- The build agent runs tests in a new container from the freshly-built Docker image.
- If all tests pass, the image is tagged as "tested" and pushed to a private Docker registry.
- The build agent triggers a Salt highstate on the Saltmaster box, instructing all our Salt minion machines to pull the latest Docker image and restart containers according to their assigned roles.
Voila! Within ~7 minutes, our new, fully-tested code is deployed to our fleet of instances and running as isolated Docker containers. With some optimisations, like hosting our own Docker image registry, we can probably cut this time in half.
You should use Docker too.
[^1]: Old dependencies are cached to speed up builds.
