DevOps Culture At ShopRunner
New projects flourish in a company that isn’t afraid to try new things or rip up legacy applications. Each new microservice deployed might require a cloud-based tool such as databases, queues, or CDN’s. With the creation and maintenance of each project’s infrastructure, you essentially need a full or half time DevOps engineer per team. Unintuitively, ShopRunner is able to manage all this infrastructure with just a single core Cloud Operations team. The way it has accomplished this feat is by allowing the CloudOps team to focus on building tools and spreading a culture of DevOps across the organization. The results are not only efficient to time and cost but also delight our various engineering teams.
The tools that the CloudOps team focuses on have enabled our engineering teams to easily deploy and iterate on microservices in our cloud infrastructure. As business scaled up it became painfully obvious to them that it would be impossible to personally service each new project. The CloudOps team philosophy became to build infrastructure or tools that would implicitly aid or allow for self serviceability. Our standard deployment pipeline consists of Github for code, Jenkins for CI, Spinnaker for deployments and Terraform for provisioning. Some of the projects CloudOps worked on include global Github webhook triggers, an internal Jenkins functions library, Spinnaker templates and a pipeline for applying application-specific terraform. To illustrate how these tools fit together, it’s helpful to walk through an example of an engineer deploying a brand new microservice to production.
An engineer seeds a fresh Github repository from our microservice code generator and adds in their custom business logic. Once a PR is created, a Jenkinsfile in the root of the new project starts the CI process. Eventually, that process will trigger a library function that will take care of building and publishing a Docker image and passing it along to Spinnaker. The application-specific pipeline in Spinnaker will ensure it deploys successfully with all the bells and whistles and is accessible with a respective domain. If the service needs its own datastore the team creates a new branch in the central infrastructure repo and writes terraform using modules or common examples. A separate pipeline is available to plan and apply that code, and only a CloudOps PR approval is required before it can run in production. By default, everything just works, but it’s worth noting that the owner has the flexibility to make modifications at every level since configuration is exposed in either the root of their project or the infrastructure repositories. It sounds simple on paper, but underlying the process is a steady effort of educating every engineering team in DevOps practices.
Free from the burden of supporting every individual project, the CloudOps team is able to dedicate time to teach others. The team makes itself accessible once a day with virtual office hours where it fields questions about existing tools or goes through support work. During this time each member of the team has a chance to teach engineers but also learn about remaining pain points or bottlenecks. Ideas for new tools or updates to existing ones are born from these sessions. Occasionally an engineer is motivated to issue a PR against a CloudOps tool. Common operational workflows are also documented in our wiki to answer repeated questions and tools have respective README’s. By breaking down the traditional barriers between developers and operations and making it a priority to spread knowledge, DevOps isn’t just a team but a ubiquitous culture.
If you’re interested in being delighted while you solve hard problems or delighting others by building a world-class cloud infrastructure, come join ShopRunner!
