How Kubernetes has been ‘transformational’ to productivity and culture at uSwitch
Challenge
For years, uSwitch had a decentralized infrastructure; teams ran all their own AWS infrastructure and were responsible for configuring load balancers, EC2 instances, ECS cluster upgrades, and more. Though many teams had converged on the idea of using containerization for web applications and deploying to ECS, “everyone was running different kinds of clusters with their own way of building and running them, with a whole bunch of different tools,” says Head of Engineering Paul Ingles. With the increasing cloud and organizational complexity, they had difficulty scaling teams. “It was just inefficient,” says Infrastructure Lead Tom Booth. “We wanted to bring some consistency to the infrastructure that we had.”
Solution
In late 2016, Booth and Ingles began working on building a Kubernetes platform for uSwitch. All 36 teams at uSwitch are now running at least some of their applications on the new platform.
Impact
The rate of deployments has increased almost 3x. The release rate per person per week has almost doubled, with releases now happening as often as 100 times a day. And those improvements have been sustained, even as more people were added to the platform. Resource utilization of AWS machines went from 3% to 10% without much effort. With node startup time just a couple of minutes, coordination among teams is no longer needed, even when they have large workloads.
Projects used
By the numbers
Releases
Happen as often as 100x a day
Deployments
Went up almost 3x
Resource utilization
Went from 3% to 10% without much effort
When it was founded in 2000, uSwitch helped consumers in the U.K. compare prices for their utilities.
The company eventually expanded to include comparison and switching tools for a wide range of verticals, including broadband service, credit cards, and insurance.
Internally, those verticals translated into a decentralized technology infrastructure. Teams were organized around the markets their applications served. So when the company moved from data centers to AWS in 2010, each team ran its own AWS infrastructure and was responsible for configuring load balancers, EC2 instances, ECS cluster upgrades, and more.
Though many of uSwitch’s engineering teams had converged on the idea of using containerization for web applications and deploying to ECS with a continuous delivery tool, “Everyone was running different kinds of clusters with their own way of building and running them, with a whole bunch of different tools,” says Head of Engineering Paul Ingles. “Teams would use containers because it helped them reach their goals, but they weren’t necessarily optimizing for the repeatability of that across the different teams. People were just left to invent their own ways of rolling out stuff to precisely the same environments.”
With the increasing cloud and organizational complexity, they had difficulty scaling teams. “It was just inefficient,” says Infrastructure Lead Tom Booth. “We wanted to bring some consistency to the infrastructure that we had.”
The solution, they believed, was a container orchestration system. “We were convinced that if we just nudged the organization into a different style of working, that could have quite a significant impact on the productivity of teams,” says Ingles.
Booth and Ingles had both experimented with Kubernetes before, when some uSwitch teams were adopting containerization. Separately, the infrastructure team had done an evaluation of several orchestration systems, in which Kubernetes came out on top. “The fundamental model of Kubernetes works very well for us,” says Ingles.
In late 2016, they began building a Kubernetes platform for uSwitch. “Our early customers were people who were most in need of support or were most amenable to having somebody come and take problems away from them,” says Ingles. For one thing, the platform took care of security issues that product teams didn’t have time to focus on. “There were really old machines that weren’t patched, commissions and security boots all over the place,” says Booth. “We took that off their plates, so they could just focus on product development.”
“The fundamental model of Kubernetes works very well for us. Its extensibility continues to give us scope for expansion of what we’re doing in the future, which is very cool.”
— PAUL INGLES, HEAD OF ENGINEERING AT USWITCH
Ingles and Booth also worked on developing tools that would become a big draw for the new platform. For instance, running on the Kubernetes clusters gives teams easier integration to uSwitch’s secrets management system (Hashicorp Vault) and distributed tracing (Google Cloud Trace).
The platform uses a number of CNCF technologies, including gRPC, OpenTracing, and Fluentd. “All these pieces seem to be coming together to provide all of the components to build really awesome infrastructure platforms,” says Ingles. They’re building their own components to work with the CNCF technologies, too. One component, Yggdrasil, works with Envoy to allow teams to deploy across multiple clusters. “That helped us convince the bigger teams with more significant revenue in the organization that we’ve got a good answer to you being concerned about clusters changing without your knowledge.” Another component, Heimdall, supports pre-programmed and custom alerts in Prometheus. Using gRPC and Prometheus, their Kiam tool integrates Kubernetes pods with IAM roles in AWS, which allowed teams to move their systems to Kubernetes without having to make application code changes.
The team then built a command line tool that could work across end-user platforms, to replace the many different ones that engineers had created on their own for their individual needs and languages. “We started a project that all people have read and write access to, that’s written in Go so it will work across Linux, then Mac and Windows,” says Ingles. “It just has a bunch of different commands that are useful for triggering deployment or authenticating. We wanted to piggyback the permissions and access control within Kubernetes, so that we could hand off user management to the IT team rather than that being something that the infrastructure team have to run.”
“The amount we can do with three or four people, because of the consistency we gained using Kubernetes and the extensibility of its language, is amazing. We’re so empowered. The growth is only about bringing in new features that other teams require. We could take what we have and run double [the size of] uSwitch, and it would feel the same to us as it is now.”
— TOM BOOTH, INFRASTRUCTURE LEAD AT USWITCH
In little more than a year, all 36 teams at uSwitch were running at least some of their applications on the new platform. The speed of adoption shouldn’t come as a surprise, given the impact Kubernetes has had. The rate of deployments has increased almost 3x. The release rate per person per week has almost doubled, with releases now happening as often as 100 times a day. And those improvements have been sustained, even as more people were added to the platform. Resource utilization of AWS machines went from 3% to 10% without much effort. With node startup time just a couple of minutes, coordination among teams is no longer needed, even when they have large workloads.
Moreover, the culture changed—which was an overarching goal from the beginning. “We wanted to change the way the teams work and talk to each other, and it’s been entirely transformational,” says Ingles. “You recognize it through the Slack conversations and the movement of people between desks, problem-sharing and having to do a lot less stuff on their own that they used to. Without Kubernetes, it almost certainly would have been a vastly more difficult thing to undertake.”
Kubernetes is also enabling uSwitch to plan for the future. “The amount we can do with three or four people, because of the consistency we gained using Kubernetes and the extensibility of its language, is amazing,” says Booth. “We’re so empowered. The growth is only about bringing in new features that other teams require. We could take what we have and run double [the size of] uSwitch, and it would feel the same to us as it is now.”
As Ingles puts it: “Kubernetes’s extensibility continues to give us scope for expansion of what we’re doing in the future, which is very cool.”