With cloud native, Intuit brings ‘lightning fast’ deployment to thousands of developers
Challenge
To become more agile, Intuit was well on its way to migrating much of its infrastructure and services to public cloud. However, the lift-and-shift approach was not providing as much benefit as expected. Intuit realized that containerization and investment in cloud native technologies was needed. “Migration to public cloud freed us from managing infrastructure, but our lift-and-shift approach resulted in only slight increases in development velocity,” says Jeff Brewer, Chief Architect, Small Business and Self-Employed Group at Intuit. “Teams were still taking days and in some cases, weeks, to create and deploy new software releases.”
Solution
To accelerate containerization and adoption of cloud native technologies, Intuit decided to adopt Kubernetes, and to enable this transformation, acquired Applatix, the startup behind Argo, an open source application delivery platform for Kubernetes. “We finally knew where we wanted to go, but found ourselves in uncharted territory,” says Brewer. “We needed to quickly inject a team that already knew the way.”
Impact
With Kubernetes, developer productivity increased, as creating or upgrading a service takes less than 10 minutes, including the setup of automated end-to-end build and deployment pipelines. In the QuickBooks platform, the deployment cycle decreased from days to minutes, and MTTR went from 45 minutes to less than 5 minutes for a rollback or roll forward. In 2018 alone, Intuit operationalized more than 100 Kubernetes clusters running 500 services in production and pre-production across multiple business units.
Projects used
By the numbers
MTTR
Went from 45 minutes to
less than 5 minutes
Deployment cycle
Decreased from days to minutes
Creating or upgrading a service
Takes less than 10 minutes
For Jeff Brewer, the value of moving to cloud native is seen over and over again when Intuit developers receive feedback from their customers minutes after they’ve written the code.
“Those are the wow moments,” says Jeff Brewer, Chief Architect, Small Business and Self-Employed Group at Intuit. “Developers want to make something better, and they can get it into a customer’s hands within minutes of when they’re done coding it, and they’re getting that feedback soon after they’ve finished development. Before, you’d generally wait 30 days, and if you go way back in Intuit history, it was a year, because we were shipping software on a CD.”
Founded in 1983, the company is guided by the mission of “powering prosperity around the world” with its three main products, QuickBooks, TurboTax, and Mint.
Intuit was an early adopter of the public cloud, declaring they were all-in on the public cloud in 2013, and driving innovation at the intersection of speed, operability, and security.
Velocity was a major driver for the next step toward cloud native, which accelerated when containerization became a priority. Before, “a new developer on QuickBooks Online would take about three days to onboard onto the code base—you had to set up all kinds of complicated Tomcat and Apache and a database and all this stuff on your laptop,” he says. “We are growing fast, hire people frequently, and have people internal in the company who want to fix a bug, so you just can’t have code bases that take that long to onboard.”
Containerization brought the QuickBooks Online setup in the development environment down to under an hour. Given those results, Brewer then started looking to create a robust developer experience for use in production. “We needed to go from zero to sixty in cloud native, and at KubeCon in 2017, I met Applatix, who were behind the open source project Argo,” he says. “Their mission was to make Kubernetes easy to use for enterprises.”
Intuit acquired Applatix in early 2018, and accelerated its objective to bring cloud native to the company. “They’d been developing their own technologies to help enterprises get on Kubernetes, and they treated it like they were delivering a product to their customers,” says Brewer. “Now, Intuit was their only customer. It was amazing!”
At Intuit, the Kubernetes-based end-to-end platform is called “Modern SaaS”: “The charter for the Applatix team was simple: Create a self service developer platform that is easy to use, takes away the toil of infrastructure management, and lets them use the cloud native technologies that they want, with an amazing developer experience that’s really understandable,” says Pratik Wadher, Applatix’s Founder and CEO and now VP of Intuit Development Platform. “A huge benefit to developers is being able to understand how it goes from what they want, which is the declaration, to what it looks like in the system, and when you change our declaration, the system changes. It’s really about Git being a source of truth for your code and deployments, from build to apps in production, or GitOps. No more manual arcane scripts. That’s really the reason for the move from just development to also using the cloud native orchestration for deployment.”
The new platform delivered on the velocity front. “Even in a standard public cloud infrastructure, those things can take hours to roll those kind of deployments out, and with the cloud native technologies, we get just lightning fast speed of deployment and iteration and mean time to recovery in minutes,” says Wadher. In the context of QuickBooks suite of products, “releases used to be a ceremony. It would actually take days to roll out a release across our fleet, because we were updating all these manual scripts. Now you can release a new version of some very complicated steps of services under five minutes,” Brewer adds.
Plus, if anything goes wrong, things can be rolled back quickly; MTTR went from 45 minutes to less than 5 minutes.
“With the cloud native technologies, we get lightning fast speed of deployment and iteration and mean time to recovery in minutes. In the rare instance that something breaks, we have a built-in contingency plan to roll things back. So those are all the things that our developers love about the cloud native technologies and have spurred them to go and adopt it themselves.”
— JEFF BREWER, CHIEF ARCHITECT, SMALL BUSINESS AND SELF-EMPLOYED GROUP AT INTUIT
Scaling was another pain point solved by the cloud native platform. With high-scale periods throughout the year, such as the April 15 tax filing deadline in the U.S., when millions of transactions and filings are happening, “we need to be able to scale up super quickly and then when it’s over, scale back down,” says Brewer. “Before, on January 1, we’d be like, ‘Well, it’s going to look like a monster tax season, so we’d better order some more servers.’ But with cloud native we don’t have to worry about it. They run Game of Thrones on Kubernetes. They can handle anything.”
In fact, cloud native has “spread like wildfire” at Intuit. In 18 months, the company went from zero to about 2,000 services running on Kubernetes. “During engineering days at Intuit, the technology community has a week to work on a passion project. A vast majority of those projects from our last two engineering days have been onboarding their services onto Kubernetes. Using Kubernetes is not a mandate: It’s choose your own adventure, and that’s the adventure a lot of development teams chose,” says Brewer.
Intuit hasn’t looked back since adopting Kubernetes. Its confidence in the cloud native community and the ecosystem that has grown around Kubernetes is evidenced by the slew of CNCF technologies now incorporated into the platform, including Envoy, Open Telemetry, Jaeger, Fluentd, Prometheus, gRPC, and Open Policy Agent. “We expanded the platform’s abilities as we onboarded more services—some legacy—with a diverse set of needs,” says Wadher.
As a company that manages customers’ financial data, Brewer says, “we have a lot of compliance needs, and Open Policy Agent is something we adopted after realizing we really need to have much more declarative security policies on the clusters and explain easily to our auditors. Some technologies you just adopt right off the bat because they makes your overall management easier. You start out really, really simple, but you quickly realize you have these other needs, and there’s usually something in the CNCF or general open source that will solve those needs.”
“During engineering days at Intuit, the technology community has a week to work on a passion project. A vast majority of those projects from our last two engineering days have been onboarding their services onto Kubernetes. Using Kubernetes is not a mandate: It’s choose your own adventure, and that’s the adventure that they choose.”
— JEFF BREWER, CHIEF ARCHITECT, SMALL BUSINESS AND SELF-EMPLOYED GROUP AT INTUIT
Having benefited from community development, Intuit is committed to contributing back. “Applatix started the Argo project to bring Kubernetes native workflows project to the community, and after the acquisition Intuit has continued to make major contributions,” says Wadher. Intuit has made available to the community more projects under the Argo umbrella that provide continuous delivery using Argo CD and progressive delivery using Argo Rollouts. Additionally, Intuit has partnered with BlackRock, a fellow member of the CNCF end user community, to contribute Argo Events, which is an event-based dependency manager. Adds Wadher: “The projects are used at scale at Intuit, so they are battle tested.”
After operating 150+ clusters for more than a year, Intuit also open sourced a set of declarative CRDs for managing Kubernetes at scale in production, Keikoproj. “Anyone running hundreds of clusters at scale in an enterprise in any cloud would need these kinds of cluster lifecycle management tools,” says Wadher.
As a result of Intuit’s contributions to the community, the company was awarded the CNCF End User Award at KubeCon + CloudNativeCon EU in 2019. Also, in recognition of the active participation of Intuit as an end user member, Brewer was elected to be part of CNCF’s Technical Oversight Committee as the End User Representative.
The advice Brewer and Wadher give to other companies starting on this road is twofold: Think of the developer experience, and “create the paved road that solves your inherently focused use case, take that all the way through, and make sure that everything is solved well before you start exploding it out to trying to do everything,” says Brewer. And “hire a few engineers that deeply understand the cloud native technologies. You need to find a few of these people that you can really see the journey with.”