Fidelity Investments’ developers release 20x more frequently with multi-cloud strategy
Challenge
One of the largest financial services companies in the world, Fidelity Investments serves more than 35 million investors with more than 76 million accounts. A few years ago, the company launched a digital transformation focused on “leveraging next generation platforms and technologies to improve business value, increase speed to market, and harness the power of innovation,” says Amr Abdelhalem, SVP, Head of Cloud Platforms.
Solution
That effort included embracing a multi-cloud strategy, which meant migrating thousands of critical, highly regulated, low-latency applications to multiple cloud providers over the course of several years. The solution would be a multi-level platform built on top of Fidelity Cloud Fabric, which has Kubernetes and other CNCF technologies at its foundation.
Impact
Fidelity’s cloud native journey began in 2018 when the cloud platform team began offering Kubernetes as a platform, powered by multiple managed cloud Kubernetes services to targeted application teams. “We looked at the issues the applications teams have in general when adopting a brand-new technology,” says Cloud Platform Architect Rajarajan Pudupatti. Relying on an active feedback loop with developer focus groups, the team worked on creating a platform that also took into consideration Fidelity-specific requirements, primarily information security and data protection.
Projects used
By the numbers
Scale
3,000 Kubernetes services in cloud, close to 200 Kubernetes clusters, over 1,000 namespaces, and 10,000 containers
Releases
20x more frequently
Deploy to production
Reduced to a matter of minutes.
One issue that quickly arose was that Fidelity also had distributions of Kubernetes on-prem, as well as on other cloud providers. How could they introduce, for example, a new security process across 1,000 distributed applications?
“We really started to work on building conformity across all of the platforms that a business unit may consume with a goal of a consistent developer experience,” says Niraj Amin, Cloud Platform Architect. “If it’s Kubernetes, it’s Kubernetes. So we work to eliminate some of the complexities or the differences between running Kubernetes on-prem versus it being a specific cloud provider.”
So now, adds Pudupatti, “Everything rolls through the Kubernetes construct. We can simply roll out the particular add-on in a particular release, and it will change the hundreds of Kubernetes clusters in a uniform way. With thousands of microservices running in a particular cluster, within a simple change, all of them start conforming to a particular security process. This was one of the important points for us when we started.”
Fidelity was able to address some of its most pressing regulatory and security requirements by creating its own operators and automating processes specific to the financial services industry. “Because of Kubernetes’s extensibility feature, we can actually plug in our own logic,” says Pudupatti. “Any time there is a problem, we can go back and look at the Kubernetes design, and there’s always something that we can actually do about it.”
For example, the team built and open sourced KConnect, a CLI that allows users to onboard to the platform and discover and securely connect to Kubernetes clusters they have access to across multiple operating environments. Another operator was built to limit who can create namespaces, and once they’re created, to make sure that they conform to the established requirements. “That entire process is automated,” says Amin. “We’re leveraging that now to have the ability to potentially do other things around governance with the use of Open Policy Agent, and tying constructs that we’ve built internally and building on top of that.”
Pudupatti points out that as they build out the platform, “The very first thing we do is look at the CNCF tool kit and the projects that are going in. We’ll try our best to reuse whatever possible. This is another important thing for us: We always look at the community direction, so even if there is an easy way, but we think that the community is actually going in a different direction for certain reasons, we take the approach of not going for anything short term. Let’s try to stay together with the community in terms of doing something.”
The end result of all this work is a multi-level platform on top of what the team calls Fidelity Cloud Fabric. “Fidelity Investments is leveraging multiple CNCF projects to power our next generation of cloud native platforms,” says Pudupatti. “We use CoreDNS for service discovery, etcd as KV store, Fluentd for logging, Helm as a package manager, Kubernetes for container orchestration, rely on CNI for Networking API, and Open Policy Agent for policy management. We are also leveraging CNCF sandbox projects such as cert-manager for certificate management and Flux for GitOps.”
For the thousands of applications teams at Fidelity, “the Fabric itself is a way to develop and innovate within a multi-cloud, hybrid-cloud model,” says Abdelhalem. “Our ecosystem itself, all our application lifecycle management tools, our observability layers, our caching layers, our security and governance layers, our AI and machine learning layers are managed across this multi-cloud provider. Then our business platform is on top of that. The goal is that one day, we’re going to be floating between cloud providers, and be able to move workloads everywhere worldwide.”
The Fidelity team acknowledges that there is still much work to be done on the company’s journey to the cloud. “We still have certain business units talking back to on-prem services, talking to other accounts and other services that are already in the cloud, talking to SaaS solutions,” says Amin. “It will be complex getting everything to the cloud, but I’m hoping this platform will ease some of those difficulties as developers make that journey. Kubernetes gives us a platform within our platform to easily commonly build on top of.”
So far, the benefits have been clear: “The CNCF technological breadth has had a significant impact for Fidelity, as we have seamlessly accelerated application migrations to cloud,” says Pudupatti. “In a short period of time, we have reached close to 3,000 Kubernetes services in cloud, close to 200 Kubernetes clusters, over 1,000 namespaces, and 10,000 containers.”
“The speed of innovation is very important for the future of Fidelity products. Because of the adoption of these CNCF technologies, developers release 20x more frequently than before and the amount of time a developer spends in trying to deploy to production has significantly reduced. What was previously several days (involving lots of manual interventions) has now been reduced to a matter of minutes. When you have thousands of developers working in an organization, the impact is exponentially higher.”
Rajarajan Pudupatti, CLOUD PLATFORM ARCHITECT, Fidelity Investments
Portability has also been a big benefit, given Fidelity’s data center migrations and eventual move of many applications to the cloud. “Migrating to a different cloud provider is a matter of hours, against what was before a matter of months or even impossible in some cases. If you’re running something on-prem, it’s in the Kubernetes world, so you for the most part should be able to run it on any cloud provider,” says Amin.
More broadly, cloud native will have an impact for years to come. “One of the mandates that we’re getting from our CTO is a very aggressive cloud journey in a few years, and at the same time, avoid vendor lock-in and technology lock-in and cloud lock-in,” says Pudupatti, “and the main driver is Kubernetes. That’s how we’re achieving cloud portability right now.”