With an eye toward standardization and security for its media brands, Verizon Media turned to cloud native
Challenge
With a growing portfolio of media brands, “We had n number of stacks managed by n number of teams, each using their own tools and platforms, and none of them were containerized,” says Suresh Visvanathan, Senior Director of Engineering at Verizon Media. Additionally, the company had its own in-house, on-prem, homegrown solution for pushing and deploying code, and “it was painful and took a long time,” he says.
Solution
The Verizon Media team decided to move toward immutable infrastructure and containerization. After evaluating a number of orchestration solutions, they chose Kubernetes. The team also built and open-sourced a certificate-based authentication & authorization system, Athenz, to implement zero-trust security principles in the platform.
Impact
Moving services into containers and using Kubernetes as an acquisition platform “helped us to reduce duplication, standardize the way we build and deploy code, and increased our cluster utilization too,” says Visvanathan. With Athenz integrated into all the Kubernetes workloads, “Security is now from the ground up,” says Mujib Wahab, Senior Director of Engineering Platform Organization. “The developers don’t need to worry; by default, everything is secured.”
Projects used
By the numbers
Cluster utilization
Increased 20%
Number of services
1,400
Requests per second
2 million at peak
With brands ranging from Yahoo to HuffPost, TechCrunch, and many others, Verizon Media is focused on entertaining, informing, and connecting people.
But as the portfolio grew, connecting all of the brands’ infrastructure became a challenge. When Verizon Media acquired Yahoo in 2017, “We had n number of stacks managed by n number of teams, each using their own tools and platforms, and some of them were containerized,” says Suresh Visvanathan, Senior Director of Engineering. Additionally, the company had its own in-house, on-prem, homegrown solution for pushing and deploying code.
The Verizon Media team decided to move toward immutable infrastructure. “We wanted to take the same piece of code and run it everywhere we needed without any modification,” says Visvanathan.
The team evaluated a number of orchestration solutions, and “Kubernetes solved many of our use cases and fit our requirements nicely compared to other platforms,” he says. As a proof of concept, they built a small Kubernetes cluster and ran one very critical application on it: Yahoo Sports. “We ran 5% of the traffic on the small Kubernetes cluster, and we compared the legacy platform versus the new platform to make an informed, data-driven decision. With Kubernetes, velocity was much quicker, deploying was much faster, and containerizing was pretty much easier. Based on the success we saw in the small 5% of the workload, we decided to march forward with Kubernetes as our orchestration layer for all things containers.”
Today, Verizon Media has more than 1,400 services running on Kubernetes, with 34 production-grade clusters managed across seven different data centers. At the peak, the platform handles 2 million requests per second. Moving services into containers and using Kubernetes as an acquisition platform “helped us to reduce duplication, standardize the way we build and deploy code, and increased our cluster utilization,” says Visvanathan.
The platform also uses a number of other CNCF technologies. The entire core components are monitored through Prometheus. With OPA, which is used to enforce custom policy on Kubernetes Objects, enabled the team to reduce its validation workflow from hundreds of lines of core code to just a few lines of code. Jaeger enables the team to provide a distributed tracing platform and helps with debugging and optimizing. “Bringing in all those cloud native tools helped us modernize our stack,” says Visvanathan.
“The entire culture has changed. If there’s a use case, on public cloud or on-prem, where a service can run in Kubernetes, developers automatically go with that.”
— Mujib Wahab, Senior Director of Engineering Platform Organization at Verizon Media
With this platform, “We make it a lot easier for developers,” says Mujib Wahab, Senior Director of Engineering Platform Organization. The team built a templating engine, which simplified the developer experience and helped drive the adoption of the platform. “All they need to do is define their YAML, and the platform takes care of the rest,” Wahab adds. “That’s a huge win for developer productivity.”
From a platform perspective, says Wahab, “We also wanted to make sure that it’s completely modern and secure, so we follow core zero-trust security principles: Anything between the client, a component, or our services should be encrypted while in transit. Every client has to have a unique identity, so we can do a mutual TLS authentication, and whatever action it’s trying to perform on the resource has to be authorized. And whatever trust we give to that entity is limited and time-sensitive so that the trust ends when whatever the client is performing ends.”
With the legacy infrastructure, signed host tokens along with the client IP address validation was used for authentication. But to uphold these zero-trust security principles in a dynamic infrastructure like Kubernetes, the team realized that they needed a short-lived certificate-based identity. At that time, a suitable system did not exist, so they built Athenz, a platform for X.509 certificate-based service authentication and fine-grained authorization, and open sourced it.
“Kubernetes helped us to reduce duplication, standardize the way we build and deploy code, and increased our cluster utilization too.”
— Suresh Visvanathan, Senior Director of Engineering at Verizon Media
“It’s a huge benefit for the entire industry,” Wahab says. “We constantly look to see how we can improve and harden our security. We provide a certificate for every single workload or client running out there. The corresponding private keys are unique per pod and the issued service identity certificates are short-lived and automatically rotated by the identity agents. In addition to mTLS authentication, Athenz issued standards-based mTLS bound access tokens are used by services to authorize all incoming requests.”
With Athenz integrated into all the Kubernetes workloads, Verizon Media produces 3+ million certificates a day. Visvanathan says, “Athenz enriches Kubernetes workload security at Verizon Media with fine grain RBAC and Service Authentication. Athenz’s rich set of APIs integrates seamlessly with any Container as a Service Platform.” Wahab adds, “Security is now from the ground up. By default, everything is secured.”
All told, Verizon Media’s cloud native platform has impacted the company immensely. “The entire culture has changed,” says Wahab. “If there’s a use case, on public cloud or on-prem, where a service can run in Kubernetes, developers automatically go with that.” And with the platform came the benefits of CNCF, says Visvanathan: “CNCF established helpful standards and has an engaged and helpful community. Collaborating and contributing has been a great experience.”