With brands ranging from Yahoo to HuffPost, TechCrunch, and many others, Verizon Media is focused on entertaining, informing, and connecting people.
But as the portfolio grew, connecting all of the brands’ infrastructure became a challenge. When Verizon Media acquired Yahoo in 2017, “We had n number of stacks managed by n number of teams, each using their own tools and platforms, and some of them were containerized,” says Suresh Visvanathan, Senior Director of Engineering. Additionally, the company had its own in-house, on-prem, homegrown solution for pushing and deploying code.
The Verizon Media team decided to move toward immutable infrastructure. “We wanted to take the same piece of code and run it everywhere we needed without any modification,” says Visvanathan.
The team evaluated a number of orchestration solutions, and “Kubernetes solved many of our use cases and fit our requirements nicely compared to other platforms,” he says. As a proof of concept, they built a small Kubernetes cluster and ran one very critical application on it: Yahoo Sports. “We ran 5% of the traffic on the small Kubernetes cluster, and we compared the legacy platform versus the new platform to make an informed, data-driven decision. With Kubernetes, velocity was much quicker, deploying was much faster, and containerizing was pretty much easier. Based on the success we saw in the small 5% of the workload, we decided to march forward with Kubernetes as our orchestration layer for all things containers.”
Today, Verizon Media has more than 1,400 services running on Kubernetes, with 34 production-grade clusters managed across seven different data centers. At the peak, the platform handles 2 million requests per second. Moving services into containers and using Kubernetes as an acquisition platform “helped us to reduce duplication, standardize the way we build and deploy code, and increased our cluster utilization,” says Visvanathan.
The platform also uses a number of other CNCF technologies. The entire core components are monitored through Prometheus. With OPA, which is used to enforce custom policy on Kubernetes Objects, enabled the team to reduce its validation workflow from hundreds of lines of core code to just a few lines of code. Jaeger enables the team to provide a distributed tracing platform and helps with debugging and optimizing. “Bringing in all those cloud native tools helped us modernize our stack,” says Visvanathan.
With this platform, “We make it a lot easier for developers,” says Mujib Wahab, Senior Director of Engineering Platform Organization. The team built a templating engine, which simplified the developer experience and helped drive the adoption of the platform. “All they need to do is define their YAML, and the platform takes care of the rest,” Wahab adds. “That’s a huge win for developer productivity.”
With the legacy infrastructure, signed host tokens along with the client IP address validation was used for authentication. But to uphold these zero-trust security principles in a dynamic infrastructure like Kubernetes, the team realized that they needed a short-lived certificate-based identity. At that time, a suitable system did not exist, so they built Athenz, a platform for X.509 certificate-based service authentication and fine-grained authorization, and open-sourced it.
With Athenz integrated into all the Kubernetes workloads, Verizon Media produces 3+ million certificates a day. Visvanathan says, “Athenz enriches Kubernetes workload security at Verizon Media with fine grain RBAC and Service Authentication. Athenz’s rich set of APIs integrates seamlessly with any Container as a Service Platform.” Wahab adds, “Security is now from the ground up. By default, everything is secured.”Read more about Verizon Media’s cloud native journey in the full case study.