Case Study

Utmost

Zero Trust Networking with Cilium

Challenge

Utmost handles sensitive personal data and needed SOC 2 Type II attestation, ISO 27001 certification, and zero trust networking to demonstrate their commitment to security and data privacy as core values of their business.

Solution

They implemented Cilium as their CNI for network policies to default deny all traffic and created automated pipelines for their developers to create new policies. They also implemented Hubble for visibility and auditing of their network.

Impact

Utmost has implemented zero trust networking while still being able to keep pace with their development. They currently handle over 4,000 flows per second and can audit their infrastructure.

Challenges:

Auditing, Automation, Compliance, Security

Industry:

Extended Workforce Management

Location:

Ireland

Cloud Type:

Public

Published:

August 31, 2022

Projects used

Utmost provides a worldwide SaaS extended workforce solution

They differ from the legacy Vendor Management System solutions by changing the conversation from one that is just focused on vendor management to one focused on how work gets done with an extended workforce – managing a series of intersecting and changing relationships between suppliers, workers, and the enterprise. Their IT team is no different. With a fully cloud based platform run by a team of four skilled system engineers supporting multiple teams of developers, they need to carefully manage their infrastructure and relationships to allow everyone to focus on their work.

Utmost runs all of its infrastructure in the cloud on AWS, but it is stretched around the globe across multiple regions to service their customers where they are. Since they are completely cloud-based and responsible for holding sensitive data, security is paramount for the trust their customers put in them. Utmost’s key objective was to build layers of security on top of a zero-trust base. Being a small team they also needed their infrastructure to be stable, robust, and have built in automation to keep operations running smoothly.

They started building out their infrastructure on Kubernetes, but found it placed additional demands on security as the Kubernetes network is open to everyone by default. They also needed support for their required levels of control, visibility, monitoring, compliance, and auditing and to be able to integrate into their security gates and fail-safes. With all of this integration, ongoing management also became a critical factor so that future implementations and scaling up did not generate escalating levels of complexity to challenge support resources.

When looking for solutions to implement zero trust networking, they knew it was not something that they could just buy off the shelf. They needed tools that would help them change the way that they worked and build a layered security approach so there weren’t any single points of failure. They were also conscious of the performance overhead of any solution to control costs. eBPF came out as an early leader because of its power in the kernel, but eBPF rules are not user-friendly so they needed an abstraction that could manage it in a way that could be more widely utilized by the company. Finally, to ensure everything flowed through the network correctly, they needed visibility of the traffic and integration into their monitoring systems for when things go wrong.

Utmost considered a few different options, but wanted an open source project with good documentation, an active community with many adopters, and good support around it to ensure the maturity and longevity. They found the Cilium community to be responsive on Slack and Github and were impressed with the project.

“Cilium ticked all the boxes for us in terms of maturity, stability, performance, visibility, debugging, monitoring and the list goes on. We were also impressed with the responsiveness of the community. You push something to Github and it doesn’t just die there. I think we can kind of pat ourselves on the back that we made the right choice with Cilium.”

Andrew Holt, Senior Systems Engineer

Making Cilium and Hubble the standard for Zero Trust Security

With Kubernetes as the core of their infrastructure, everything is fully automated for deployment which makes upgrading and downgrading easy for them. They replaced their existing CNI with Cilium and now use Terraform, Helm and scripting for the deployment and management. When they started using Cilium, they chose a default deny network policy. All environments, regardless of the tier, were now in a zero trust configuration, but no predefined policies also meant no access. They had to map out and write a lot of rules to make sure everything could connect to what it needed to. 

Once they got the initial framework set up, they started to involve the developers in the process too. It began with developing a pipeline that gave teams the ability to self-manage their network by creating or modifying existing policies. Once the policy is written, it gets validated and tested for any errors. If it passes the tests, the policy is then sent for review by one of the system engineers before an automated deployment into production.

“We don’t want everything landing on the systems engineers. We needed the ability for our developers to understand the rule sets themselves as well as how to update them for what they need and validate the fix. We’ve seen it with banks and other companies where they implement something, but the maintenance of that thing becomes a nightmare and ends up strangling developers. They can’t get their changes through without bumping their head against red tape. We wanted to compliment the service, not take away from it. Developers should be able to deploy when and where they need to and Cilium enables our developers to do this.”

Andrew Holt, Senior Systems Engineer

In addition to Cilium, Utmost also uses Hubble for visibility, debugging, and auditing. Hubble was easily integrated into their existing Prometheus monitoring stack. “Having a UI and being able to pump those metrics into our monitoring tools is amazing because if there is an issue, attack, or any kind of troubleshooting, you need to be able to see what’s going on,” says Holt. Hubble is also key for their developers so they can understand when and where new components are being added and how they are connecting to the existing infrastructure. Finally on the auditing side, being able to see all of the transactions and actually prove what is happening, including a nice visualization, makes it so much easier to complete.

Future Roadmap

Utmost is always looking for ways to improve its infrastructure to serve their customers better. In addition to being active on Github issues and feature requests for Cilium, they are also looking at some of Cilium’s more advanced functionality. They are considering using Cilium for load balancing and to replace kube-proxy to simplify their infrastructure and increase performance. mTLS for encryption is also something they want to examine when it comes out in the next release. Zero trust is never a thing you can achieve, it must be constantly worked on from the infrastructure to the application and Utmost plans to keep evolving with Cilium to keep their customer’s data safe.