Passes with flying colors
Istio’s ambient mode splits the service mesh into two distinct layers: Layer 7 processing (the “waypoint proxy”), which remains powered by the traditional Envoy proxy; and a secure overlay (the “zero-trust tunnel” or “ztunnel”), which is a new codebase, written from the ground up in Rust.
It is our intention that the ztunnel project be safe to install by default in every Kubernetes cluster, and to that end, it needs to be secure and performant.
We comprehensively demonstrated ztunnel’s performance, showing that it is the highest-bandwidth way to achieve a secure zero-trust network in Kubernetes — providing higher TCP throughput than even in-kernel data planes like IPsec and WireGuard — and that its performance has increased by 75% over the past 4 releases.
Today, we are excited to validate the security of ztunnel, publishing the results of an audit of the codebase performed by Trail of Bits.
We would like to thank the Cloud Native Computing Foundation for funding this work, and OSTIF for its coordination.
Scope and overall findings
Istio has been assessed in 2020 and 2023, with the Envoy proxy receiving independent assessment. The scope of this review was the new code in Istio’s ambient mode, the ztunnel component: specifically code relating to L4 authorization, inbound request proxying, transport-layer security (TLS), and certificate management.
The auditors stated that “the ztunnel codebase is well-written and structured”, and had no findings relating to vulnerabilities in the code. Their three findings — one of medium severity and two of informational — refer to recommendations regarding external factors, including software supply chain and testing.
Resolution and suggested improvements
Improving dependency management
At the time of the audit, the cargo audit report for ztunnel’s dependencies showed three versions with current security advisories. There was no suggestion that any vulnerable code paths in ztunnel dependencies could be reached, and the maintainers would regularly update the dependencies to the latest appropriate versions. To streamline this, we’ve adopted GitHub’s Dependabot for automated updates.
The auditors pointed out the risk of Rust crates in the dependency chain of ztunnel that either unmaintained or maintained by a single owner. This is a common situation in the Rust ecosystem (and indeed all of open source). We replaced the two crates that were explicitly identified.
Enhancing test coverage
The Trail of Bits team found that most ztunnel functionality is well-tested, but identified some error-handling code paths which were not covered by mutation testing.
We evaluated the suggestions and found that the gaps in coverage highlighted by these results apply to test code, and to code that does not affect correctness.
While mutation testing is useful to identify potential areas to improve, the goal is not to get to a point where a report returns no results. Mutations can trigger no test failures in a number of expected cases, such as behavior with no ‘correct’ result (e.g., log messages), behavior that impacts only performance but not correctness (measured outside of the scope the tooling is aware of), code paths that have multiple ways to achieve the same result, or code used only for testing. Testing and security is a core priority for the Istio team and we are constantly improving our test coverage — using tools like mutation testing and by developing novel solutions to testing proxies.
Hardening HTTP header parsing
A third-party library was used for parsing the value of the HTTP Forwarded header, which may be present on connections made to ztunnel. The auditors pointed out that header parsing is a common area of attack, and expressed concern that the library we used was not fuzz tested. Given that we were only using this library for parsing one header, we wrote a custom parser for the Forwarded header, complete with a fuzzing harness to test it.
Get involved
With strong performance and now validated security, ambient mode continues to advance the state of the art in service mesh design. We encourage you to try it out today.
If you would like to get involved with Istio product security, or become a maintainer, we’d love to have you! Join our Slack workspace or our public meetings to raise issues or learn about what we are doing to keep Istio secure.