Before DevSecOps came to the U.S. Department of Defense, software delivery could take anywhere from three to ten years for big weapons systems.
“It was mostly teams using waterfall, no minimum viable product, no incremental delivery, and no feedback loop from end users,” says Nicolas M. Chaillan, Chief Software Officer of the U.S. Air Force. Plus, “cybersecurity was mostly an afterthought.”
Chaillan was brought in to turn the ship around in the summer of 2018, and his solution was simple, if revolutionary for the department. He and Peter Ranks, Deputy Chief Information Officer for Information Enterprise, DoD CIO, created the DoD Enterprise DevSecOps reference design, with a mandate to use CNCF-compliant Kubernetes clusters and other open source technologies. “The DoD Enterprise DevSecOps reference design defines the gates on the DevSecOps pipeline,” says Chaillan. “As long as teams are compliant with that reference design, they can get a DoD-wide continuous ATO (authority to operate).”
Kubernetes was chosen “as an abstraction layer for us, so we know it’s going to behave the same,” he says. “The value for us was in the abstraction, the orchestration, the resiliency, and the self-healing.” Envoy and Istio provided a control and data plane, which “is critical so we have no drift between environments, because we have multiple classified environments,” he adds.
The initiative was presented as a way to “learn fast, fail fast, and don’t fail twice for the same reason,” Chaillan says. “Particularly when it comes to AI, machine learning, and cybersecurity, everyone realized we have to move faster.”
In order to drive adoption, teams were selected to build Minimum Viable Products (MVPs) using cloud native best practices. Some teams chose simple applications to demonstrate that it could be done. But Chaillan took a different route: “I tackled the weapon systems so General Officer and Senior Executives will pay attention, and that’s where you usually end up getting the funding. So if you get people excited and show you can do it, then you can demonstrate there is something there.”
So in the fall of 2019, the SoniKube team based at Hill Air Force Base in Utah set out to get Kubernetes running on an F-16 jet. Members of the DoD’s Platform One team, led by Jeff McCoy, were embedded with the group to teach them how to put Kubernetes on the jet’s legacy hardware. “We had to be able to boot Kubernetes with Istio on the jet within two minutes, because that’s a requirement for the jet if something goes wrong, and it has to be able to spin back up within two minutes,” says Chaillan. “That was the biggest challenge.”
Within 45 days, the team accomplished that goal, and were able to do a demo on the jet for Dr. Will Roper, Assistant Secretary of the Air Force for Acquisition, Technology and Logistics. “We got the cluster on Istio running and then we launched five or six microservices,” says Chaillan. “A lot of the jet runs in older programming languages, and so being able to run Go, Python, and Java was pretty exciting.”
A total of 37 teams are currently working on building applications on top of Kubernetes: “We have teams doing this at every side of the weapons systems, from the space systems to the nuclear systems to the jets,” he says.
To find out more about the Department of Defense’s cloud native journey, read the full case study.