With Kubernetes, the U.S. Department of Defense is enabling DevSecOps on F-16s and battleships
Challenge
In the recent past, software delivery within the U.S. Department of Defense could take anywhere from three to ten years for big weapons systems. “It was mostly teams using waterfall, no minimum viable product, no incremental delivery, and no feedback loop from end users,” says Nicolas M. Chaillan, Chief Software Officer of the U.S. Air Force. “Particularly when it comes to AI, machine learning, and cybersecurity, everyone realized we have to move faster.”
Solution
Chaillan and Peter Ranks, Deputy Chief Information Officer for Information Enterprise, DoD CIO, created the DoD Enterprise DevSecOps reference design, with a mandate to use CNCF-compliant Kubernetes clusters and other open source technologies across the DoD.
Impact
Releases, which once took as long as 3 to 8 months, now can be achieved in one week. An authority to operate (ATO) for a cloud enclave can be obtained within one week, plus “we have a continuous ATO on the platform stack,” says Chaillan. “Anytime it’s going to pass the gates, the software is automatically accredited. So you can push software multiple times a day.” All told, “we’re thinking with the 37 programs, it’s going to be a 100-plus years saved off planned program time,” he adds.
Projects used
By the numbers
Releases
Went from 3-8 months to 1 week
Overall time saved
100+ years across 37 programs
The MVP to install Kubernetes on an
F-16 was completed in 45 days
Before DevSecOps came to the U.S. Department of Defense, software delivery could take anywhere from three to ten years for big weapons systems.
“It was mostly teams using waterfall, no minimum viable product, no incremental delivery, and no feedback loop from end users,” says Nicolas M. Chaillan, Chief Software Officer of the U.S. Air Force. Plus, “cybersecurity was mostly an afterthought.”
Chaillan was brought in to turn the ship around in the summer of 2018, and his solution was simple, if revolutionary for the department. He and Peter Ranks, Deputy Chief Information Officer for Information Enterprise, DoD CIO, created the DoD Enterprise DevSecOps reference design, with a mandate to use CNCF-compliant Kubernetes clusters and other open source technologies. “The DoD Enterprise DevSecOps reference design defines the gates on the DevSecOps pipeline,” says Chaillan. “As long as teams are compliant with that reference design, they can get a DoD-wide continuous ATO (authority to operate).”
Kubernetes was chosen “as an abstraction layer for us, so we know it’s going to behave the same,” he says. “The value for us was in the abstraction, the orchestration, the resiliency, and the self-healing.” Envoy and Istio provided a control and data plane, which “is critical so we have no drift between environments, because we have multiple classified environments,” he adds.
The initiative was presented as a way to “learn fast, fail fast, and don’t fail twice for the same reason,” Chaillan says. “Particularly when it comes to AI, machine learning, and cybersecurity, everyone realized we have to move faster.”
In order to drive adoption, teams were selected to build Minimum Viable Products (MVPs) using cloud native best practices. Some teams chose simple applications to demonstrate that it could be done. But Chaillan took a different route: “I tackled the weapon systems so General Officer and Senior Executives will pay attention, and that’s where you usually end up getting the funding. So if you get people excited and show you can do it, then you can demonstrate there is something there.”
So in the fall of 2019, the SoniKube team based at Hill Air Force Base in Utah set out to get Kubernetes running on an F-16 jet. Members of the DoD’s Platform One team, led by Jeff McCoy, were embedded with the group to teach them how to put Kubernetes on the jet’s legacy hardware. “We had to be able to boot Kubernetes with Istio on the jet within two minutes, because that’s a requirement for the jet if something goes wrong, and it has to be able to spin back up within two minutes,” says Chaillan. “That was the biggest challenge.”
Within 45 days, the team accomplished that goal, and were able to do a demo on the jet for Dr. Will Roper, Assistant Secretary of the Air Force for Acquisition, Technology and Logistics. “We got the cluster on Istio running and then we launched five or six microservices,” says Chaillan. “A lot of the jet runs in older programming languages, and so being able to run Go, Python, and Java was pretty exciting.”
A total of 37 teams are currently working on building applications on top of Kubernetes: “We have teams doing this at every side of the weapons systems, from the space systems to the nuclear systems to the jets,” he says.
“The DoD Enterprise DevSecOps reference design defines the gates on the DevSecOps pipeline. As long as teams are compliant with that reference design, they can get a DoD-wide continuous ATO (authority to operate).”
— NICOLAS M. CHAILLAN, CHIEF SOFTWARE OFFICER, U.S. AIR FORCE
While some of these programs are classified, others have been able to share their results. For instance, the United Platform is working to push cyber offense and defense capabilities on the cloud. “We’ve been using Kubernetes to orchestrate the stack and containerize the tools we’re building for the last six months,” Chaillan says. “Unified Platform has been very successful with delivering rapid iterations of containers.”
The Navy’s Compile to Combat in 24 Hours program is another example: “Kubernetes is enabling them to do a push to production on the ship while being disconnected [at sea],” Chaillan says. The Kessel Run team “is delivering value to the warfighter on different types of devices and iPads and other cool edge use cases in a matter of weeks.”
Armed with those success stories, Chaillan advocated for enterprise-level offerings that would be available across the DoD. Two teams were created: Cloud One, to bring a cloud environment to development teams, and Platform One, to set up DevSecOps environments on Kubernetes clusters. Teams are allowed to choose from a selection of distributions, such as OpenShift, Tanzu, D2IQ, and Rancher.
With these cloud native technologies and practices in place, the DoD has achieved great time savings. Releases, which once took as long as 3 to 8 months, now can be achieved in one week. An ATO for a cloud enclave can be obtained within one week, plus “we have a continuous ATO on the platform stack,” says Chaillan. “Anytime software passes the testing and security gates, the software is automatically accredited. So you can push software multiple times a day.”
“We already took a team that had never done Kubernetes before, brought a few people in and trained them, put them off-base in a commercial building, and built the F-16 MVP within 45 days. That’s something that probably would have taken a year or two before.”
— NICOLAS M. CHAILLAN, CHIEF SOFTWARE OFFICER, U.S. AIR FORCE
Another benefit is that containers can now be accredited DoD-wide. “Instead of each team doing the accreditation, we can do the accreditation, one time, centrally,” says Chaillan, “and that can go all the way to the classified environments.”
All told, “we’re thinking with the 37 programs, it’s going to be a 100-plus years saved off planned program time,” he adds.
Additionally, bringing DevSecOps to all environments, both classified and unclassified, “is a really big enabler,” Chaillan says. “They can run all the classification levels where the data is, which is where you need it. So having that turnkey where you can go and have the environment ready, we just awarded 55 contracts for cloud services, talent, and licenses. So if a team like F-16 comes and says, ‘I need cloud computing, storage, a Kubernetes distribution, a source code repo, a bunch of different tools for the CI/CD pipeline, and five DevSecOps engineers,’ they can get all that within 30 days.” That process used to take 3 to 6 months.
Chaillan’s next goal is to complete training for all 100,000 employees who work on DoD applications. That may sound daunting, but as Chaillan points out, “We already took a team that had never done Kubernetes before, brought a few people in and trained them, put them off-base in a commercial building, and built the F-16 MVP within 45 days. That’s something that probably would have taken a year or two before.”