NASA OIG Audit of the Spaceport Command and Control System for SLS and Orion
WHY WE PERFORMED THIS AUDIT
NASA’s Ground Systems Development and Operations (GSDO) Program is responsible for preparing the Kennedy Space Center (Kennedy) to launch the next generation of rockets and spacecraft, including the Space Launch System (SLS) and Orion Multi-Purpose Crew Vehicle (Orion) NASA plans to use for deep space exploration. To accomplish this mission, the GSDO Program must move vehicles to launch pads, manage and operate the equipment required to connect spacecraft with rockets, and send the integrated vehicles into space. As part of this effort, the GSDO Program is developing the Spaceport Command and Control System (SCCS) a software system that will control pumps, motors, valves, power supplies, and other ground equipment; record and retrieve data from systems before and during launch; and monitor the health and status of spacecraft as they prepare for and launch. To create the SCCS, NASA is writing a large amount of computer code to “glue” together multiple existing software products or, in some cases, the parts of those products the Agency deems most effective for its purposes.
In the past, NASA has experienced difficulties with similar large, complex software development efforts. For example, between 1995 and 2002, the Agency spent more than $500 million on two separate attempts to update command and control software at Kennedy. Both efforts failed to meet their objectives and were substantially scaled back or cancelled prior to completion.
In this audit we examined whether NASA is effectively managing the SCCS software development effort. To complete this objective, we performed work at Kennedy, interviewed GSDO Program officials and commercial companies involved with command and control software, and reviewed various studies concerning the SCCS, Federal laws, and NASA policies.
WHAT WE FOUND
The SCCS development effort has significantly exceeded initial cost and schedule estimates. Compared to fiscal year 2012 projections, development costs have increased approximately 77 percent to $207.4 million and the release of a fully operational version has slipped by 14 months from July 2016 to September 2017. In addition, several planned capabilities have been deferred because of cost and timing pressures, including the ability to automatically detect the root cause of specific equipment and system failures. Without this information, it will be more difficult for controllers and engineers to quickly diagnose and resolve issues. Although NASA officials believe the SCCS will operate safely without these capabilities, they acknowledge the reduced capability could affect the ability to react to unexpected issues during launch operations and potentially impact the launch schedule for the combined SLS-Orion system.
The root of these issues largely results from NASA’s implementation of its June 2006 decision to integrate multiple products or, in some cases, parts of products rather than developing software in-house or buying an off-the-shelf product. Writing computer code to “glue” together disparate products has turned out to be more complex and expensive than anticipated. As of January 2016, Agency personnel had developed 2.5 million lines of “glue-ware,” with almost two more years of development activity planned. In comparison, NASA reengineered the Hubble Space Telescope command and control system with approximately 500,000 lines of “glue-ware” code.
Audit of the Spaceport Command and Control System March28,2016 IG-16-015 (A-15-008-00)SCCS Project managers told us the decision to develop SCCS in this manner was motivated by several factors. Managers did not want to rely on a single company’s software because if that company encountered financial difficulties or stopped providing technical support NASA’s space exploration efforts could be negatively impacted. In addition, at the time the decision was made, managers believed the effort to integrate the various software products would not be overly time-consuming or technically complex. While that decision may have been reasonable based on what managers knew at the time, it is now clear they underestimated the complexity of the software integration activities that would be required.
In the past, NASA has encountered difficulties with large and complex command and control software development efforts, failing on two occasions to meet expected requirements despite spending more than $500 million. In something of a repeat of this pattern, the SCCS development effort is more than 1 year behind schedule and significantly over cost, and several planned software capabilities have been deferred.
NASA made its decision regarding the SCCS software architecture nearly 10 years ago, but in our view this may no longer be the most prudent course of action given significant advances in commercial command and control software over that time. For example, the two companies under contract with NASA to deliver supplies to the International Space Station Orbital Sciences Corporation and Space Exploration Technologies both use commercial software products to accomplish their missions. In our judgment, the GSDO Program’s reluctance to change course reflects a cultural legacy at NASA of over-optimism and over-promising what the Agency can achieve in a specific timeframe.
WHAT WE RECOMMENDED
In a draft of this report, we recommended the Associate Administrator for Human Exploration and Operations commission an independent assessment to evaluate the status of the SCCS software development effort and determine the necessary steps to reduce the risk of further cost, schedule, and performance issues, including consideration of acquiring commercial command and control software to replace some or all of the system currently under development.
NASA agreed to conduct an independent assessment of the command and control system once software for Exploration Mission-1 the first launch of the combined SLS-Orion system scheduled for November 2018 is successfully delivered. We consider management’s plan responsive to our recommendation. Therefore, the recommendation is resolved and will be closed upon completion and verification of the proposed corrective action.
