NASA OIG: A-02-005-00 Independent Verification and Validation of Software
Full Report (PDF)
Independent verification and validation (IV&V) is a critical management control for minimizing the risk of software-related, catastrophic mission failure. The audit focused on the effectiveness of NASA’s procedures for ensuring that the appropriate level of IV&V is performed on its software development projects. Details regarding the audit objectives, background, scope, and methodology are in Appendix B.
We found that NASA had not effectively ensured that all applicable software development projects were assessed to determine their appropriate level of IV&V.
As a result of this condition, the safety, quality, and reliability of some of the Agency’s programs and projects, including mission-critical programs and projects, could be compromised. For example, the Fluids and Combustion Facility (FCF) project, managed by Glenn Research Center, was a likely candidate for independent software reliability assurance. The FCF will be a permanent multi-rack research laboratory for conducting microgravity experiments onboard the International Space Station. The $72 million project consists of hardware and related software that is expected to host more than 100 fluids and combustion experiments over its anticipated lifespan. After we brought our concerns about this project to management’s attention, the NASA IV&V Facility told the project manager to assess the FCF project for IV&V; project management officials subsequently concluded that the FCF required IV&V. Without IV&V, the microgravity research program faced an unnecessary risk that the FCF project would not perform as intended.
Management Control Needed to Ensure Effective Application of IV&V Policy
NASA Policy Directive (NPD) 8730.4 requires that the IV&V Facility maintain “explicit involvement” with project managers in determining the appropriate level of IV&V for their software development projects. To ensure effective implementation of this requirement, the Agency should provide the IV&V Facility a complete list of all applicable software development projects. Facility personnel could then identify software projects for which IV&V assessment criteria should be applied and, in turn,
advise the cognizant project managers to apply the criteria to those projects. Details regarding NASA’s IV&V policy for software are in Appendix C. Absent a complete list of applicable software development projects, IV&V Facility personnel initiated their own software project identification process by contacting Center personnel and searching the Agency’s mission and project Web sites for information on software development activities. The Facility’s efforts were commendable but did not result in a complete and accurate list of software development projects.
IV&V Requirements Not Included in Current JPL Contract
NPD 8730.4 states that the Agency’s IV&V policy is applicable to JPL to the extent specified in the Agency’s contract with the California Institute of Technology. In this regard, NASA had not incorporated the NPD requirement into the current contract, and JPL’s internal policies did not require project managers to complete the Agency’s IV&V assessment criteria. Further information regarding the Agency’s IV&V policy relative to the California Institute of Technology contract is in Appendix D.
We discussed the Agency’s IV&V policy with JPL procurement officials, NASA Office of Space Science officials, and officials in the NASA Offices of Safety and Mission Assurance and Chief Engineer. NASA management told us that it would not be prudent or cost-effective to incorporate software IV&V requirements into the current JPL contract; however, management agreed to incorporate the requirements into the follow-on contract effective October 1, 2003. The follow-on contract was awarded in November 2002 and requires JPL to comply with NPD 8730.4.
Conclusion
NASA should apply effective management controls, on a recurring basis, to ensure the IV&V policy is fully implemented. Until needed corrective actions are implemented, NASA’s software management, engineering, and assurance processes will not be fully integrated with the Agency’s program and project management processes. Further, NASA has not ensured that it has developed risk analyses and risk management strategies at each stage of the software development life-cycle.
Recommendations for Corrective Action
1. The NASA Chief Engineer, in coordination with the Associate Administrator for Safety and Mission Assurance, should establish a process that provides the NASA IV&V Facility, on a recurring basis, a complete and accurate list of the Agency’s
programs and projects governed by either NASA Procedures and Guidelines 7120.5A or NASA Technical Standard 8719.13A.
2. The NASA Chief Engineer should verify that the NASA IV&V Facility initiates appropriate actions to ensure that the programs and projects identified in Recommendation 1 comply with the Agency’s software IV&V policy.
Management’s Response and our Evaluation of the Response
NASA concurred with the intent of the recommendations and has initiated corrective actions. Although we consider management’s comments (Appendix F) to be responsive to the recommendations, the success of the alternative actions to be taken will depend on effective coordination between the Program Management Councils and the IV&V Facility. Effective coordination will help ensure that the Agency has adequately considered the need for IV&V and has performed IV&V where appropriate. Details related to disposition and closure of the recommendations are in Appendix A.
