NASA OIG: More Stringent Entrance Criteria Needed for Project Life-Cycle Reviews (Orion Project)
The Issue
In January 2004, the President announced a Vision for U.S. Space Exploration that directed NASA to retire the Space Shuttle by 2010 and develop new launch vehicles for missions beyond low Earth orbit. To achieve the Vision, one of NASA’s first challenges is the acquisition and development of the Orion Crew Exploration Vehicle (Orion). In September 2006, NASA awarded a $3.9 billion contract to Lockheed Martin Corporation (Lockheed) for the design, development, testing, and evaluation of Orion with a period of performance through December 2011. In April 2007, the Agency and Lockheed negotiated a change to the contract effectively increasing the contract value to $4.3 billion and extending the performance period to October 2013.
Since the 1980s, NASA has had difficulty bringing a number of projects to completion including the building of a next-generation spacecraft. The Agency spent an estimated $4.8 billion and years of effort on never completed projects such as the National Aero- Space Plane, X-33, X-34, Space Launch Initiative, and the International Space Station Propulsion Module. To improve project outcomes, the Government Accountability Office (GAO) issued several reports recommending that NASA develop sound acquisition concepts to capture specific product knowledge at key junctures in product development prior to proceeding with the project. As a result, NASA revised its program and project management guidance to require program and project “life cycles” with life-cycle reviews at key decision points.
We initiated this audit because of the overall importance of the Orion Project and because it is one of the first space flight projects to implement the revised guidance. We evaluated NASA’s management of the Orion acquisition during the initial life-cycle phase established by NASA Procedural Requirements (NPR) 7120.5D, “NASA Space Flight Program and Project Management Requirements,” March 6, 2007, and the lifecycle reviews that occur during the initial life-cycle phase, “Concept and Technology Development” (Phase A).
Results
The Orion Project Office (Project Office) conducted a Phase A life-cycle review with a vehicle configuration (606 vehicle) that was not at the proper maturity level to proceed to Phase B. Specifically, a required engineering design analysis conducted prior to the life-cycle review disclosed that the vehicle configuration required a reduction in weight, power, and instrumentation. However, instead of delaying the Phase A life-cycle review until the correct vehicle configuration (607 vehicle) could be reviewed, the Project Office proceeded with a nonconforming vehicle. As a result, a significant portion of the vehicle configuration that eventually did proceed to Phase B did not receive the benefit of a Phase A life-cycle review, nor was it completely evaluated for compliance with requirements.
NASA policy requires space flight projects to conduct life-cycle reviews during each phase of the project’s life cycle. These reviews are considered essential elements of conducting, managing, evaluating, and approving space flight projects. The Project Office conducted each of the required Phase A life-cycle reviews for projects–the System Requirements Review (SRR) and the System Definition Review (SDR)–in accordance with NPR 7120.5D. The Orion Standing Review Board (SRB) conducted an independent life-cycle review–the Preliminary Non-Advocate Review (PNAR)–which provides the Agency an expert assessment of the progress against the project baseline.
As part of Orion’s systems engineering process, the Project Office performs analysis cycle reviews during and after each life-cycle review. Included in Phase A is a design analysis cycle (DAC) that integrates the analysis required to support the SDR and demonstrates that the proposed system design and operational concept meet the mission goals and objectives. The pre-SDR DAC analysis disclosed that the 606 vehicle configuration required a redesign to reduce weight, power, and instrumentation to fulfill the mission. In addition, the analysis showed that because the vehicle did not meet the mass and weight requirements, the 606 vehicle would not be at the maturity level to proceed to Phase B, “Preliminary Design and Technology Completion.”
NPR 7123.1A, “NASA Systems Engineering Processes and Requirements,” March 26, 2007, establishes the entrance criteria that each project needs to fulfill and the success criteria that the project must successfully demonstrate for each life-cycle review. NPR 7123.1A entrance criteria do not require known changes from engineering design analysis be incorporated into the relevant technical baseline products prior to holding the life-cycle review. Therefore, the Project Office could hold the SDR, on schedule, on the nonconforming vehicle instead of delaying the SDR until meeting the Constellation Program (CxP) mass and weight requirements. The Project Office entered the SDR process with the nonconforming vehicle configuration because it initiated a parallel “point-of-departure” (POD) activity, which it believed would ultimately satisfy NPR 7120.5D requirements with the least expenditure of Agency resources.
By conducting the SDR with the 606 vehicle configuration and declaring the SDR a success, the Project Office (and the SRB) communicated the impression that the 606 vehicle was at the maturity level needed to proceed to Phase B of the project’s life cycle, when it was not. In addition, the Agency was obligated to pay Lockheed its award fee of $41.4 million, which was based, in part, on the SDR being held by August 31, 2007. The Project Office was able to conduct the review with the nonconforming vehicle because NASA’s life-cycle reviews’ entrance criteria do not require that the project incorporate the results from engineering analysis cycles conducted prior to the life-cycle review.
Without a fully successful SDR, the acquisition methodology is materially undermined and the Agency cannot be assured that life-cycle milestones vital to success are achieved. NASA needs to ensure that the vehicle configuration and requirements are at the proper maturity level prior to starting and completing future life-cycle phases. By not doing so, there is increased risk that new development is conducted prematurely, with concomitant risk of costly rework and schedule slips in managing the multibillion dollar Orion Project.
Management Action
Our September 9, 2008, draft of this report recommended that NASA revise NPR 7123.1A entrance criteria for internal life-cycle reviews to require that the technical products reviewed incorporate into the technical baseline known requirement changes resulting from engineering analysis cycle assessments. In addition, to ensure that the 607 vehicle configuration was scrutinized at a minimum of the Phase A SDR level, we recommended that the Agency evaluate the vehicle configuration to determine the impact of any missed work and, if material, perform an SDR on affected systems. Finally, to ensure that the contractor does not prematurely receive award fees, we recommended the Agency ensure the contractor receive award fees based on the maturity of the relevant technical baseline rather than on holding a scheduled milestone.
Management’s Comments and OIG Response.
In response to a draft of this report (see Appendix G, “Management Comments”), the Chief Engineer did not concur with our recommendation to revise NPR 7123.1A entrance criteria, stating that the current policy is effective as written. The Chief Engineer also stated that there is no technical baseline of the design at this point in the life cycle. Trade studies and technology maturation are still occurring such that by the Preliminary Design Review (PDR), a preliminary design is in place that meets all system requirements.
We consider management’s comment to this recommendation to be nonresponsive. We disagree that the current policy is effective. The policy as written provides for what amounts to a system status review rather than a life-cycle review to examine whether NASA had achieved a milestone in project development. While system status is useful, it is not what we believe is intended by the life-cycle review process, which focuses on technical maturity as opposed to calendar milestones. If the technical maturity established pursuant to criteria does not exist, a life-cycle review should not be declared a “success.” Material deficiencies should be addressed and resubmitted for review.
The policy as written allowed the Orion Project Office to hold the SDR on a nonconforming vehicle configuration that did not meet known requirements nor was the vehicle able to meet the review’s success criteria. In addition, although there may not be a technical baseline for the actual design available for review at the SDR, NASA’s guidance is clear that the Project is developing the technical baseline during this lifecycle phase. We commend the Project Office for delaying its PDR on two different occasions (and its consideration of a third delay), to ensure that the relevant technical baseline is reviewed. We believe these delays are consistent with making the PDR a true life-cycle review. However, without a revision to NPR 7123.1A requiring review of the relevant technical baseline at each life-cycle review, instances might again arise when reviews are performed on a premature configuration in compliance with the entrance criteria, yet be unable to meet known technical requirements or the success criteria. We request that the Chief Engineer reconsider his response to this recommendation and provide additional comments by November 26, 2008.
The Associate Administrator for Exploration Systems Mission Directorate did not concur with our recommendation to evaluate the Phase B vehicle configuration to assure that it was scrutinized at a minimum of the SDR level, stating that no additional review is warranted. However, the Associate Administrator did concur with our recommendation that the contractor receives award fees based on the maturity of the relevant technical baseline, stating that NASA had already acted accordingly for the first Orion award fee period covered during the audit.
Although the Associate Administrator for Exploration Systems Mission Directorate did not concur with our recommendation to evaluate the Phase B vehicle configuration, the Agency’s actions are responsive to the intent of our recommendation. Technical subject matter experts and Agency stakeholders eventually evaluated the 607 vehicle configuration, albeit outside of the formal SDR process. We believe the Agency’s delay of the Phase B PDR because the known technical baseline changes are not at the proper maturity level is consistent with our recommendation. Likewise, although we disagree with the Associate Administrator’s assertion that NASA acted in accordance with the recommendation during award fee period one, Agency actions taken during award fee period two are in accordance with the recommendation. NASA adjusted the period two award fee milestone to coincide with the delayed PDR and by so doing, NASA demonstrated that the Orion Project acted in accordance with the intent of our recommendation. We consider these recommendations resolved and closed.
