GAO Report: NASA: Long-Term Commitment to and Investment in Space Exploration Requires More Knowledge

Download full report GAO-06-817R with enclosures

July 17, 2006

The Honorable Sherwood Boehlert Chairman
The Honorable Bart Gordon Ranking Minority Member
Committee on Science
House of Representatives

Subject: NASA: Long-Term Commitment to and Investment in Space Exploration Program Requires More Knowledge

The National Aeronautics and Space Administration (NASA) plans to spend nearly $230 billion over the next two decades implementing the Vision for Space Exploration. In January 2006, NASA publicly released its Exploration Systems Architecture Study (ESAS), which is an effort to identify the best architecture and strategy to implement the President’s 2004 Vision for Space Exploration (Vision).¹ The cost estimate for implementing the ESAS through fiscal year 2011 exceeds $31 billion. The estimate through fiscal year 2018 is over $122 billion, and the estimate through fiscal year 2025 is nearly $230 billion.² These estimates include the architecture, robotic precursor missions, supporting technologies, and funding needed to service the International Space Station (ISS).³ NASA plans to implement this architecture through a “go as you can afford to pay” approach, wherein lower-priority efforts would be deferred, descoped, or discontinued to allow NASA to stay within its available budget profile. This approach assumes NASA’s budget will increase moderately to keep pace with inflation. Given the long-term fiscal imbalances that will challenge the entire federal government now and in the future, it would be prudent for NASA to establish a program that reduces the risk that significant additional funding, beyond moderate increases for inflation, will be required to execute the program.⁴ Government leaders will have to make difficult decisions to resolve such challenges, and the debate over the potential cost and the federal government’s role in implementing the Vision are emblematic of the challenges the nation will need to resolve in the years ahead. Because of the significance of this investment, competing demands on the federal discretionary budget, and the importance of the success of NASA’s exploration program to the future of U.S. human spaceflight, you requested that we assess (1) the extent to which NASA has identified the architecture and costs necessary to implement the Vision, (2) whether NASA’s exploration architecture cost estimates fit within the agency’s projected available budgets, and (3) the risks associated with NASA’s acquisition strategy for the CEV project.

We presented our preliminary findings to your staff in May 2006. Because of your committee’s interest in how NASA is implementing the Vision for Space Exploration, we are enclosing the full briefing that supported that May presentation with this report (see encl. II), along with a summary of our findings and conclusions. We are recommending that the NASA Administrator modify the current CEV acquisition strategy to ensure that the agency does not commit itself, and in turn the federal government, to a long-term contractual obligation prior to establishing a sound business case at the project’s preliminary design review. In written comments, NASA nonconcurred with our recommendation and stated that it has the appropriate level of knowledge to proceed with its current acquisition strategy. As a result of its nonconcurrence, we are including as a matter for congressional consideration that the Congress should consider restricting NASA’s appropriations and obligations for the CEV project to only the amount of funding necessary to successfully complete the project’s preliminary design review.

Background

The Vision includes plans to explore the moon, Mars, and beyond.⁵ The first step in implementing the Vision is to retire the space shuttle after completing assembly of the ISS by the end of the decade. NASA currently plans to retire the space shuttle in 2010, creating a potential gap in U.S. human spaceflight of up to 4 years before development of the CEV and the CLV is complete. Congress has voiced concern over the United States not having continuous access to space, and NASA has made it a priority to minimize the gap by accelerating the CEV project to have it in service as close to 2010 as possible. NASA’s Exploration Systems Mission Directorate’s (ESMD) Constellation program is responsible for the development of both the CEV and the CLV. NASA awarded concept development contracts for the CEV project to both Lockheed Martin and Northrop Grumman in July 2005 and plans to award a contract for design, development, production and sustainment in September 2006. That contract could extend through 2019. For the CLV, NASA plans to award a sole-source contract for the first stage of the CLV to ATK-Thiokol, the manufacturer of the Shuttle’s Reusable Solid Rocket Motor, in October 2006. Also, the agency plans to award Pratt & Whitney Rocketdyne, the developer of the Space Shuttle Main Engine (SSME) and J-2 engines, a sole-source contract for development of the J-2X engine in November 2006. These contractors are currently planning their respective efforts under interim contract arrangements. NASA has started in-house preliminary design work on the CLV upper stage structures and avionics and plans to begin awarding competitive contracts for production of these items in May 2007.

Despite many successes in the exploration of space, such as landing the Pathfinder and Exploration Rovers on Mars, the loss of life, unsuccessful missions, and unforeseen cost overruns have recently increased the level of concern over the benefits of such exploration, particularly with regard to human spaceflight activities. NASA has had difficulty bringing a number of projects to completion, including several efforts to build a second generation of reusable human spaceflight vehicle to replace the space shuttle. NASA has attempted several expensive endeavors such as the National Aero-Space Plane, the X-33 and X-34, and the Space Launch Initiative, among others. While these endeavors have helped to advance scientific and technical knowledge, none have completed their objective of fielding a new reusable space vehicle. We estimate that these unsuccessful development efforts have cost approximately $4.8 billion since the 1980s. The high cost of these unsuccessful efforts and the potential costs of implementing the Vision make it important that NASA achieve success in its new exploration program.

Our past work has shown that developing a sound business case, based on matching requirements to available and reasonably expected resources before committing to a new product development effort, reduces risk and increases the likelihood of successful outcomes.⁶ At the heart of a business case is a knowledge-based approach to product development that is a best practice among leading commercial firms and successful government system developers. For a program to increase its chances of delivering a successful product, high levels of knowledge should be demonstrated before managers make significant program commitments. In essence, knowledge supplants risk over time. This building of knowledge can be described as three levels that should be attained over the course of the program:

(1) At program start, the customer’s needs should match the developer’s available resources in terms of availability of mature technologies, time, human capital, and funding.

(2) Midway through development, the product’s design should be stable and demonstrate that it is capable of meeting performance requirements.

(3) By the time of the production decision, the product must be shown to be producible within cost, schedule, and quality targets, and have demonstrated its reliability.

Our work has shown that programs that have not attained the level of knowledge needed to support a sound business case have been plagued by cost overruns, schedule delays, decreased capability, and overall poor performance. With regard to NASA, we have reported that in some cases the agency’s failure to define requirements adequately and develop realistic cost estimates—two key elements of a business case—resulted in projects costing more, taking longer, and achieving less than originally planned.

Summary

Although NASA is continuing to refine its exploration architecture cost estimates, the agency cannot at this time provide a firm estimate of what it will take to implement the architecture. The absence of firm cost estimates is mainly due to the fact that the program is in the early stages of its life cycle. According to NASA cost-estimating guidance, early life cycle phase estimates are generally based upon parametric models, which use data from projects with similar attributes to predict cost because there are usually many unknowns and actual cost or performance data are not available. NASA preliminarily identified the resources needed to implement the architecture as outlined in the architecture study primarily through the use of such models. NASA conducted a cost risk analysis of its preliminary estimates through fiscal year 2011. On the basis of this analysis and through the addition of programmatic reserves (20 percent on all development and 10 percent on all production costs), NASA is 65 percent confident that the actual cost of the program will either meet or be less than its estimate of $31.2 billion through fiscal year 2011. For the cost estimates for beyond 2011, when most of the cost risk for implementing the architecture will be realized, NASA has not applied a confidence level distinction. Since NASA released its preliminary estimates, the agency has continued to make architecture changes. For example, following the issuance of the architecture study, NASA conducted several analysis cycles during which various aspects of the architecture have evolved, such as the diameter of the CEV, the engine used to support the upper stage of the CLV, and the size of the Reusable Solid Rocket Booster on the CLV. While these changes, and others, are appropriate for this phase of the program, when concepts are still being developed, they leave the agency in the position of being unable to firmly identify program requirements and needed resources, which can also be expected at this phase of the program. According to NASA officials, once they receive more detailed contractor inputs, the agency will be able to produce higher-fidelity estimates of program cost. NASA plans to commit to a firm cost estimate at the preliminary design review (PDR) in 2008, when the program’s requirements, design, and schedule will all be baselined.

NASA will be challenged to implement the architecture recommended in the study within its projected budget. Whether using the architecture study estimates of funds available or NASA’s Fiscal Year 2007 Budget Submission for ESMD that was based on the architecture study cost estimates, there are years when NASA does not have sufficient funding to implement the architecture. Some yearly shortfalls exceed $1 billion, while in other years the funding available exceeds needed resources. NASA maintains that the architecture could be implemented within the projected available budgets through fiscal year 2011 when funding is considered cumulatively. In addition, NASA preliminarily projects multibillion-dollar shortfalls for ESMD in all fiscal years from 2014 to 2020, with an overall deficit through 2025 of over $18 billion. In the short term, NASA is attempting to address this problem within the Constellation program by redirecting funds to that program from other ESMD activities to provide a significant surplus for fiscal years 2006 and 2007 to cover projected shortfalls beginning in fiscal year 2009. In addition, the Constellation program has requested more funds than required for its projects in several early years to cover shortfalls in later years. For example, the Exploration Communication and Navigation Systems project within the Constellation program plans to roll over $56.2 million from the fiscal year 2007 budget to make up for budget shortfalls in fiscal years 2008, 2009, and 2010. NASA officials stated the identified budget phasing problem could worsen given that changes made to the exploration architecture following issuance of the study will likely add to the near-term development costs, where the funding is already constrained. In addition, NASA’s estimates beyond 2010 are based upon a surplus of well over $1 billion in fiscal year 2011 due to the retirement of the space shuttle fleet in 2010. However, NASA officials said the costs for retiring the space shuttle and transitioning to the new program are not fully understood, and thus the expected surplus could be less than anticipated.

NASA’s current acquisition strategy for the CEV places the project at risk of significant cost overruns, schedule delays, and performance shortfalls because it commits the government to a long-term product development effort before establishing a sound business case. NASA plans to award a contract for the design, development, production, and sustainment of the CEV in September 2006—before it has developed key elements of a sound business case, including well-defined requirements, a preliminary design, mature technology, and firm cost estimates. The period of performance for the contract scheduled for award in September 2006 will extend through at least 2014, with the possibility of extending through 2019. This contract will comprise all design, development, and test and evaluation activities, including production of ground and flight test articles and at least four operational CEVs. Although NASA is committing to a long-term contract, it will not have the elements of a sound business case in place until the project level PDR in fiscal year 2008. Awarding a contract for design, development, production, and sustainment of the project as NASA has planned places the CEV project at increased risk of cost growth, schedule delays, and performance shortfalls. At PDR, NASA will likely (a) have the increased knowledge necessary to develop a sound business case that includes high-fidelity, engineering-based estimates of life cycle cost for the CEV project, (b) be in a better position to commit the government to a long-term effort, and (c) have more certainty in advising Congress on required resources.

Implementing the Vision over the coming decades will require hundreds of billions of dollars and a sustained commitment from multiple Administrations and Congresses over the length of the program. The realistic identification of the resources needed to achieve the agency’s short-term goals would provide support for such a sustained commitment over the long term. With a range of federal commitments binding the fiscal future of the United States, competition for resources within the federal government will only increase over the next several decades. Consequently, it is incumbent upon NASA to ensure that it is wisely investing its existing resources. As NASA begins to implement the Vision with several key acquisition decisions planned to occur this fall, it will be essential that the agency ensure that the investment decisions it is making are sound and are based upon high levels of knowledge. NASA should make the prudent decision now to ensure that it has attained the appropriate level of knowledge to support a sound business case before it commits to the project. However, under the current acquisition strategy for CEV, key knowledge—including well-defined requirements, a preliminary design, mature technology, and firm cost estimates—will not be known until over a year after the expected contract award date. Nevertheless, NASA plans to commit the government to a long-term contract. This approach increases the risk that the project will encounter significant cost overruns, schedule delays, and decreased capability. Given the nation’s fiscal challenges and those that exist within NASA, the availability of significant additional resources to address such issues, should they occur, is unlikely. With the impending decisions pertaining to the CEV, NASA has the opportunity to establish a firm foundation for the entire Constellation program by ensuring that the appropriate level of knowledge is available before proceeding with its acquisition strategy and committing the government to a long-term design, development, and production effort.

Recommendation for Executive Action

Because of the importance of the CEV project to NASA’s overall implementation of the Vision, NASA should focus on ensuring that its acquisition approach for the CEV project does not place the government at risk by committing to a long-term design and development effort without the knowledge needed to make wise investment decisions. We therefore recommend that the NASA Administrator modify the current CEV acquisition strategy to ensure that the agency does not commit itself, and in turn the federal government, to a long-term contractual obligation prior to demonstrating, through the establishment of a sound business case at the project’s preliminary design review, that the project is affordable and executable.

Matter for Congressional Consideration

Based on its response to our report, it appears that NASA plans to proceed with its acquisition strategy for the CEV and award a long-term contract for the project, although it continues to lack sufficient knowledge and a sound business case for doing so. Congress is currently being asked to approve NASA’s fiscal year 2007 funding request and will be asked to approve fiscal year 2008 and perhaps the fiscal year 2009 funding requests for the CEV project before NASA has demonstrated such knowledge and has provided evidence, based on that knowledge, that the project will be executable within existing and expected resources. In light of the fact that NASA plans to award the contract for the CEV in September 2006, Congress should consider restricting annual appropriations and limiting NASA’s obligations for the CEV project to only the amount of funding necessary to support activities needed to successfully complete the project’s preliminary design review.

Agency Comments and Our Evaluation

In written comments on a draft of this report (see encl. I), NASA nonconcurred with our recommendation that it modify the current CEV acquisition strategy to ensure that the agency does not commit itself, and in turn the federal government, to a longterm contractual commitment prior to establishing a sound business case at the project’s preliminary design review. NASA stated that it has the appropriate level of knowledge to proceed with its acquisition plan to “down select” to a single Crew Exploration Vehicle prime contractor in September 2006. NASA added that it is maximizing competition by soliciting from industry a development, production, and management approach with an emphasis on life cycle cost. In the area of technology maturity, NASA stated that it has a plan and process in place to address the Thermal Protection and Landing subsystems technology risks through in-house development work and collaboration with the prime contractor. NASA also noted that during its design, development, and test and evaluation effort, the agency will be using an enditem award fee, which would make all award fees subject to a final evaluation to determine how well the product met requirements, including cost and schedule.

The CEV acquisition strategy is not knowledge-based in that it calls for maturing technologies, designing systems, and preparing for initial production concurrently— an approach that our work has shown carries the increased risk of cost and schedule overruns and decreased technical capability. Therefore, we disagree with NASA’s statement that it has the appropriate level of knowledge to proceed with its current acquisition strategy and award a long-term contract for the project prior to obtaining sufficient knowledge. Specifically:

• In its response, NASA suggests that there would be no benefit in retaining two prime contractors for the CEV project through the preliminary design review and that the best return on its investment would be gained by down-selecting to one contractor and awarding the contract in September 2006. Contrary to NASA’s response, addressing our recommendation would not preclude the agency from down-selecting to one contractor. The thrust of our recommendation is that NASA should lessen the government’s obligation to the project at such an early stage when realistic cost estimates have yet to be established and requirements are not fully defined, and therefore limit the scope of the contract to activities needed to successfully complete the preliminary design review. At that point the project should have in place a sound business case for proceeding and hence be in a better position to justify continued investment. Implementation of the recommendation could be accomplished through various means, including by retaining two contractors through the preliminary design review and awarding a contract at that time or by down-selecting as planned in September 2006 and limiting the scope of the contract as described above.
• NASA’s suggestion that it is maximizing competition by soliciting from industry its development, production, and management approach and that it will receive firm competitive prices from industry for completion of development and demonstration of two vehicles has little basis. First, while the current structure will allow for competition in the short term, the benefits of such competition will be short-lived. Without well-defined requirements, mature technologies, an approved preliminary design, and realistic cost estimates, NASA has insufficient information to ensure that it is obtaining firm competitive prices for the work conducted for the entirety of Schedule A— especially for activities beyond the project’s preliminary design review. Because NASA continues to refine the project’s requirements, as demonstrated by the numerous changes to the exploration architecture as discussed in our report, it cannot provide a firm estimate of project cost. Without such information, it will likely be difficult for NASA to establish realistic “not-toexceed” prices for Schedule B activities. Under the current strategy, NASA will not have high-fidelity, engineering-based estimates of life cycle costs for the CEV until the preliminary design review. As outlined in this report, projects with cost estimates based on early, evolving designs and top-level requirements are at increased risk of cost growth relative to estimates based on mature designs and detailed requirements—which could be achieved at the preliminary design review. According to NASA, it plans to obtain this and further knowledge about program cost, schedule, and risk elements following the contract award and in conjunction with the contractor. In the absence of such information, it is not clear how NASA can substantiate its statement that it has the knowledge necessary to commit to activities beyond the project’s preliminary design review. Further, it cannot provide Congress with assurance of the appropriateness of requested funding for the project.
• NASA stated that its current acquisition strategy for the CEV minimizes the government’s obligation during development by dividing the CEV contract into three separate schedules. All three schedules, however, will be awarded in September 2006 as part of one contract. Although NASA plans to include language in the negotiated CEV contract to state that the minimum quantity under Schedule B will not be applicable until that schedule’s period of performance begins in 2009—a step that would lessen the government’s obligation during production—it will continue to be responsible for all Schedule A activities at the time of contract award. These activities include all design, development, and test and evaluation activities, as well as the production of two operational vehicles. Contractually obligating the government to even these Schedule A activities, before it has established a sound business case to support such a commitment, is not in line with our knowledge-based approach and is ultimately not in the best interest of the government.
• NASA’s investment in identifying and maturing the Thermal Protection and Landing Subsystems is a step in the right direction to ensure that these technologies are mature and available when needed. NASA has no guarantee, however, that these critical technologies will be mature by the time of the project’s preliminary design review—the point at which our work has shown that technologies should be mature in order to decrease the risk of cost and schedule growth. NASA’s proposed commitment to the project for activities beyond the preliminary design review before retiring these technology risks increases the likelihood that the project will experience schedule delays and cost overruns.
• NASA maintains that program risks have been marginalized and that the agency will utilize incentives, including end-item award fees, to ensure contractor performance. NASA suggests that the incentives it plans to use in the form of end-item award fees will be a powerful tool for meeting cost schedule, technical, and quality goals. The use of these tools, however, does not compensate for proceeding with a risky acquisition, nor do they lessen NASA’s responsibility to implement an executable program from the start. For them to function as intended, NASA needs to address the more fundamental issues related to its acquisition strategy, including its lack of a sound business case for the CEV project.
• Finally, the use of cost-reimbursable contracting, while appropriate for early development and design efforts, places most of the cost risk for the project on the government. Given the nature of this effort, it is likely that the project will change significantly as it moves forward. Therefore, any scope changes or schedule slips could translate into additional contract cost for NASA. Such cost impacts could be minimized if NASA limited its contractual obligation to those activities needed to achieve a successful preliminary design review, as we recommended. In addition, limiting the scope of the CEV contract would allow both NASA and Congress to assess the project’s progress at the preliminary design review and to decide if continued investment in the project is prudent and in the best interest of the government.

It is important to note that Congress will continue to be asked to make funding commitments in advance of CEV project events that would demonstrate that the project has the knowledge necessary to support a sound business case. Specifically, NASA’s funding request for fiscal years 2007 and 2008 are scheduled to be approved before the CEV holds its preliminary design review. Since the preliminary design review is currently scheduled for March 2008, this may also be the case for fiscal year 2009. Congress should safeguard against a situation in which contractual and budget decisions could hinder its ability to tie further investments in the CEV project to demonstrated progress at the preliminary design review. As such, we have included a matter for congressional consideration.

We also received technical comments from NASA, which have been addressed in the report, as appropriate.

Scope and Methodology

To assess the extent to which NASA has identified the architecture and costs necessary to implement the Vision and whether NASA’s exploration architecture cost estimates fit within the agency’s projected available budgets, we reviewed and analyzed NASA’s Exploration Systems Architecture Study, fiscal year 2007 budget request, ground rules and assumptions provided from the Constellation program to project level management estimators to perform the bottom up review, guidance for use in preparing the fiscal year 2008 budget request, NASA cost-estimating guidance in the NASA Cost Estimating Handbook, and congressional hearings and testimonies pertaining to NASA and the Vision. We also conducted interviews with NASA headquarters officials from the Cost Analysis Division, the Exploration Systems Mission Directorate, and Constellation program officials, Constellation program and CEV project officials at Johnson Space Center; CLV project officials at Marshall Space Flight Center; and cost analysts from the Kennedy Space Center. During these interviews, we discussed the methodologies used in preparing the ESAS and subsequent cost estimates, architecture changes after the ESAS and the trades being considered, budgeting issues, and procurement strategies and activities.

To assess the risks associated with NASA’s acquisition strategy for the CEV project, we reviewed and analyzed CEV project documentation, including draft project plans, draft requirements documents, technology development plans, documentation included in the contract request for proposals, and past NASA human spaceflight acquisition programs. We compared NASA’s plans for the CEV with criteria contained in GAO best practices work on systems acquisition. We also conducted interviews with NASA headquarters officials from the Exploration Systems Mission Directorate and Constellation Systems officials, Constellation program and CEV project officials at Johnson Space Center, and CLV project officials at Marshall Space Flight Center. We conducted our work from January 2006 to May 2006 in accordance with generally accepted government auditing standards.

As agreed with your offices, unless you announce its contents earlier, we will not distribute this report further until 10 days from its date. At that time, we will send copies of the report to NASA’s Administrator and interested congressional committees. We will also make copies available to others upon request. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov. Should you or your staff have any questions on matters discussed in this report, please contact me at (202) 512-4841 or lia@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Principal contributors to this report were James L. Morrison, Assistant Director; Rick Cederholm; Shelby S. Oakley; Guisseli Reyes; Sylvia Schatz; and

John S. Warren, Jr.
Allen Li
Director
Acquisition and Sourcing Management

Enclosures

Footnotes

¹ The ESAS architecture supports the development of a new Crew Exploration Vehicle (CEV), Crew Launch Vehicle (CLV), a Cargo Launch Vehicle (CaLV), and other supporting systems. The architecture also calls for various Research and Technology (R&T) and Robotic Lunar Exploration Program (RLEP) projects.

² All cost estimates related to the Vision are reported as inflated (“real year”) dollars.

³ NASA’s cost estimate through 2011—$31.2 billion—included the costs of the R&T and RLEP projects needed to support the architecture. Its estimate for the first lunar landing—$104 billion—did not include $18 billion in funding for R&T and RLEP projects. To ensure consistency, the estimates for 2018 and 2025 are presented with R&T and RLEP funding included.

⁴ GAO, 21st Century Challenges: Reexamining the Base of the Federal Government, GAO-05-325SP (Washington, D.C.: Feb. 2005); 21st Century: Addressing Long-Term Fiscal Challenges Must Include a Reexamination of Mandatory Spending, GAO-06-456T (Washington, D.C.: Feb. 15, 2006); and Highlights of a GAO Forum: The Long-Term Fiscal Challenge, GAO-05-282SP (Washington, D.C.: Feb. 1, 2005).

⁵ The Vision includes a return to the moon that is intended ultimately to enable future exploration of Mars and other destinations. To accomplish this, NASA initially plans to (1) complete its work on the International Space Station by 2010, fulfilling its commitment to 15 international partner countries; (2) begin developing a new manned exploration vehicle to replace the space shuttle; and (3) return to the moon no later than 2020 in preparation for future, more ambitious missions.

⁶ Examples of our best practices reports include GAO, Best Practices: Using a Knowledge-Based Approach to Improve Weapon Acquisition, GAO-04-386SP (Washington, DC.: Jan. 2004); Space Acquisitions: Committing Prematurely to the Transformational Satellite Program Elevates Risks for Poor Cost, Schedule, and Performance Outcomes, GAO-04-71R (Washington, D.C.: Dec. 4, 2003); Best Practices: Capturing Design and Manufacturing Knowledge Early Improves Acquisition Outcomes, GAO-02-701 (Washington, D.C.: Jul. 15, 2002); and Best Practices: Better Matching of Needs and Resources Will Lead to Better Weapon System Outcomes, GAO-01-288 (Washington, DC.: Mar. 8, 2001).

Download full report with enclosures