NASA OIG Audit of NASA’s Joint Cost and Schedule Confidence Level Process
WHY WE PERFORMED THIS AUDIT
Throughout its history, NASA has struggled with accurately predicting the amount of time and money required to complete its space flight projects. The resulting cost and schedule overruns have in turn led to challenges in the project development process, diversion of funding from other projects, and an overall reduction in the number and scope of projects the Agency can undertake. Over the years, studies have identified several root causes for NASA’s challenges in producing accurate cost and schedule estimates. While some of the causes are outside the Agency’s control, NASA has developed tools that can improve the fidelity of its cost and schedule estimates. To this end, since 2006 NASA has incorporated progressively more sophisticated probabilistic estimating techniques into Agency policy, culminating in 2009 with formal adoption of a Joint Cost and Schedule Confidence Level (JCL) requirement.
A JCL analysis generates a representation of the likelihood a project will achieve its objectives within budget and on time. The process uses software tools and models that combine cost, schedule, risk, and uncertainty to evaluate how expected threats and unexpected events affect a project’s cost and schedule. To generate this data, project managers develop comprehensive project plans, inputs, and priorities that integrate costs, schedules, risks, and uncertainties. NASA officials contend that gathering this data encourages better communication among project personnel; improves cost, schedule, risk, and uncertainty analyses; and fosters an understanding of how project elements impact one another. Accordingly, a JCL analysis not only establishes the basis for proposing project and program budgets, but may improve project planning and provide stakeholders the rigor and documentation to better justify funding requests. Since 2009, NASA has completed a JCL analysis for 22 projects with a combined price tag of more than $49 billion.
We initiated this audit to determine whether NASA had implemented appropriate controls and procedures to establish a JCL process capable of improving cost and schedule estimates and therefore providing more reliable information to decision makers.
WHAT WE FOUND
Based on our review of these 22 projects, it appears the JCL policy is having a positive impact on NASA’s historical challenges with cost and schedule fidelity. That said, the process is relatively new, still evolving, and not a one-stop solution to solving all root causes of cost overruns and schedule delays. Specifically, the process has inherent limitations in that, like any estimating practice, it does not fully address the issue of predicting “unknown/unknowns” or address some of the root causes of NASA’s project management challenges such as funding instability and underestimation of technical complexity.
We identified varied expectations and understandings among Agency stakeholders about the JCL process, ranging from those who see JCL as a multifunctional tool that can significantly improve cost and schedule management to others who view it as just another task projects must complete before moving into the development phase. We also identified issues with the quality of some JCL cost, schedule, and risk data inputs for several of the projects we reviewed. In-depth assessments of 9 of the 22 projects revealed 5 projects that had significant weaknesses in project scheduling, risk assessment, and cost estimating. Remedying these weaknesses would improve the overall accuracy of JCL analyses.
Moreover, the effectiveness and consistency of the process NASA uses to review projects’ JCL analyses could be improved. For example, the extent and type of review varied widely from project to project. We attributed this inconsistency to a lack of formal guidance, inadequate training for review board members, and inconsistent expectations among the review board chairs regarding how projects should consider and incorporate the results of board reviews. We also found training for project personnel could be improved.
Finally, the confidence levels stipulated in the JCL policy may not be suitable for single-project programs, which cannot leverage funding from other projects in the same portfolio that finish under budget. Accordingly, holding those programs to the levels stipulated in the policy may not be appropriate.
WHAT WE RECOMMENDED
To improve the Agency’s JCL process, we made eight recommendations to NASA: (1) clarify that project managers and Decision Authorities are to use JCL results as the basis for proposing and establishing project budgets rather than as a validation tool; (2) assess the effectiveness of the scheduling function at NASA and develop a plan to ensure all NASA Centers have access to trained and qualified schedulers with experience commensurate with the complexity of assigned projects; (3) require use of historical data in JCL analyses; (4) establish formal guidance and clarify expectations for the review process; (5) establish a formal, JCL-specific training program for involved personnel; (6) work with JCL software providers to add a function that tracks and creates a report reflecting modifications to input data and require review boards to consider this information; (7) assess the appropriateness of the current confidence level requirement for single-project programs and consider clarifying or supplementing that requirement; and (8) require projects to include all identified, relevant, and discrete development risks with potential cost and/or schedule impacts in their JCL models.
In response to a draft of our report, the Acting Director of the Office of Evaluation concurred with seven of our recommendations and described corrective actions the Agency has or will take. The Acting Director did not concur with our recommendation to add a function to JCL software that would track and create a report reflecting modifications to input data. However, the Agency’s proposal to work with JCL software vendors to implement other features and functions that can aid with input data organization and verification is potentially responsive to our recommendation. Accordingly, although we continue to encourage NASA to further assess the economic and operational feasibility of adding a data input tracking and reporting function to the JCL software, we consider all recommendations resolved.
