From: Stafford-Covey Return to Flight Task Group (SCTG)
Posted: Friday, January 28, 2005
Download Full Report (PDF)
As NASA approaches the first launch of the Space Shuttle since the Columbia accident (presently scheduled for the May-June 2005, launch window), it has become clear to the Return to Flight Task Group (RTF TG) that the recommendations of the Columbia Accident Investigation Board (CAIB) need to be considered not just individually, but also as a collection of activity that will result in reduced risk to the continued flights of the Shuttles and their crews. An ancillary benefit of all the activity thus far is the increased understanding of the remaining risk of return to flight and beyond. (See the risk evaluation framework included on pages 11-14.)
Various members of the CAIB have suggested their recommendations could be summarized as: 1) make the maximum effort possible to improve the Shuttle's safety related-performance; 2) establish and promote open communications within the Agency; and 3) go fly again. Neither the CAIB nor the RTF TG expects that risk can be eliminated. We have often heard the safest Shuttle is one that never leaves the ground.
NASA has not interpreted the CAIB recommendations to be a checklist, but rather has in many cases undertaken activities that far exceeded the intent of CAIB. In other instances, technological and other barriers have thus far prevented the kind of progress CAIB had hoped for, and NASA has striven for. Taken together, the RTF TG believes it is entirely possible NASA will be able to make sufficient progress on the CAIB recommendations before their current launch date. Several activities will nonetheless be incomplete, and several issues raised by the CAIB, such as scheduling and resources, are timeless. Other oversight bodies, such as the Aerospace Safety Advisory Panel (ASAP) will be called upon to pick up the agenda after return to flight. Obviously, Congress will continue its usual oversight of NASA as well.
It is important to reiterate: NASA, and not the RTF TG, will have to ultimately determine if the remaining risk is sufficiently low to justify the return to flight. The RTF TG's charter is limited to the evaluation of NASA's implementation of the 15 CAIB recommendations for return to flight. We will not make a determination of the safety or reliability of the next flight. Despite press reports to the contrary, only NASA can make that determination.
Summary of Plenary
Over the course of the three days (December 14-16, 2004), it was determined NASA has made substantial progress on meeting the CAIB recommendations for return to flight. The panels recommended, and the RTF TG approved, the complete closure of six recommendations and the conditional closure of one additional recommendation.
However, considerable work remains. Eight items remain open including some of the toughest technological challenges the recommendations present: shedding of debris, strengthening the Reinforced Carbon-Carbon (RCC), hardening of the Orbiter, and repair of thermal protection tile and RCC. Most of the operational issues have been addressed, with the largest remaining concern remains to create the potential use of the Space Station as a viable "safe haven" for the crew of a damaged Shuttle while a rescue mission can be mounted. NASA has also made substantial progress on the various management issues the CAIB cited as "half" the cause of the demise of Columbia, but sufficient detail of plans, exercise of new capabilities and responsibilities, and adequate documentation remain open issues.
Several of the CAIB return to flight recommendations involve enhanced imagery of the Shuttle during launch and while on orbit. NASA has made sufficient progress on imagery to allow the RTF TG to fully or conditionally close three of the recommendations (3.4-1, 3.4-2, 6.3-2) and note substantial progress on a fourth (3.4-3, which will be formally considered as part of 6.4-1, On- Orbit Inspection and Repair).
Taken together, the changes in the capability to observe and examine the Shuttle on launch will allow a more complete evaluation of the adequacy of the design and process changes made to the External Tank (ET) in the reduction of critical debris. The enhanced imagery will also contribute to the ability to focus on-orbit inspections. There will undoubtedly be foam shed from the ET during the next and subsequent launches. The questions will be: how large are the pieces, where on the tank did the shedding occur, and where did the debris impact? The ascent imagery will help answer these questions.
Some months ago, it became clear to the RTF TG the immense amount of new data, much in the form of imagery, would require a new approach to integration. The RTF TG therefore constituted a sub-panel for Integrated Vehicle Assessment. In response, NASA formed a development team that has produced a Thermal Protection System (TPS) Operations Integration Plan (OIP) intended to allow the Mission Management Team (MMT) to make a timely entry readiness, repair, or safe haven determination. The latest version of the OIP, while benefiting from further simulation and testing, is very robust and a potential model for other integration activities within the Shuttle Program as well as the Agency.
Two recommendations affecting closeout procedures were also dispositioned at the plenary (4.2-3 and 10.3-1). "Closeout" refers to the process of finalizing work on the Shuttle, often in an area that is then sealed from further view or inspection. The requirement for "two-person" closeouts is simply intended to add an additional pair of eyes to the evaluation of the completed work before being sealed. The requirement for "digitized closeout photos" is intended to yield an adequate ability to both examine work after closeouts and the ability to easily recall the images, particularly while the Shuttle is on orbit.
During the course of their investigation, the CAIB uncovered a technical deficiency in the bolt catcher, a device that prevents the explosive bolts used to mate the Solid Rocket Boosters (SRB) to the ET from becoming debris that might impact the Orbiter (4.2-1). NASA has successfully redesigned, tested, and requalified the bolt catcher.
Although most of the management-related recommendations remain open, NASA has made substantial progress since the last plenary. Most notably, the response to Recommendation 7.5-1 to create an Independent Technical Authority (ITA) has been formulated and implementation has begun. The first "warrants," the official delegation of ITA to specific individuals, have been issued.
The role of the MMT, which received much attention post-Columbia, has been clarified and expanded. The new MMT has conducted ten simulations of various aspects of the next mission and plans an end-to-end, full mission simulation beginning in late February and lasting several days.
The systems engineering and integration function, which the CAIB noted had atrophied over the course of time, has been reinvigorated and has an expansive role in return to flight. However, the RTF TG remains concerned that without adequate documentation the renewed vigor will dissipate after return to flight.
The Use of Analytical Models in Return to Flight
One way to view the loss of Columbia and her crew is in the analytical framework in which NASA must often work. That is, NASA must make assumptions from which can be derived "solutions" or "answers," the quality of which are highly dependent on those assumptions. It is simply the nature of highly technical, cutting edge endeavors.
It was, simplistically stated, two faulty assumptions that were direct causes of the Columbia tragedy: 1) foam insulation used on the ET cannot develop sufficient ballistic momentum to catastrophically damage the RCC on the wing leading edge; and 2) the aerodynamics of the wings, the airflow around the wings, will carry debris around/away from the leading edges of the wings. Both assumptions proved wrong, despite the successful completion of over 100 flights that seemed to validate these assumptions.
The RTF TG is concerned that NASA not replicate the reliance on faulty assumptions and the results of analytical models to justify return to flight. For example, significant progress has been made by the ET Project in improving both the design of the tank and the processes for the manual application of foam. These actions should serve to significantly reduce the risk of the liberation of critical debris during Shuttle operations. Many of these changes were made on the basis of TPS impact testing and debris flow modeling which has significantly improved the characterization and knowledge base associated with debris.
However, the testing and modeling of the debris flow and impact will not provide statistically significant absolute values nor provide the basis for making on-orbit damage assessments. As such, the current models cannot be used to precisely determine "allowable" debris nor precisely assess the magnitude of risk reduction. For return to flight, given the current state of model development and the remaining test program, additional analytical modeling is unlikely to provide a sound basis for additional design, process, or operations changes.
Similarly, modeling of debris liberation, flow, and damage, while providing engineering insight, cannot provide for the "certification" of flight hardware unless the models themselves undergo a rigorous process of validation and certification. NASA's determination of readiness and successful return to flight relies heavily on a full understanding of material condition, suitability for the intended operating environment, and clear assessment and acceptance of associated risk.
During a Program Requirements Control Board (PRCB) conducted April 15, 2004, the following definitions were presented as part of PRCB Directive S062235:
These definitions generally reflect widely held engineering and industry standards, even though slight variations may exist among certain disciplines. Central to the safe and reliable conduct of high risk complex technical endeavors is rigorous and consistent understanding of, and adherence to, these terms and the processes they describe.
This understanding and adherence also applies to methods leading to the end state (i.e., models and analysis tools utilized during validation). As an example, if one is to assert "validation has been accomplished through probabilistic analysis," the analysis must rest upon fundamental mathematical principles and undergo unflinching rigor. This rigor must include a predefined validation process for the tools and models utilized. This validation process must be founded on objective success criteria and the plan for validation documented and approved prior to undertaking the validation process. NASA has yet to demonstrate the rigor of the models necessary to certify the integrity of the Space Shuttle TPS, including the ET. Without validation of models, they should not be used for certification or risk assessment.
The RTF TG notes in the aftermath of the Challenger accident, a verification committee required for any Reusable Solid Rocket Motor changes that could not be tested and which changed flight configuration, verification required "two independent analytical models with a factor of safety of greater than 2.0."
The RTF TG also notes critical debris modeling is not yet complete and many requirements and current assumptions are based on preliminary debris modeling.
NASA's Determination of Readiness for Return to Flight
Risk acceptance and management are fundamental to leadership in high risk technical activities and is the leaders' ultimate responsibility. Space flight in general, and Shuttle operations in particular, are of such a nature that it is impossible to drive the risk to zero. While return to flight activities can be shown to reduce the risk, Shuttle operations will always be "accepted risk" operations. The basis for judgment on accepted risk relies upon a number of factors that are well accepted, understood, and documented.
NASA must be vigilant to prevent the development of a false sense of security by accepting faulty assumptions, or otherwise inappropriate analyses, to justify return to flight.
// end //