NASA OIG: Final Memorandum on the Review of Rocket Segment Handling

National Aeronautics and
Space Administration

Office of Inspector General
Washington, DC 20546-0001

September 5, 2008

TO: Associate Administrator for Space Operations

FROM: Assistant Inspector General for Auditing

SUBJECT: Final Memorandum on the Review of Rocket Segment Handling (Report No. IG-08-029; Assignment No. S-08-020-00)

The Office of Inspector General (OIG) conducted a review of the reporting procedures used by Alliant Techsystems, Inc. (ATK) for a rocket segment handling incident. We initiated this review in response to a complaint alleging that personnel at ATK did not follow proper reporting procedures subsequent to an incident involving the handling and movement of a rocket segment at ATK’s Component Refurbishment Center in Clearfield, Utah. Specifically, the complainant alleged that

• the rocket segment was inadvertently lifted off of its washout fixture;
• production and refurbishment of the segment was allowed to continue without adequate inspection and testing; and
• an inappropriate report was issued to document the incident. Noncompliance with reporting procedures raises potential safety concerns, including substandard handling techniques, oversight, and quality assurance of flight articles (program critical hardware). See the Enclosure for details on our scope and methodology.

Executive Summary

The allegation we received concerned an April 8, 2008, move¹ of an aft rocket segment in order to install a wash cover. A wash cover is a round stainless-steel plate that installed onto the aft segment of an expended rocket during the refurbishment process. During the removal of the tool used to maneuver the wash cover into place on the segment², is the lower crossbeam of the installation tool inadvertently engaged one of the wash cover’s installation brackets. Although this applied some degree of stress to the rocket segment and could have caused the segment to move, our review of the incident disclosed that the rocket segment had not been lifted off of its washout fixture. However, production and refurbishment of the segment continued without adequate inspection and testing. In addition, ATK initially reported the incident as a tooling issue, which did not involve the rocket segment, instead of a hardware issue.

A three-person ATK crew (the move crew) began the move at about 6 p.m. on April 8, 2008. When the installation tool failed to clear the wash cover’s installation bracket, the move crew immediately halted the operation, lowered the installation tool to relieve tension on the bracket, and notified the crew supervisor. The supervisor and the move crew conducted a visual inspection that revealed no damage to the installation tool or wash cover. The supervisor directed the crew to document the incident and resume the refurbishment process. She then reported the incident to the Operations Manager. On April 9, the Clearfield Center Director issued a report that categorized the incident as a tooling issue without mentioning the possibility of stress or damage to the rocket segment.

We found that ATK policy requires documentation of all incidents that could result in damage to program critical hardware. E-mail documentation of the incident by the move crew stated that the rocket segment was involved. However, the Clearfield Center Director was not aware of those e-mails until our review revealed them during our visit to the Clearfield Center, April 28-30, 2008. At that time, the Director took immediate and appropriate corrective actions. Therefore, this report contains no recommendations. We provided NASA management an opportunity to comment on a draft of this memorandum, issued August 19, 2008, but comments were not required and no formal comments were received.

Background

ATK’s Launch Systems Group manufactures rocket motor systems for human-rated and unmanned space launch vehicles, strategic missiles, prompt global strike missiles, and missile defense interceptors. Refurbishment and manufacturing services for space programs and commercial markets are provided at the ATK Launch Systems Groups’ Component Refurbishment Center in Clearfield. The Clearfield Center’s primary function is to return used Space Shuttle hardware to a reusable condition under the Space Shuttle Reusable Solid Rocket Motors (RSRM) contract, NAS8-97238, which is managed by Marshall Space Flight Center.

Handling and Movement of the Rocket Segment

Moves at the Clearfield Center are categorized as either critical or simple moves. A critical move is a handling operation where close quarters, minimal clearance, poor visibility, tight tolerances, interfaces with tooling or facilities, or program importance requires extreme care and control. A simple move is a handling operation where there is sufficient visibility, clearance, and control to perform the operation safely. Simple moves involve moving program critical hardware in an early process stage (e.g., housings, single parts) or non-flight hardware.

The complaint we received concerned a simple move on April 8, 2008, of an aft rocket segment, which is an early process stage of program critical hardware (a single part). The purpose of the move was to install a wash cover (see Figure 1). The wash cover is a stainless-steel plate weighing approximately 4,200 pounds and is used in the refurbishment of post-fired rocket segments. Once the wash cover is installed onto the segment, high-pressure water lines are attached and water is flushed through the rocket segment to remove any remnant thermal insulation debris.

Figure 1. Wash Cover Installation

Source: Form FA-0043-100, “Aft Segment Wash Cover Installation Procedures,” May 20, 2008

ATK uses an overhead bridge crane and a three-person move crew for the installation of a wash cover onto a rocket segment, in accordance with ATK Launch Services Group policy.³ See Figures 2 and 3 for side and front views of the installation components. Figure 4 is a close-up that shows the installation tool attached to the wash cover’s installation bracket. Installation details from the policy include the following:

• When the cover is 6 inches away from the rocket segment, the crew rotates the segment to align the cover’s holes with bolts in the rocket segment and then mounts the cover onto the bolts.
• After mounting the wash cover, remove the installation tool. To accomplish this, unpin the installation tool from the wash cover’s installation brackets and, to ensure there is enough clearance between the tool and the wash cover, move the tool away from the cover prior to the crane raising the installation tool. Figure 2. Diagram of Wash Cover Installation Components (Side View)

Figure 2. Diagram of Wash Cover Installation Components (Side View)

Figure 3. Diagram of Wash Cover Installation Components (Front View)

Figure 4. Close-Up View of Mounted Wash Cover

In accordance with ATK policy,⁴ if an unusual condition or event occurs during an operation, the move crew is required to stop the operation immediately and contact the crew supervisor as soon as it is safe to do so. The supervisor is required to contact the most knowledgeable individuals and resources to review the condition and determine the best course of action. The move crew can resume the operation once all of the requirements for documenting and correcting the unusual condition are fully implemented and approved by the appropriate level of ATK management.

We found that the move crew members–a crane operator (the move director) and two spotters–acted in accordance with ATK policy by stopping the April 8, 2008, operation and reporting the incident to the crew supervisor. The following table shows the chronology of the incident, as detailed in ATK documentation.

The move crew stopped the operation immediately when “an unusual condition or event” occurred. In this case, the unusual event occurred when the crane operator raised the installation tool. Because the tool had not been moved far enough away from the cover to clear the wash cover’s installation brackets, the lower crossbeam of the installation tool encountered one of the installation brackets. With the lower crossbeam wedged under the bracket, the installation tool was inappropriately poised to bear the combined weight of the cover and the rocket segment, potentially moving the entire assembly and exceeding design parameters of the installation tool, which is designed to accommodate only the weight of the cover. When move crew members observed what was occurring, they immediately halted operations and, after lowering the installation tool to relieve tension on the bracket and to remove any stress on the rocket segment, they notified the crew supervisor.

The move crew supervisor and the move crew visually inspected the installation tool and wash cover for indications of damage. There was no noticeable damage to the tool’s crossbeam or the wash cover. The supervisor directed the move crew to document the incident and resume the refurbishment process. She notified the Operations Manager as soon as practicable. The next morning, the Clearfield Center Director formed a management team to investigate the incident before issuing his April 9 report.

Allegations

Allegation 1. The segment was inadvertently lifted off of its washout fixture.

Our review of ATK documentation of the move and subsequent management actions disclosed that the move crew did not lift the rocket segment off of its washout fixture. Although two of the three crew members stated in writing that the segment had lifted from the washout fixture, and we concur that the rocket segment was subjected to some degree of stress and possible movement, subsequent inspections and engineering analyses disclosed that the segment could not have been lifted off of the washout fixture without visibly damaging the installation tool.

During our April visit, we requested that an ATK tooling engineer perform a stress analysis on the installation tool to determine whether it could have sustained the combined weight of the wash cover and the rocket segment. The tooling engineer, based on his analysis, concluded that the rocket segment could not have been lifted off of its washout fixture during this incident. Specifically, his analysis determined that the material yield point of the tool’s crossbeam was 11,823 pounds and that the combined weight of the cover and rocket segment was 33,616 pounds. That means that the installation tool’s crossbeam would have yielded, causing permanent deformation and catastrophic failure, with any load greater than 11,823 pounds. The installation tool’s crossbeam showed no damage, indicating that raising the tool had applied only a minimal degree of stress to the rocket segment. The tooling engineer stated that any movement witnessed by the move crew could not have been the rocket segment lifting off of the washout fixture, because it would have been structurally impossible for the tool’s crossbeam to support the weight of the wash cover and rocket segment.

Allegation 2. Production and refurbishment of the rocket segment was allowed to continue without adequate inspection or testing.

We determined that refurbishment of the rocket segment continued throughout April without adequate inspection or testing. The refurbishment work continued until our visit at the end of April, at which time the Clearfield Center Director was made aware of the move crew’s e-mail documentation of the incident and took immediate and appropriate corrective actions.

The incident occurred shortly after the operation began at 6 p.m., April 8. We determined that the move crew acted in accordance with ATK policy by stopping the operation and reporting the incident to the crew supervisor. The supervisor and move crew performed a visual inspection of the wash cover and installation tool, which revealed no visual damage to the installation tool. At 6:30, the move crew supervisor instructed the crew members to provide her a written description of the incident via e-mail and directed the crew to resume the refurbishment process. However, because damage could have occurred that would not have been detectable with a visual inspection, we found the move crew supervisor’s actions to be inappropriate in that she allowed the refurbishment process to continue after only a visual inspection and prior to ATK review of the incident.

The supervisor was aware that ATK policy required her to “ensure that the most knowledgeable individuals and resources . . . are contacted to review the condition and determine the best course of action.” Although policy does not require notifying the Handling Department when an incident occurs during a simple move, the supervisor told us that her intent was to involve Handling Department personnel. However, they were performing a critical move in another area of the building at the time of the incident. At 9 p.m., the supervisor notified the Operations Manager of the incident by telephone. We concluded that this was as soon as practicable, because the supervisor and her move crew were asked to assist with another move at 6:30. Her involvement with that move lasted until 9 p.m. She followed up with an e-mail to the Operations Manager at 12:11 a.m., April 9.

The Operations Manager notified the Clearfield Center Director of the incident on the morning of April 9, 2008. The Director formed a management team to investigate the incident prior to issuing his report later that day. The report, which was based on information from the management team, categorized the incident as a tooling issue. ATK policy states that the refurbishment process can resume “once all procedural requirements for documenting and correcting the unusual condition are met and properly approved” by ATK management. The Director’s April 9 report, categorizing the incident a tooling issue, met that requirement. However, neither the Director nor the management team was aware of the crew members’ e-mail documentation of the incident. That documentation stated that the rocket segment was involved, which made the incident a hardware issue requiring more stringent testing than a visual inspection before continuing the refurbishment process. It was not until our review brought the e-mail documentation to the Director’s attention that such testing was conducted. The magnetic resonance imaging (MRI) examination of the rocket segment on April 30 did not reveal any damage.

Allegation 3. An inappropriate report was issued to document the incident.

We found that the Clearfield Center Director initially reported the incident as a tooling issue, which was inaccurate. In our interview with the Director, he stated that no incident that could have resulted in damage to program critical hardware should be handled as a tooling issue. The incident was inaccurately reported because the Director did not have all the available information prior to his initial report.

When the Director was notified of the incident on the morning of April 9, he formed a management team to investigate the incident. However, the management team did not know that the move crew members had prepared written statements on the day of the incident. Without the benefit of those written statements, the management team performed a visual inspection to assess damage and, in conjunction with a verbal statement from one of the move crew members, categorized the incident as a tooling issue. Later that day, April 9, the Clearfield Center Director issued a report of the incident based on the management team’s assessment. Because tooling issues are a low priority, no corrective action was taken until our visit, when the Clearfield Center Director became aware of the move crew’s e-mails documenting the incident.

The e-mail documentation included statements from two of the three move crew members that the segment had lifted off of the washout fixture. The third crew member stated that she had observed movement, but could not determine what had moved. These statements specify that the rocket segment was involved in the incident. Any incident in which program critical hardware is involved must be reported as a hardware issue and treated accordingly.

Management Actions

When the Clearfield Center Director became aware of the e-mail documentation, which stated that the rocket segment had been involved in the incident, he took immediate and appropriate action.

During our visit, an ATK tooling engineer determined that the installation tool could not have lifted the rocket segment off of its washout fixture, reducing the probability that the segment sustained damage. Regardless, the Director ordered an MRI examination of the segment. A Discrepancy Report issued April 30, 2008, stated that the MRI did not reveal any damage to the segment.

The Director also instructed the move crew, in coordination with the handling and safety engineers, to reenact the steps involved in the move and to provide input on how the Clearfield Center could improve the process. Based on this input, ATK management issued a standard work instruction, Form FA-0043-100, “Aft Segment Wash Cover Installation Procedures,” May 20, 2008, that provides details on the process for the installation and removal of the wash cover. At the time of the April 8 incident, ATK did not have a standard work instruction for mounting a wash cover onto a rocket segment.

We did not include a recommendation in this report because ATK management took appropriate corrective actions during and subsequent to our visit. We provided NASA management an opportunity to comment on a draft of this memorandum, issued August 19, 2008, but comments were not required and no formal comments were received.

We appreciate the courtesies extended during our review. If you have any questions, or need additional information, please contact Mr. Raymond Tolomeo, Mission Programs and Projects Director, Office of Audits, at 202-358-7227.

signed

Evelyn R. Klemstine

Enclosure

cc:
Chief Engineer
Director, Marshall Space Flight Center

¹ ATK defines a move as a handling operation involving flight hardware or the hardware required for the refurbishment of flight hardware.

² Wash cover installation beam (2U141027-401). For this report, we refer to the wash cover installation beam as the installation tool. We also use the term wash cover (2U141027-201), although it is variously referred to as a splash cover, splash shield, and splash shield cover.

³ Management Policy IHM Section 29-999, “Space Operations Handling,” October 10, 2006.

⁴ Management Policy OP-E, “Operating Protocol for Unusual Conditions,” February 2, 2006.

Scope and Methodology

We performed this review from April through August 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform our work to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on the objectives. We believe that the evidence obtained during this review provides a reasonable basis for our findings and conclusions based on our objectives.

For this review, we interviewed

• the ATK move crew members involved in the incident to determine their roles and the procedures they followed to report the incident;
• the move crew’s supervisor, the Operations Manager, and the Clearfield Director to determine how they categorized and reported the incident; and
• the Clearfield Safety Manager to discuss information on the incident report and to discuss his role in investigating and categorizing incidents.

In addition, we interviewed Safety and Mission Assurance personnel at Marshall Space Flight Center to determine quality assurance functions related to the incident. We also reviewed documentation and applicable regulations, policies, and instructions relating to the allegations, including

• ATK Launch Systems Group, Management Policy SA-D, “Safety Commitment,” April 20, 2005;
• ATK Launch Systems Group, Management Policy OP-E, “Operating Protocol for Unusual Conditions,” February 2, 2006;
• ATK Launch Systems Group, Management Policy OP-25, “Special Tooling,” January 22, 2008;
• NASA Technical Standard NASA-STD-8719.9 w/Change 1, “Standard for Lifting Devices and Equipment,” May 9, 2002;
• Safety Plan for Space Shuttle Reusable Solid Rocket Motor Project for Contract NAS8-97238;
• Quality Assurance Letter of Delegation for Contract NAS8-97238;
• obtained and reviewed requirements documents for the segment and safety move work authorization documents;
• toured the area in which the incident occurred and reviewed the condition of the rocket segment;
• obtained a copy of the Incident Report;
• obtained copies of written descriptions of incident prepared by the ATK employees involved;
• obtained training information for the three employees involved in the incident; and
• obtained copies of the discrepancy report and tooling report.
• Computer-Processed Data. We did not use computer-processed data to perform this review.

Prior Coverage. There was no prior coverage pertinent to our review of this incident.