Learning from the NuSTAR Launch Delay

By Don Cohen NuSTAR, the Nuclear Spectroscopic Telescope Array, contains the first focusing telescopes designed to look at high-energy X-ray radiation. It is expected to contribute to a better understanding of collapsing stars and black holes.

Because NuSTAR is designed to function in an equatorial orbit, it launched on a Pegasus XL rocket from a point south of Kwajalein Atoll, in the Marshall Islands, on June 13, 2012. Built by Orbital Sciences Corporation, the Pegasus is carried to approximately 39,000 ft. by an L-1011 aircraft. Released at that altitude, the three-stage, winged rocket ignites its first-stage motor to continue its journey to orbit.

The June launch came almost three months after a planned early March launch date. The story of that delay–why it happened and what both NASA and Orbital Sciences learned from the experience–offers insight into how NASA deals with technical risks and into the agency’s developing relationships with commercial providers of launch vehicles and spacecraft now and in the future.

Why the Launch Delay?

Two issues needed to be resolved before NuSTAR could be approved for launch. One involved the Pegasus fairing–the streamlined shell at the nose of the rocket that protects the payload during its climb to orbit. The Pegasus fairing hardware was similar to that of the Taurus XL, which had failed to separate on two recent NASA missions; its added weight kept the Orbiting Carbon Observatory and the Glory spacecraft from reaching orbit. The cause or causes of those failures had not been definitively determined–the rockets fell into the sea so there was no physical evidence to examine. The Pegasus fairing had been somewhat redesigned to reduce the likelihood of a similar failure, but that created its own uncertainty, since the new design had never been tested in flight.

A second issue had to do with the fact that the flight computer aboard Pegasus and the associated flight software and simulation software were new. This change was a jointly funded reliability improvement by Orbital and the NASA Launch Services Program (LSP) to replace an obsolescent, out-of-production industrial microcomputer (albeit with two decades of excellent performance) and bring the flight software and simulations up to current standards. Initially, the fairing issue seemed the more serious of the two. That expectation changed. The team studying the fairing issue concluded that the risk of a malfunction was minimal; the software concerns proved harder to resolve. NASA’s software team expressed growing concern over the lack of adequate simulation and test data.
Technicians review their checklists after joining NASA’s NuSTAR spacecraft with the Orbital Sciences Pegasus XL rocket.

Reliable simulation data are essential. Omar Baez, NuSTAR’s launch director, notes, “Rockets are not forgiving,” and Director of Launch Services Jim Norman adds, “All the vehicles need to reach 17,000 mph. Errors are amplified by the energies expended.” And, as NASA Chief Engineer Mike Ryschkewitsch points out, the only live “test” for a rocket is an actual launch. New aircraft, by contrast, can be tested bit by bit through a series of increasingly demanding flights that start by determining basic airworthiness and eventually map the limits of safe performance. Simulations matter for aircraft design and construction, too, of course, but not as critically.

Although data were arriving late from Orbital, the LSP technical team worked extremely hard to execute the plan during February and early March, and the mid-March launch date still seemed achievable, provided no further serious issues were identified. Unfortunately, as the date for the all-important guidance, navigation, and control review approached, both Orbital and LSP were finding that simulations exhibited far too many failed cases to proceed.

With Orbital management responding to the magnitude of the problems, the contractor was providing large quantities of data and the LSP flight-analysis team demonstrated an ability to process it quickly and accurately. Suspected errors identified by NASA were being confirmed by Orbital right up until the night before the Flight Readiness Review (FRR). Both the LSP and Orbital teams put in extremely long hours that did not compromise the rigor and careful technical review and risk analysis. The LSP flight-analysis team held a final five-hour peer review on March 14, where every finding was either closed or identified as still open. Their rigor and diligence in the face of a launch deadline is an example of technical excellence not compromised by schedule pressure.

Late on March 14 it became clear that Orbital could not resolve all the remaining items without making changes to the flight code and simulation models. The technical team informed management, and the launch opportunity was scrubbed.

“Take the Time to Do It Right”

Part of the NuSTAR story is about the support the mission team got for carrying out the analytical work that needed to be done, even if that meant a delayed launch. Because the Kwajalein Atoll launch site was reserved for a classified mission after the NuSTAR March launch window, taking more than a few extra days to resolve the technical issues would force the mission to wait months to launch the spacecraft. Realistically, the team was looking at a delay of at least three months and the extra costs associated with it.

NASA has long been sensitive to the tension between technical risks that need study and possible mitigation and the desire–sometimes the pressure–to launch on schedule. The 1986 Challenger disaster brought the issue into tragic prominence. Reluctance to delay that launch was one of a complex of organizational factors that led to the disaster. Since then, the agency has improved its FRR process and practice to ensure all technical issues are heard and discussed, and that “launch fever” does not drown out voices expressing concerns about unresolved risks. (See “Getting to ‘Yes’: The Flight Readiness Review,” by Matthew Kohut and Don Cohen, in the Winter 2010 issue of ASK Magazine for the story of a series of FRRs and technical work done before STS-119 was cleared for launch.)

Virtually everyone involved with NuSTAR agrees that technical teams got strong support for doing the work necessary to ensure a successful launch. Some individuals say they heard “mixed messages” from leadership–both “take all the time you need” and “hurry up and get it done.” Certainly the desire to
solve the problems and launch as soon as possible was clear, but the strongest and most consistent message seems to have been “do it right.”

NuSTAR mission manager Garrett Skrobot recalls the meeting where Ryschkewitsch said, “If you guys need the time, take the time to do it right.” Recalling the delay discussion later, Ryschkewitsch commented, “It was a hard conversation, but not really that hard”–suggesting that, although no one welcomes a launch delay, it was clearly the right choice in this case.

Mike Luther, deputy associate administrator for programs in the Science Mission Directorate, communicated the same message, saying, “We won’t launch until we’re ready.”

Amanda Mitskevich notes that the project carried out regular extensive teleconferences with stakeholders about progress on the technical issues. The entire NuSTAR community (which included Goddard Space Flight Center, Jet Propulsion Laboratory, Orbital Sciences, and NASA Headquarters, among others) knew what was happening: why the delay was necessary and what was being done to resolve the software issues. So there were no groups within NuSTAR pushing for an earlier launch or expressing frustration because they were out of the loop and did not understand what was going on.

As a result of extensive support and good communication, Mitskevich believes, the teams working on the technical issues were not especially burdened by what she calls “additional pressure” to solve the problems faster–that is, in addition to their own internal drive to do the work thoroughly and as quickly as possible.

As soon as the specific nature of the difficulties came to light, the Orbital and NASA engineering and management teams blended complementary technical approaches to identifying and solving problems. The mutually reinforced technical rigor overcame problems in a relatively short time, while the management teams cooperated to delay the launch to give the engineers the breathing room they needed to implement all necessary fixes and validations.

If Orbital was initially largely “reactive” to NASA’s concerns, it soon became much more proactive and constructive. What could have been an adversarial situation developed into a partnership. Both Orbital and NASA software teams worked “tremendous hours” to solve the problems, according to Baez. And Orbital began reviewing simulation software for other vehicles on its own initiative.

Later in the spring and comfortably before the rescheduled launch date, NASA and Orbital had made enough progress to be confident they would be ready to OK that June launch.
An Orbital Sciences technician completes final checks of NASA’s NuSTAR inside the Orbital processing facility before the Pegasus payload fairing is secured around it.

An Orbital Sciences technician completes final checks of NASA’s NuSTAR inside the Orbital processing facility before the Pegasus payload fairing is secured around it.
Photo Credit: NASA (Click image for full size.)

Some Lessons

Baez notes that the NuSTAR experience was “a software education for a lot of people.” Certainly the problems were a reminder that software has grown to be an increasingly complex and absolutely critical element of all space missions. Failing to give it the attention it deserves invites disaster. (For a good analysis of this issue, see “Is Software Broken?” by Steve Jolly in the Spring 2009 issue of ASK.) The generally high morale of the NuSTAR technical team was tempered by the nagging suspicion that if software testing had occurred sooner–a prudent approach for new code and simulation tools–many of the problems could have been caught and corrected earlier.

In the case of Pegasus, NASA and Orbital failed to fully anticipate the difficulty in maintaining communication, continuity, and comprehension of the full software and simulation as a coupled system. This complexity added to the now obvious rationale to start simulation and software testing sooner.

The more general lesson, Skrobot points out, is that any new element in a launch vehicle should be looked at as early and as thoroughly as possible. Figure out what the hard questions are, says Skrobot, and ask them.

Toward a New Way of Working

The NuSTAR launch delay experience is important beyond this particular mission because it is a step toward defining NASA’s developing working relationship with the commercial providers of launch vehicles and spacecraft that will be an important part of NASA’s future. Both NASA and those companies are in the process of learning what they need to do–individually and together–to produce launch vehicles that are reliable but also relatively economical and profitable for their creators.

NASA has never developed rockets on its own, of course. Boeing, Douglas, Lockheed Martin, and other aerospace companies have had a major role in designing and building the Atlas, Saturn, Delta, and other launch vehicles the agency has depended on until now. But those vehicles were the products of extremely close (and expensive) cooperation between NASA and those contractors. In effect, those vehicles were jointly designed and extensively tested by both NASA and contractors.

Today, commercial companies like Orbital Sciences and SpaceX are building new rockets with much less direct involvement and oversight from NASA. The agency needs to be sure that these new vehicles are reliable, but must do it in ways that allow those companies to keep their costs down, ultimately reducing the cost to NASA as well.

In other words, NASA needs to develop–and is developing–some version of what Ryschkewitsch calls “parenting mode,” trying to find the right balance of guidance and help on one hand and letting commercial providers make and correct their own mistakes on the other. Being too involved–asking for too much documentation or too much testing to prove reliability–reduces risk but drives up cost when the rationale for the new relationship with commercial developers is to find less expensive ways to send cargo and crews into space.

The NuSTAR experience is helping NASA and Orbital learn to define that balance. For a time, NASA may have been too hands-off in regard to the software issues. As Skrobot suggests, it is important to ask the hard questions. The lesson for NASA may be to carefully target its “parental” oversight–to identify the potential problem areas early and focus attention and resources on them. Asking tough questions about everything would be intrusive and wastefully expensive; asking the right tough questions is essential. Knowing what those questions are is not necessarily easy, though, except in hindsight. James Wood, LSP chief engineer, says, “I don’t know how to ask the mythical ‘hard questions’ and neither does anyone else.”

As NASA reduces its traditional high level of oversight, Orbital and other commercial providers need to ensure they devote the resources necessary to ensure vehicle reliability. Having a relatively lean team is important to efficiency and therefore profitability, but they need to know when lean is too lean. As NASA’s “parenting” becomes less intrusive, their responsibility for quality and performance increases.

Testing or Flight Success

There are, notes Ryschkewitsch, two ways of determining acceptable risk: testing and documentation, or a history of flight success. Seventy to eighty successful Soyuz flights are a reasonable substitute for a lot of testing and documentation. Vehicles recently developed or under development obviously don’t have that kind of flight history. Building a record of success through flights whose failure would not harm crews or programs is one strategy for developing the next generation of vehicles. So, for instance, NASA was willing to let SpaceX take responsibility for the launch of the Falcon 9 and Dragon that carried cargo to the International Space Station in May and October 2012. NASA’s main involvement was ensuring that the approach and docking would work and not endanger the station. The success of that flight is (ideally) the beginning of a track record that will give NASA confidence in the reliability of a vehicle designed without extensive agency oversight.

Similarly, the successful NuSTAR launch helps build confidence in the current version of the Pegasus. That success and all the testing done are important preparation for the next Pegasus-based mission. The fairing analysis done for NuSTAR similarly will serve future missions. As part of the analysis, the NASA team removed a tiny piece of the frangible joint of the Pegasus fairing hardware to test its hardness. This made NuSTAR people unhappy, as would any change to their launch vehicle, no matter how small, but the information gained will benefit the Interface Region Imaging Spectrograph, which is expected to launch via Pegasus in 2013, and later missions.

But the flight-success criterion is not always as straightforward as it sounds. Pegasus had been in operation for more than twenty years before the NuSTAR launch and has had more than forty successful flights–the kind of success record that normally inspires confidence. But the modified fairing design and new flight computer and software had not been flight tested and therefore needed oversight. And this is far from a unique or even an unusual problem; long-lived launch vehicles frequently have some elements that become obsolete or unavailable and must be replaced–and tested to ensure their reliability.

An additional way to manage the new oversight relationship, Ryschkewitsch suggests, is to have NASA engineers sit in with commercial designers as companies develop their new vehicles or new vehicle elements. If the NASA people are satisfied with the design process and testing within the company, they recommend the appropriate (limited) amount of documentation NASA should require.

Shaping a New Partnership

Whatever ultimately characterizes the relationship between NASA and the developers of future launch vehicles, it is certain that it will be shaped by experiences like NuSTAR and the Falcon 9 program. The general outlines of what will be required are clear now–less control by NASA, more responsibility taken on by the commercial companies. But precisely how the partners should work together–the details that fill in that general outline–can only be developed through multiple experiences of facing and solving problems like the NuSTAR software issues.

Since every NASA mission has some unique elements, that learning process will continue, with better and better understanding of the potential and pitfalls of the new relationships. In the new environment the agency is operating in, NASA’s Launch Services Program is both the pathfinder and the partner in a new way of working.