NASA Runs the Press Gauntlet: Mission Operations and Email
NASA’s Associate Administrator for Space Flight Bill Readdy sat down for an hour or so with a dozen reporters earlier this week. The intent was to provide background on how launch, on-orbit, and landing decisions were considered and implemented during a Space Shuttle mission.
Half way through the session NASA Administrator Sean O’Keefe popped in. Soon, with the prodding of reporters, what had been a routine briefing with lots of detail and minimal sound bites became a bit of a low grade confrontation between O’Keefe and some of the assembled NASA press corps – and the original purpose of the briefing was all but forgotten. Indeed, no one, except me seems to have bothered to try and report the full breadth of what was said.
Part 1: An Astronaut’s Perspective on NASA Culture
Readdy opened the session noting that many news stories of late all seem to turn to wards the issue of NASA ‘culture’. As such, he thought it would be useful to share his own insight of that culture and how he entered into NASA.
He began with a look back to the days just before he joined the agency. “I was naval pilot on the U.S.S. Coral Sea in the Mediterranean” he began. “One day at 2:00 AM, after a night mission, a parachute rigger grabbed me and said that he had just heard that Shuttle crashed. We were not sure what the cause of the accident was.”
“This was 1986. At that time I was in the midst of the astronaut candidate interview process. This was a time when there was no high-bandwidth communication aboard ships. All we had was a teletype. We had to rely on this until we actually got a news paper days later.”
“That was a pivotal moment for me. I knew that I would come back from deployment and work for NASA and work on return to flight activities. As it happened, I was selected into the next group of astronauts – the first group selected after Challenger.”
Readdy then went on to describe the NASA ‘culture’ he became familiar with in a post-Challenger environment.
“We were not flying. There was investigation underway to determine the cause of the accident. By the time summer rolled around everyone had pretty much decided that the O-rings and joint design were the cause of the accident. We also went across Shuttle systems to look at other areas for possible deficiencies. One of the deficiencies that was noted was the cavalier approach that had been used in the certification of flight readiness – the reviews of hardware that lead to the signoff to fly on a particular mission.”
“One of the things they did during this process was to take us to other NASA centers and expose us to different elements in the Shuttle program and brief us on the redesign. During the process of returning to flight, we also brought more astronauts into the process since we were, after all, the ultimate stakeholders. That was one of the conclusions that had been reached (during the Rogers Commission activity) – that there were no astronauts playing key roles in NASA’s structure (safety, management etc.) Over time, we have seen that this has changed a lot. You might say (through no prior planning on my part) that I am a product of that aftermath – especially given that I am Associate Administrator for Space Flight right now and that I have had other program jobs along the way.”
“While I worked on the redesign, I spent a lot of time with orbiter management in Downey, California with the people who had design center expertise. Even though the orbiter was not suspect in the accident, there were things that were in the queue to be upgraded prior to 51L (Challenger’s last flight). As a result we wound up with a much safer orbiter and the entire process of design certification came under review. Things that we waived such as nose wheel steering, lack of redundancy in critical systems, etc. were addressed.”
“In addition to spending time on design issues, we also spent time (as astronauts) going to sub-tier vendors. We did this to make sure that even though their businesses may not be primarily focused on space per se (perhaps just a small portion of their product actually went into shuttle program) that they were cognizant of the fact that people flew on those spacecraft. We wanted to reinforce this fact on a personal level: that where we live – in human spaceflight is graded on a harsh curve and the environment is extremely unforgiving. 99% is not a passing grade for us. We always need to strive for 100%.”
“Every class that has come into the astronaut program since Challenger has had interactions with project and program level personnel to reinforce the fact that we participate in most every review – and have insight into all of technical issues that go on in the Shuttle and Space Station programs.”
Part 2: Mission Ops 101
At this point Readdy moved on to discuss Shuttle mission management as it is practiced at NASA today.
Readdy began with the FRR – the Flight Readiness Review. “The FRR takes place at a number of levels. The question that often arises is ‘given that there seems to be a spirited debate at a number of levels, are people hurt in the process?’ ” Readdy then described the main levels in the review process – Level 1 relates to the overall program, Level 2 is focused on integration, Level 3 concerns hardware elements such as the External Tank, Solid Rocket Boosters, etc. as well as some processing functions.
“At each crucial juncture in the processing of a vehicle we have a review. When we get ready to roll out of the OPF (Orbiter Processing Facility) to the VAB (Vehicle Assembly Building) we have a ‘Rollout Review’ during which [representatives for] all elements that play a part in processing are polled and [issues] heard. The same thing happens before we ‘stack’ the Shuttle (Connect the Shuttle orbiter to the External Tank and Solid Rocket Boosters), and again before we rollout to the launch pad. All of this culminates in a ‘pre-FRR’ for different elements. Finally, the Program Manager has a review before the formal FRR which is held 2 weeks prior to launch.”
Readdy went on to compare this current process with what happened up until the Challenger’s last launch. “This is in stark contrast to pre-Challenger when this process was done on a telecon” he said “one which was informally attended and not recorded. Now we have a very sober and detailed proceeding – one that is chaired by the Associate Administrator for Space Flight and co-chaired by the Associate Administrator for Safety and Mission Assurance (former astronaut Bryan O’Connor). These proceedings are also recorded. People have to ‘be there’ in order to make their presentations. The entire room is polled at the end of the proceedings for any outstanding issues. Any issues that arise are taken forward to be worked.”
The next step in the process is the ‘L-2 day Review’ (Launch minus two days). Readdy said “this may actually occur on L-1 depending on mission launch time (time of day). At that point, execution of the launch countdown and the flight itself is turned over to the Mission Management Team (MMT). The chair of the MMT chairs the L-2 Review where any open paper work, waivers, and unresolved issues brought forth. Any lingering things that cannot be dispositioned (resolved) at L-2 are moved to a ‘Tanking Meeting’ which is held 12 hours before launch. At this meeting unresolved items are either resolved or NASA does not proceed with fueling (‘tanking’).”
Once within that 12 hour period, the Launch Control Team is in place and is supervising the fueling of the launch vehicle and executing the final portion of the countdown. The Launch Control Team coordinates with the MMT for any final resolutions of any outstanding issues. Once the process moves into the final countdown, whether or not launch occurs can depend on weather, Shuttle systems, etc.”
As soon as the vehicle is launched” Readdy said “Mission Operations and the Flight Control Team takes over. They take care of the execution of mission objectives. This is part of a defined mission profile that is actually put together many months before flight and baselined in a ‘Flight Definition Requirements Document’. Flight Operations then executes the mission according to the baselined flight timeline.”
Every console in Mission Control has a notebook (‘Console Notebook’) of procedures to deal with various subsystem issues that might arise – these responses are pre-planned. Shuttle crews also have checklists for these procedures on board in case they lose communications.”
[Editor’s note: see ISS Operations Documents at SpaceRef. This page contains links to a large collection of actual operations documents used during Space Shuttle missions to the International Space Station.]
NASA’s use of its own special jargon is legendary. Some of it is decipherable. Other terms – including every day words – take on a special meaning often different that a dictionary would suggest. According to Readdy “the word ‘contingencies’ has a whole number of connotations. ‘Contingencies’ to us are the planned for use in response to an incident. Each console has a book of so-called ‘Flight Rules’ which contains pre-planned responses to anomalies.”
[Editor’s note: see Contingency and Emergency Operations at SpaceRef. This page contains links to a large collection of actual contingency operations documents used during Space Shuttle missions to the International Space Station.]
For every person you see on TV during the course of a mission, many more labor away from the cameras. According to Readdy “There is a back room for each position you see on TV in the Mission Control Center. Not only are the teams in these rooms armed with Console Notebooks, they also have several people listening in to what is going on and are able to provide more detailed research should that be required. Anything requiring expertise beyond that level of knowledge goes to the Mission Evaluation Room (MER) which is staffed with contractor personnel who have phone and data access to contractor facilities. They are the ones who work problems offline. This is done via special teams – Problem Resolution Teams.”
“The Mission Evaluation Room tackles the ongoing anomaly log and works each item out in excruciating detail. Their products go not only the Mission Control room, but also to the MMT for inclusion in their daily status – as well as for decision making.”
“The picture I am trying to paint here is that during any typical mission those kinds of discussion on any anomaly – be it a data drop out, communications issue, humidity concern, or debris impact, are worked out at a number of different levels. All information is published in anomaly logs during a flight and briefed to the MMT for whatever decision making is appropriate.”
As mentioned before, the MMT steers the execution of a flight. “Typically”, Readdy said, “Flight Control Teams operate within their preflight parameters and within the Flight Rules they have. But if there is a deviation from the rules required, that is where the MMT provides guidance. Some examples include weather, i.e. should we delay a day and wait for a Kennedy landing opportunity? or should we go ahead and prepare to land on the lakebed at Edwards? Those kinds of discussions are typical.”
In closing his short tutorial, Readdy said “there is one final thing I’d like to leave you with: the discussions that go on within the Mission Management Team within the Mission Evaluation Room and the Problem Resolution Teams.” This surfaced in the form of emails between JSC and Langley Research Center (LaRC) during the STS-107 mission.
“These discussions were not ‘ad hocery’.” Readdy said. “This was part of a pre-planned Problem Resolution Team dealing with landing and rollout. We have other such teams that we call on as the case may be on ay given mission. In this case, they engaged in a what-if discussion regarding landing and rollout issues.”
[
Editor’s note: see Routine and Emergency Medical Operations at SpaceRef. This page contains links to a large collection of actual contingency operations documents used during Space Shuttle missions to the International Space Station.]
“The training for astronauts and Flight Control Teams occurs for almost a year for each mission. We exercise the judgment of console operators and the mission team together with the crew – and we built that team over a long period of time. As such, thinking soon becomes transparent from one group to the other – even without voice and data links. Eventually, you get to the point where – and you have probably herd this if you have listened in on mission communications – where the CAPCOM anticipates what the crew will do – or need. As such you hear a call back from the crew saying “you were reading our minds”. This is a result of the relationships that develop over time. Part of the reason why you develop this relationship is so you can exercise it under extreme conditions. You exercise not only the crew, but also the CAPCOM, the immediate team in the control room, the Mission Evaluation Room contractors, and other people located across the agency in other centers.”
Part 3: Q&A
At the conclusion of his tutorial, Readdy took questions from reporters. The first dealt with how astronauts trained for disasters such as the one that struck Columbia. Readdy replied “the training starts the second you volunteer to be an astronaut. We all know when we sign up that this is a very risky and unforgiving business.”
“As test pilots or Mission specialists we appreciate that there is inherent risk in this and we are prepared for that. The risk is most intense during liftoff but it is elsewhere as well. As we saw that in the Apollo 204 fire (in 1967) There is very little about this that is without risk. Indeed, all you have to do is look at the number of astronauts killed in T-38 jet aircraft during training.”
“Over the years, while you prepare for a mission, you spend a lot of time preparing – mentally and physically – and working as a team with launch and flight control teams on things that you can affect mission outcome.”
“With regard to the events that befell Columbia, Readdy said “The energy expended in the 8 1/2 minutes it takes to get you into orbit at 17,500 miles per hour is dissipated (later) over the course of an hour during reentry starting with the deorbit burn over Australia.” As for the possibility of dealing with Columbia’s destruction, Readdy said “Structural break up? No. There is nothing that can prepare you for that. This was beyond any kind of ejection seat or bailout system. Its all about Risk vs. gain. It is not about us individually. What I think is important is that we do it – and that it affords a better future for our kids and grand kids.”
I then asked Readdy about the regular status reports issued by the MER – Mission Evaluation Room during a mission. I noted that several years back I had received them from several sources within NASA. After posting them [sample MER report] without incident my sources made it very clear that they had been told not to send them to me any more. One person suggested that his job might be in jeopardy. Reminding Readdy of his opening comments regarding NASA culture, I asked Readdy if this reluctance to share information – much of it highly technical, routine, and harmless – was in any way indicative of a management issue that needed to be addressed.
At this point NASA Administrator Sean O’Keefe popped (unexpectedly) into the room and sat at the head of the table with Readdy.
Readdy discussed these reports in detail noting that the MER Status was an exhaustive list of all in-flight anomalies – including things as innocuous as one time the failure of a sensor. The impact might be minimal during the current mission, but the item would need to be dealt wit at some point after the mission. I repeated my question asking if the MER reports from STS-107 could be related. Public Affairs Chief Glenn Mahone nodded that this would be something that could be done. I have had several email exchanges with PAO and am awaiting further information.
Readdy then returned to the broader issue of openness and freedom to debate. “The fact that there was this sported debate going on talks about open culture and people free to discuss whatever they need to discuss. I think it is important to say that as engineers and as controllers we are always trying to anticipate the next possibility – ‘worst upon worst upon worst’. That is what they were doing. The Problem Resolution Team was dealing with landing. They talked about what would happen if there were blown tires, or of the gear did not work. They – and we – encourage this sort of discussion. This why we have those teams.”
What followed were a series of questions from reporters focusing on the content of emails and why one individual or another had not yet been made available to discuss what they wrote.
One reporter asked about one email exchange where the participants complained that they had not received a Boeing analysis – and whether this was all being interpreted out of context. Readdy replied that normally he would not expect that they’d see a structural or thermal report (like that produced by Boeing) – “the fact that the got it shows that we encourage that [sort of openness]” Readdy said. “The fact that those folks had access shows that we had an open system I don’t think that they would have gotten it any sooner unless they were physically at JSC. With our Internet culture and the speed with which it can move things this allows you to use people across the agency”.
At this point O’Keefe interjected a comment about these emails – something he has been vocal about in the past few days. “Think about any email” he said “This is a style that is grown up in the last 10 years. You need to think about this in the context of how each of us writes things. Look at your own e-mail commentary in the last week and determine whether you think the phrase you used today, a week later, looks different to you. If it doesn’t, then you’re a far better, more disciplined person in writing your commentary than most folks are.”
“Are we concerned about this?” O’Keefe said. “We’ll see. When we look at overall body of evidence. CAIB will come back and say what probable cause is. That may put a new light on this context that it does not have right now. Until then, whatever conclusions you want to draw are based on assumption, theory, how you read it, and what the words say out of context.”
Readdy added “concerns expressed and considered by flight controllers and possibilities were discussed. They walked their way through all of that. I probably would have thought the same thing.”
Readdy was then asked if similar procedures (to those used on Shuttle flights) was in place for the ISS. Readdy replied “we have same structure for the Flight Control Room. However it is more distributed, we have one support room in Moscow. When there is an anomaly on the ISS you use the same kind of approach. The same Mission Evaluation Room avails itself of contractor expertise. The other thing about ISS that is different is that we also have a control room in Huntsville that that deal with payloads and experiments. There is plenty of expertise to avail yourself of.”
When asked if it was uncommon to do ‘what iffing’ during a mission which results in discussions that mention ‘Loss of Crew’ (LOC) and if that should have gone up to the attention of the MMT, O’Keefe replied “that assumes that you have decided on a cause. When the cause has been decided then we can answer with some certainty.”
A reporter then asked how such problems would bubble up the MMT. Readdy replied that “the MMT has insight into everything that is going on during the mission. There is a daily ‘tag up’ for MMT. Most of MMT membership also has some involvement in other aspects of the mission so they are very aware of what is going on. A formal presentation to the MMT is not the only way that information to be communicated up or down. Everyone has NASA Select TV on. Shift reports, etc. are constantly being circulated. Anomalies have been book kept by the MER. As such, the MMT would be aware of all anomalies that MER working as well as other things such long term weather, issues that have to do with the ISS, experiments. On STS-107, they were also aware of the research complement (in Spacehab).”
Placing the whole system into context Readdy said “It is almost a neural kind of link up among all the different groups that are involved in execution of a flight. Maybe its more than thinking of this as if it’s a stratified or hierarchical arrangement. We are so electronically linked and physically close to each other. We have frequent meetings. It seems hard to imagine that things were not transmitted.”
‘Would landing and rollout issues be raised to the MMT’? a reporter asked. Readdy replied that they would not and that this would already be part of the procedures that were contained in the Console Book. As such the hypothetical scenarios would not go beyond the Problem Teams.
O’Keefe interjected some additional thoughts at this point. “This is about judgments. This is something we can debate. But the [CAIB’s] findings will not only treat the question of causes but also the systemic and management issues. We will be guided by these findings without equivocation. We have tried to do some soul searching. We have to get away from a hierarchical way of looking at things. An engagement needs to be made at all kinds of levels to get a sense of all activity. Then, at some point, a judgment is made. THAT is the issue that we will be guided on by the board. As the evidence is developing, we are looking at the process and issues. This is about the judgment calls that are made. There is a fine line between oversight and micromanagement. At the end, when findings are rendered, that is what will inform us. Right now, we’re speculating about bits and parts which may or may not support a current theory. We need to step back a bit and not make a judgment about others; not until more information is available.”
Returning to appoint made in previous interviews , O’Keefe said “I have seen no evidence that there was no malice, complacency or indifference. If anything, there was a spirited exchange that we want to encourage.”
Jumping to another issue, O’Keefe said “One story that has not been written at all involves the process of recovery. It has gotten scant coverage. This effort represents the coordination of 29 agencies. Volunteers – there are more than 3,500 folks. This is nothing short of a marvel and serves as a testimonial to the human spirit. There is a seamless relationship between Federal and State Agencies who never had occasion to meet – much less work with each other before. A clear focus has been placed on what is important. There is no debate about jurisdictions or boundaries. This is not a case where we’ve had to go out an place an ad. Some FEMA veterans were at the World Trade Center and the Pentagon and bring amazing backgrounds to this effort.”
Several reporters then pressed O’Keefe again on the issue of emails and their contents. Specifically they wanted to know why Robert Daugherty has not been made available to speak with reporters. O’Keefe reiterated a conversation he had with one of the email participants – someone he declined to identify – at NASA LaRC. He recounted that this person was concerned about characterizations in the media of his email exchanges saying that people were suggesting that he was waving a big red flag and that no one was listening to him. “Nothing could be further from the truth” O’Keefe said.
Editor’s note: At this point O’Keefe’s annoyance with the line of questioning was apparent. Indeed, after being pressed repeatedly to identify the LaRC employee with whom he had a discussion about email, O’Keefe declined, saying that it would “just add another log to the fire”. Some reporters have used the words such as ‘bristle’, ‘testy’ and so forth to describe his tone. While not at all inaccurate, these curt descriptions are offered up somewhat out of context without fully characterizing the entire session’s discussion. Indeed, readers of this article will note that it is many times the length of any other story written by reporters who attended the same event. Length does not equate to quality by any means – but longer articles do, by definition, allow more things to be said.
Getting back to how these missions are managed (the original intent of this press briefing) Readdy said “the execution of a normal mission is like a decathlon just to get normal tasks done. That is what crew is burdened with. Willie and Rick were informed that something came off of the tank. We even had a video file sent up to them. We got an analysis that said it was not an issue. If I was payload commander (given other responsibilities) that would be the last of my worries. I am convinced that when the recommendations are presented it may be hardware issues or errors in judgment – but it will not be because of complacency.”
In closing, Readdy discussed an example of how mission management can work a complex and unforeseen issue on very short notice – to a positive end. During STS-49 the INTELSAT VI (F-3) satellite, stranded in a useless orbit since 1990, was to be rescued. Special gear (a ‘crossbeam’) was developed that would fit at one end of the satellite and allow it to be grappled by the Shuttle’s robotic arm and placed in the cargo bay for return to Earth.
When the Shuttle arrived and met up with the satellite it became clear that the specifications NASA had designed the special hardware to were not the way the satellite was actually constructed.
Some quick thinking was needed. Many simulations in airlocks and water tanks (WETF) were conducted to find an alternate way, using astronaut muscle power, to grab the satellite. On the ground, a simulator for the airlock was stripped of everything inside. Three people were placed inside. This had never been done before. Since there were only 2 comm channels to use during EVAs, plans were made as to how to have three people use two channels.
The exact dynamics of the satellite’s behavior in these situations was partially unknown. It was spinning and it had tanks with fuel that was sloshing around. When rehearsals were done in the WETF with a robotic arm things did not work out they way they had hoped. Eventually, some ingenious ad hoc procedures where things were worked in reverse (at the suggestion of Story Musgrave) were developed that allowed the exact settings needed for the robotic arm to be derived.
When this was actually done in space, one astronaut was ‘standing’ on the sill of the cargo bay where no astronaut had ever stood before [image]. Others were perched in other locations. The procedures worked and the satellite was captured and returned to Earth.
