Apollo Pioneers Share Lessons Learned (Transcript)
Speakers:
STEVE ROCAMBOLI, Apollo Lunar Module Engineer
SEYMOUR BERG, Apollo Lunar Module Engineer
GERRY SANDLER, Apollo Lunar Module Engineer
BOB SCHWARZ, Apollo Lunar Module Engineer
JOE MULÉ, Apollo Lunar Module Engineer
[Moderated by Beth Dickey, Headquarters, Public Affairs] Friday, July 20, 2007
MODERATOR: The astronauts first stepped foot on the Moon 38 years ago today, and all of that was made possible by the gentlemen you see here with me today.
Now as NASA embarks on a new journey back to the Moon, this time we hope to stay. We look to the past and lessons learned from the Apollo program to help ensure success.
Welcome, everyone. I am Beth Dickey of NASA Headquarters, Public Affairs, and I am joined today by retired members of the Grumman Corporation’s Lunar Module Reliability and Maintainability Team, and I will introduce them now.
First, we have Steve Rocamboli, Seymour Berg, Gerry Sandler, Bob Schwarz, and Joe Mulé. We also have one of our Deputy Associate Administrators from NASA’s Exploration Systems Mission Directorate, Doug Cooke, here in the front row with us.
The Apollo Lunar Module was the first true spacecraft. It flew totally in a vacuum, no aerodynamic qualities whatsoever. It launched attached to the Apollo Lunar Command Service Module, separated once it was in lunar orbit, and descended to the Moon with two astronauts inside. When their work was done, the ascent stage of the Lunar Module fired its own rocket to lift off the lunar surface and rejoin the Command Service Module in orbit.
Let’s look at the LEM, as it was called, in production those many years ago.
[Video clip presentation begins.] VIDEO: Across the country in Bethpage, New York, the Grumman Aircraft Corporation is developing the third section of the spacecraft called Lunar Excursion Module, or LEM. The LEM will be used for the actual landing of two astronauts on the Moon.
[Video clip presentation ends.].
MODERATOR: Gerry Sandler, why don’t you tell us a little bit about what role your team played in that historic landing on July 20th, 1969.
MR. SANDLER: Well, Beth, before I do that, Lauri indicated from our Lessons Learned meeting that I ought to tell a little story I told.
MODERATOR: Go ahead, then.
MR. SANDLER: Well, there was a famous paleontologist who was going around giving a lot of lectures, and each time he gave a lecture, he was making a lot of money, and he decided that because he was making so much money that he was going to get himself a nice big limousine and a chauffeur and he would go in style.
After a few months of doing this, the chauffeur says to him, “You know, after this great talk I have been listening to, I think I can do the same job you do. Why is it you pay me so little and you make so much money in your talks?” And the professor said, “Well, if you think you can do the same job, let’s change uniforms. I will stand in the back of the room, and you give the talk.” The chauffeur got up and he gave the talk, and the talk went very well, until one little fellow in the back raised his hand and said, “You know, Professor, that was a fine talk. I have a very simple kind of question.
You know those dinosaurs that you talked about that lived 65 million years ago and how they suddenly died out when a meteorite hit the Earth and how they got buried into the soil and then big trees grew and they died and got buried into the soil and then there was a lot of glaciation and icebergs grew and they retreated and then the plates moved the continents around? What I really don’t understand is how do you take all of those factors into account when you calculate the pH of the soil,” and the chauffeur says, “You know, I am really embarrassed that after this fine talk I gave, you would ask such a simple trivial kind of question, and just to show you how trivial it is, I will have my chauffeur in the back of the room answer it.” [Laughter.].
MR. SANDLER: And that is how we did it today, and I have 14 chauffeurs with me.
We spent most of the morning talking about lessons learned. I was the first speaker, speaking about the lessons we learned and sharing responsibility with NASA and how we got together to work as a team and to do all of the tradeoffs and the design missions that were necessary to make those tradeoffs and to build a vehicle or design a vehicle that had all of the reliability characteristics that we were looking for.
We talked about how the program was going to be managed and the engineering management was going to be done and the kind of culture that was established at both operations, at both NASA and at Grumman, to make sure that we paid attention to detail, that we had an open communication environment, and that we were able to challenge each other effectively, and that worked very well. So we were very proud of that because that made a big impact on reliability as well.
Then finally, I talked about the technical aspects of the tradeoffs that were done and how we achieved the various target weights and control weights and how we balanced reliability with the various weight objectives and how that succeeded and the kinds of things that happened during each mission that made the difference between success and failure.
Seymour, I will take a little of your thunder.
You spoke about the System Failure Mode and Effect Analysis.
MR. BERG: Well, I compared it with the estimating reliability, then calculating the reliability, and then pointed out that no matter what that calculation is, whether it be good or bad or better, we still are left with the requirements of needing single-point failures not to impede the mission success or harm the crew and told about the FMEAs that we performed and the modifications that we made to make sure that at least most of the single-point failures were eliminated by our studies.
MR. SANDLER: Bob, you spoke about testing.
MR. SCHWARZ: Yes. I told a little story that back in the days of the Lunar Module Program, we were always accused of having tested too much. Of course, one of the questions that came up, did we test enough, too much, too little, too late, too early, and the bottom line is when you asked the people why we tested too much — and this was in Aviation Week — the answer came back, because you didn’t have any serious failures, and that sort of befuddled me, didn’t turn around and say that, hey, the test program was perfect, you eliminated all the problems on the ground before you launched and before you landed on the Moon.
So testing is the typical no-win situation. If you don’t test enough, you are going to have failures and be criticized. If you have no failures, then you obviously tested too much because you didn’t have any.
Our goal was to design a test program in the blind. Remember, nobody had been to the moon. We didn’t know the environment. We didn’t know the profile. So we started from scratch and tried to grow a program, change it as we went along, and I think one of the key topics or events during our test program was what we called our Steps Stress to Failure Testing where we took most of our equipment beyond their qualification levels and forced them to fail, so that we knew what kind of safety margin and design margins we had in the event there were changes downstream, this all being done, once again, as Gerry said, as a team, NASA and Grumman and our subcontractors.
MR. SANDLER: Joe, you talked about subsystems and analysis.
MR. MULÉ: Well, we kind of went on both ends of the test. We were very, very much involved in subsystem reliability.
In the development of models and also performing Failure Mode and Effect Analyses, each Failure Mode and Effect Analyses were very instrumental in finding many single-point failures and correcting them during the design phase of the program.
We used our experience in the Failure Mode and Effect Analysis to participate in the Measurement Review Board where this is where we defined those minimal number of measurements that would go on board that would determine whether we aborted the mission or whether we had to take immediate crew safety action. These are called Caution and Warning Measurements.
Then on the other end of tests, we did Failure Analysis and Corrective Action. This was a very, very intensive program. It was very long-lasting, but this is where we absolutely tried to determine what the causes of the failures were, and we left no stone unturned. We wanted to determine what the corrective action was, whether that corrective action was actually effective, and I have to say that, you know, we started out with some young engineers from all over the country, and these guys were blended into a fantastic team of people who knew how to work hard, who had good humor, and I think that is the secret. We needed to have guys that could work hard, but at the same time knew how to have a good time if and when we had the time to do it.
[Laughter.].
MR. SANDLER: Well, Steve, you talked about End Item Engineering.
MR. ROCAMBOLI: Yeah. End Item Engineering was, I guess, a culmination of everything else that was going on. Everybody talked about FMEAs, failure closeout, and now I had the chance to go on a vehicle itself, the first Lunar Lander, and work with the people on the program who were actually doing the manufacturing tests and then use what I had learned over in the engineering and analysis portion to do things like closing out phase, to help our test discrepancies, pick up the slack where sometimes the test team ran into a hurdle and said we can do that, we can close this out, we can make it right, and let’s go forward.
It was an exciting experience.
I then transferred to the Cape. I saw a couple Saturn V launches, and it was very exciting, a very exciting part of my life.
MODERATOR: I want to take a minute to introduce the other chauffeurs who are with you.
[Laughter.].
MODERATOR: We are honored to have them here, some other members of the Lunar Module Team, Joe Fragola, Tony Coretti, Tony Califra, Dominic Levaccari, Matt Macchio, John Purcell, Lou Nardo, Ray Capiello, and Joe Whittenberg.
Now we are going to take some questions from the audience here at Headquarters. Is there anyone who has got a question about how these guys built the Lunar Module?.
QUESTIONER: Bob Zimmerman. I am a freelance space writer.
Actually, they brought that little model of Lunar Module. You know what it looks like, I know, but I have a question about it. Why is the module itself so faceted, the shape itself? Was there an engineering reason behind putting all those facets? Why not just make it either square or a triangle or a capsule size? Why give it the constant angles and twists and turns in the body of the structure itself?.
MR. SANDLER: Well, there are a couple of reasons for that. One, it primarily is the landing gear. We designed the landing gear to have pods on the bottom and the shape, so that we could land on the surface at the time of which we didn’t know that much. We decided it, so that it could land on a 6-degree slope in about 2 feet of depression and not too well understanding the surface friction of the lunar surface.
Simultaneously, you had to land on one part and then ascend back to the Command and Service Module with the other part. So that was two things that basically determined the shape of the vehicle.
MR. MULÉ: Keeping in mind also the fact that this was a vehicle that didn’t have ground rules with regard to — let’s put it this way. They had to operate in a vacuum. That design was one that allowed us to operate in a vacuum.
MR. SANDLER: It did not have to be aerodynamically good-looking.
MR. BERG: It had no flying qualities whatsoever.
QUESTIONER: Was there any, though, logical reason for the specific set module?.
MR. MULÉ: Save weight.
MR. SANDLER: Perform those functions at minimum weight with the characteristic I talked about, with the landing gear and the ability to separate the two stages.
MODERATOR: Do we have any other questions up here?.
QUESTIONER: Thank you. George Hanover, Congressman Rohrabacher’s office.
I was just wondering about the testing of the firing of the rockets to get you back in orbit. That seems like it would be especially a need to be reliable. Could you talk a little bit about that?.
MR. SANDLER: The main part, obviously, that you mentioned is the ascent engine, and we were worried about that at one point, and we wound up having some parallel programs between Bell Aerospace and Rocketdyne. Then we wound up taking the best of both. Then with the engine from Bell and the injector from Rocketdyne and running some special tests that the Air Force was doing on their rockets, that was basically what gave us reliability and confidence in that ascent engine because that is what brought us back home.
MR. SCHWARZ: That is a good point because we had discussed this once before where they asked us what did you consider the most critical part of the mission. That happened to be it.
This engine hasn’t been tested now for days and days and days, and it sits there. You don’t know what damage is created during the trans-lunar flight or the landing, and then you are hoping and praying that when they push the button, off it goes, and that was the most critical from our standpoint.
MODERATOR: Well, the LEM really was a remarkable vehicle. Why don’t we look at the approach and landing again. Roll that clip.
[Video clip presentation.].
MODERATOR: Steve Rocamboli, tell us where you watched the landing from and what feelings were going through your head at that point.
MR. ROCAMBOLI: Well, I watched the landing from Mission Support at Bethpage, and like everybody else, I was worried. I had confidence, but this was something that had never been done, and to say I had a sigh of relief, absolutely. It was a fantastic thrill. I still get chills thinking about it and –.
MR. SCHWARZ: You used to have a — [Laughter.].
MR. ROCAMBOLI: I don’t know what he said. I can’t hear anymore.
MODERATOR: What do you men tell your grandkids about this?.
MR. SANDLER: Well, I have a grandkid who is, I guess, 7 years old now, and he has a little wooden PlaySchool in the back yard, and we play Blastoff. We play Blastoff going to the moon and then returning to Earth, and who knows, maybe he will want to be an astronaut someday.
MR. SCHWARZ: We have the Cradle of Aviation on the Island, and every one of my grandchildren have been to it at least once where there is a full-scale Lunar Module.
MODERATOR: Let’s talk to some of the chauffeurs out here for a little bit. I think we have a wired mic here for you.
I am interested in your thoughts on what some of the most challenging moments in the Lunar Module effort were for you. Let me get this mic up here. Does anybody want to weigh in on that? How about you, Joe? I will volunteer you.
MR. FRAGOLA: The most challenging moment for me was being a young kid and coming in with all these guys who had been on for such a long time and trying to catch up with them. I mean, they were running at full speed, and the young guys who came in afterwards had to run even faster just to stay within reach of them, and I was always to this day in awe of the capabilities of these guys and the amount of information they were able to store and bring to the front of their mind so quickly whenever anybody asked a question. So it was tough for me coming in as a young kid to keep up with them.
MODERATOR: Anybody else? Oh, all the way down there. Maybe Melissa can get you.
PARTICIPANT: I think the biggest problem that I thought we faced was being accepted by the other groups.
Reliability wasn’t always looked at as, you know, a helping hand. We were many times criticizing design, looking at things, and I think the designers probably took offense to that in the beginning. So we had to build a trust with them, and we did over the years, and at the end, I think they thought of us a little bit differently.
MODERATOR: We have had a lot of interest in this topic from the centers, and so I think we will take a couple of questions that we have had e-mailed to us.
We have got one from one of the engineers who is heading up the new Lunar Lander effort. Let’s see if I can find that question.
Here we go. First, many people claim that NASA is much more risk-averse today than it was during Apollo.
Are there trades or decisions that you made for the LEM that you think would be unacceptable in today’s environment? Go ahead, Steve.
MR. ROCAMBOLI: I don’t know about risk-averse.
I don’t know what is in NASA’s mind today or what was in their mind 40 years ago, but they did some gutsy things 40 years ago, like — I think we talked about it — going to the Moon without a LEM. Don’t forget the Apollo 8 went to the Moon and went into lunar orbit without all of the backup that the LEM provided. I thought that was a pretty gutsy decision to make. I think NASA probably makes gutsy decisions today, also. I just think maybe we are not all aware of them.
MR. SANDLER: The other point I guess I would make is that on the Apollo LEM program, we and NASA were really a team. It wasn’t that NASA made decisions or that Grumman made decisions. We really made those decisions together. So I never really thought of it as one versus the other or their being risk-averse and we not being risk-averse. We sort of decided together what to do, and I would hope it is the same way today.
MR. SCHWARZ: We talked about a little bit today, we had a charge or a challenge. When the President of the United States gets up and says that we are going to land on the Moon before the turn of the decade, that sort of eliminates a lot of risk because you are going to make it.
At the same time, he said it worldwide. So you had the world watching you, and you wanted to make sure that you didn’t fail. That was very, very critical, and I think it created the team environment that we had.
You know, it’s funny. You mentioned our chauffeurs here, but this group, 14 or 15 of us, and we happen to be missing our boss who is 92 years old and he couldn’t come because he’s in Hawaii, but to this day, for the last 35 years, this group gets together and has a Christmas dinner every single year. That, I think is amazing and shows the true support and team work that we had.
MODERATOR: We have got a question that goes along with a topic that you discussed at your technical exchange this morning. This is also from the current Lunar Lander Team.
The LEM had severe weight issues fairly late in the design cycle. Knowing what you know today about the enormous weight challenges that any Lunar Lander would face, would you recommend that NASA start with any different approaches or different implementations?.
MR. SANDLER: Yeah. We spent a lot of time talking about that this morning, and I guess there were a couple of key ideas that we recommended. One is to have a Design Reference Mission very early in the design cycle, and that Design Reference Mission is used to size the tanks that create how much electrical power and other consumables, et cetera, and at the same time not only look at it like a vehicle by itself, like the Lunar Lander by itself, but to look at the total system, the booster, the CEV, and the Lunar Lander as a combination because the real question is where do we get the best reliability per pound.
Maybe we are better off with some in the Lunar Lander.
Maybe we are better off with some in the CEV and how much margin might the booster have. So you really need to understand that total tradeoff process, and I think that may be a little different, Lauri, than we did before.
MODERATOR: A few minutes ago, I saw a hand up here in the back.
QUESTIONER: My name is Devin Hahne. I am an intern at Goddard in the summer.
The question I have, is there any particular discipline of engineering that proved more or less challenging than any other, thermal, structural, anything in particular?.
MR. SANDLER: If I could answer that, I would say no because all the disciplines were part of the process.
You know, working on the program like the LEM, you really had to understand everything, electronics, all the mechanical systems, the hydraulic systems, the thermal systems, whatever. So I can’t say that one was more important because all of them were critical in making sure we had a reliable spacecraft.
The only discipline, quite frankly, that wasn’t well represented at that time would be what we would today call computer science. We had a minimum amount of software, and there wasn’t no computer science degrees back in the ’60s. So that would be the only difference I guess I could see, but all the disciplines are critical.
MODERATOR: And we have got a question by e-mail from the Kennedy Space Center in Florida. Can you share a few war stories about where your team used the Reliability and Maintainability Engineering discipline to make a difference?.
MR. ROCAMBOLI: Well, I guess one of the first things that happened when I went over to LEM V, we talked about it this morning. The LEM window blew up. If that had happened on a mission, we would have a real crew safety problem if the fellows were not in their suits.
The problem looked like the vehicle was severely damaged. Some of the senior engineers, one in particular came up with a plan, said how we could repair it, and he asked me to head up how we were going to clean it up, and cleaning it up was very important because it was going to be in a Zero-G, and that is when NASA came across, and they came up with guidelines — I shouldn’t say guidelines — criteria. This is what a clean vehicle is, and so we worked together for over a month to get LEM V which eventually landed on the Moon to get it into shape where it was a reliable vehicle with great integrity.
MODERATOR: And we know there was a time when the LEM was pressed into service as a lifeboat. That was on Apollo 13. Let’s look back at that.
[Video clip presentation.].
MODERATOR: I think most of us are familiar with the events that took place on Apollo 13, but just in case there is anyone here in the audience who is not recalling that, Gerry, can you talk a little bit about that and what role your team played in a safe outcome of that mission?.
MR. SANDLER: Well, first of all, back in the early days when we were doing these weight reliability tradeoff studies, as I said, we had a Design Reference Mission, and we sort of recognized using the LEM as a lifeboat because we had a free return trajectory, and we sized the Descent Propulsion System tanks to accommodate that kind of problem.
What we hadn’t done at that time was to size the electrical power and oxygen and lithium hydroxide canisters, those kind of things, to accommodate three men in a small capsule for a much longer time, but when it did happen, 200,000 miles or so from Earth, we recognized that was going to be called into action, and we knew it had to be the Descent Propulsion System that would bring them back, and that was like a 4-day return. So the first thing we did was to fire the descent engine on the far side of the Moon, so we could cut down the time trajectory by 12 hours, which did happen, but quite frankly, it was uncomfortable for three guys in the Lunar Module all this time because we had to turn everything off.
There wasn’t enough electrical power. They had to gerryrig between the Command Module and the LEM, enough lithium hydroxide to get rid of the CO2 in the cabin, but eventually that all worked, and when they finally got back into the Command Module and returned, we were able to jettison the LEM, and the LEM did its job.
MODERATOR: Did you have something you wanted to say, Joe?.
MR. SANDLER: We lost money, yeah.
MR. MULÉ: The other kind of scary part was the fact that the firing took place on the dark side of the Moon, and it wasn’t until the LEM CSM was in line of sight that we knew that everything had been successful.
MODERATOR: That must have been a really, really tense time for all of you.
MR. SANDLER: I was in Mission Control at the time, and I have to tell you, the president of the company called at that time, and I hung up on him.
[Laughter.].
MR. SANDLER: I had no time to talk.
MODERATOR: All right. Seymour, you look like you want to say something about that.
MR. BERG: I just want to say Fred Haise came to work for Grumman after that.
[Laughter.].
MR. BERG: He must have did something right.
MR. SANDLER: And I guess to add to that, when I retired, Fred gave me a little picture of the LEM saying on there, “Thanks for bringing us back alive.”.
MODERATOR: And where do you have that hanging now?.
MR. SANDLER: At home.
MODERATOR: Here is an e-mail question from a young co-op at Johnson Space Center in Houston, and you have talked a little bit about this, but this co-op asks, the LEM was quite a versatile vehicle, even able to serve as a lifeboat on Apollo 13. Could you comment on the design requirements and philosophies that existed for the LEM that brought about this flexibility and versatility? You have touched on it a bit.
MR. BERG: That is what brought us back. I thought what you just got finished saying is an answer to that.
MR. SANDLER: Well, talk about the alternate paths.
MR. BERG: Well, let me talk about that a little bit. You asked Steve before, where were you on the day when LEM V was landing. I thought, I was hoping you would ask me.
I was in Houston. I was in the monitor room for the LEM group, and we were there all day until they landed, went through the whole works, went back to my hotel and went to sleep — well, after a little party.
Then at about 2 o’clock in the morning, the phone rang, and it was the guy who headed up the group. He said, “Seymour, we have a problem here at Houston,” and then Houston had a problem. Anyway, he said when the guy was putting on the suit — I don’t know which one of them — to go out on EVA, he had broken the handle on the circuit breaker. I think it was CV-37 or something. He said, “Do we have a redundant breaker for that? Otherwise, we can’t get off the Moon,” and I had no schematics with me. I said, “Hey, I haven’t any schematics, but I suggest you call Steve.”
So Steve got awakened about 2:30 in the morning, and he had it in his head. He knew of another breaker which was redundant to that, told him which one it is, and he felt comfortable. He asked me to come in. I got there about 4 o’clock in the morning, and then we stood by the telemetry waiting to see what was going to happen, when they took that circuit breaker. They couldn’t move the circuit breaker 37. They took their pen and pushed the button down, and that turned it on, and then they put on 11 also, and we knew darn well they were getting back.
MODERATOR: And that was one of those fancy antigravity pens. Right? [Laughter.].
MR. SANDLER: Let me answer the question, though.
We had some very basic design philosophies. One, of course, was to have as many alternate paths as possible, so that if one system failed, we had a backup system.
The rules were, of course, if one failed, you aborted, but on the other hand, you could also continue.
With some of the missions, if you look at the records, you will see that when one failed, we did continue. There was a lot of component redundancy, and there were a lot of measurements, so that Mission Control always knew what was going on, and in fact, again, if you look at the flight records, you will see that a lot of things happened during those flights, a lot of anomalies, where Mission Control was able to go through the simulated change, what had to be done, give new commands, and made those missions successful.
So fundamentally, the design and the Mission Control and the test programs were the real philosophies that made for reliability.
MODERATOR: Great. Let’s give the audience a chance to ask a few more questions. I see a bunch of hands in the back.
QUESTIONER: Hi. Andrew Barber, Aerospace Industries Association.
Probably 10 or 12 years ago, you had a chance to see at least in the press, the Russian version of the Lunar Module. Where did you see similarities? Where did you see differences, and on any of the differences, did you say we had considered that and had taken another path for whatever reason?.
MR. SANDLER: None of us know here. We really need chauffeurs now.
[Laughter.] MR. FRAGOLA: The Russians took a different approach in that they had a drop stage. Rather than taking the descent engine all the way down to the surface, they dropped off a descent stage and actually used the ascent engine to do the terminal phases of landing, which meant that they would have had to have that ascent engine active and wet and take off on the lunar surface with the same engine they came down on, and so it was a fairly different philosophy.
In fact, that is one of the alternatives that the LSAM team, the Lunar Service Activities Module team, has looked at as an alternative way to get to the lunar surface. There are certain advantages to having a drop stage, and the disadvantage is that you have a wet ascent stage, but if you haven’t seen it, you should take a look at what the Russians had been doing. It was quite different than our approach.
MR. SANDLER: Good.
QUESTIONER: Hi. I am Kathy Nado from Computer Sciences Corporation.
I had a question with regard to how your organization interfaced with what might be called a “systems engineering organization” now and talk a little bit about that tension, if there was any.
MR. SANDLER: I won’t call it tension, although Ray, I guess, did.
We were part of a systems engineering organization. We are not an assurance function, like some organizations that handle reliability or quality. We were part of the systems engineering organization, and therefore, we were part of systems design. We were deeply involved in all the tradeoffs and part of that whole tradeoff process. So it is very different than an assurance function.
Is that answering your question? Good.
QUESTIONER: I am David Schuman from the Goddard Space Flight Center. On behalf of the current NASA employees, I want to say thank you. You guys are the real heroes in the program and don’t get as much publicity as the astronauts.
My question is, with respect to the hardware that is left on the lunar surface, do you have any feelings as to what should be done with that? Do you have any sentimental feelings about it? [Laughter.].
MR. SANDLER: Well, I am not sure, but we were talking at lunch about the Surveyor. Apollo 12’s job was to go back and land near the Surveyor and to retrieve its camera. When they brought back the camera, they had found some bacteria that was still alive that had survived on the lunar surface. That must have gotten there from JPO and — [Laughter.].
MR. SANDLER: It wasn’t Grumman.
So there probably are advantages of going back and retrieving some of that because I would probably like to see whether there are things that corroded over a long period of time.
One of the things we worried about was stress corrosion, and so I would like to see some of that hardware, if we could retrieve it, and see what that really means in terms of reliability for the next missions, and the dust, of course, the dust is probably one of the major problems that we would worry about for a long stay now, and because it has been there for so long, you would want to see what the dust has done basically to the various components, electronics or mechanical systems.
MODERATOR: Who else has a question?.
QUESTIONER: I am with NASA Headquarters, and I am involved with writing requirements, Level 1 requirements for Exploration.
When you talk about reliability and setting reliability criteria, like loss of mission or loss of crew, how did you go about setting that? How did you go about enforcing that? How did that work as the discipline, and is it more of an art than a science?.
MR. SANDLER: Well, we talked about that also at lunch in terms of how far down do you go in this requirements generation process, and if you really think that a lot of these decisions require real tradeoffs, the more requirements, the more detailed requirements you generate, the more you constrain the ability to make the tradeoffs between things like weight and reliability. So there is a very fine balance, and I am not too sure I know where the line is because there is really a very fine balance between how detailed you make those requirements or how functional you make them, so that you leave much more room for these tradeoffs, and then you can see where you get the most sensitivities and pick the ones that give you the best returns.
MODERATOR: Any other questions?.
QUESTIONER: A.R. Hogan, science writer and also University of Maryland journalism doctoral student working on a dissertation about the history of the space program coverage.
Can you talk about the impact and value of having the American people and folks around the world able to watch the fruits of the labor of the folks at Grumman and NASA and 400,000 people on the team that made the lunar landing possible? Also, specifically, CBS News had a full-scale Lunar Module at Bethpage, Long Island, as part of their coverage, and I was wondering if you have any anecdotes or recollections about that as well, please.
MR. ROCAMBOLI: I think, in general, having the public involved and supportive was a tremendous boost. I mean, we had politicians. We had the public. This was the highlight of people’s lives. Every place you went, it was on TV, and you made sure that you contributed your part.
You did not want to do anything but make sure that we landed on the Moon on time and those fellows got back.
MR. SANDLER: Because of the public relations that was done in the public affairs or whatever the correct term is and everybody was conscious of it, we were able to attract the best and brightest people to work on the program, and I think if it is not recognized that this is a major national priority, that people aren’t going to be as anxious to work on these kinds of things as they were back in the ’60s.
MR. SCHWARZ: We talked about that as being one of the major differences on the upcoming program as opposed to the past. We were being watched by the entire world, and we were not permitted to fail. I am not so sure there is that drive.
We mentioned that maybe the possibility if somebody can convince China to come out and announce they are landing on the Moon in 2012, we might be able to create the same competition that we had when we were trying to beat the Russians, and that was truly the old American spirit. We are going to be number one and the best.
So I think that it was very, very advantageous to us as a program because it made us capable of getting things done that we didn’t think could get done.
MODERATOR: I would like to hear a little bit more about that. Would any of you like to expound on –.
MR. MULÉ: Well, I think that we had a situation in the country where there was a lot of different kind of motivation. There was a political motivation. There was a matter of national pride, and at that time, I think we probably had a very, very large pool of engineering and scientific people, and one of the things that concerned me when the program ended was the fact that we lost a lot of engineers and scientists, and I was saying that it would probably take about 15 years starting from scratch to be able to rebuild the kind of force, technical force, that would really be required.
MODERATOR: I think we have a question on the other side of the room.
QUESTIONER: Loretta Whitesides, Aviation Week Online.
We now have a new generation of young engineers trying to build a spacecraft for their first time. What was it like for you giving advice or feedback on your experiences this morning, and what advice would you have for these young engineers, or what plans do you have that are worthwhile and that you see as staying involved in helping them versus letting them try to come up with sort of their own way of doing it?.
MR. SANDLER: I think we have had the best answer for that when you talked about what you looked at and you rediscovered that we had already got there.
Say it.
PARTICIPANT: I think in general terms — I am Clinton Dorse [ph], the deputy project manager for the new Lunar Lander Office.
In general terms, I was giving them a compliment, not that they weren’t already proud enough for what they had accomplished, but every time we peel the onion on trying to do a new design and we get back to looking at what they did, it was a significant accomplishment. Every time we think we might have a cute answer to something, we get back and realize they really had the right solution.
So it is a testament to what they actually accomplished and how eloquent.
That vehicle might not look very pretty. You know, we had questions about the shape of it and so forth, but the reality is that there is a very optimized and amazing vehicle, and so we learned a lot from what these guys accomplished. It is amazing that it was accomplished 40 years ago.
MODERATOR: Here is a question from KSC. What kind of software quality assurance practices did you have in place to reduce programming errors? [Laughter.].
MR. SANDLER: We spent a lot of time talking about that today.
MR. ROCAMBOLI: And ended up with it didn’t exist.
MR. SANDLER: And quite frankly, we had very little, but there was also very little software on the LEM program. On Apollo 5, we had a LEM Mission program which was supposed to turn the descent engine on and then separate the ascent engine, and the program really didn’t keep up with the amount of pressurization that we put in the tank. So that had to be changed in real time.
And we were talking about the fact that on the new program, there will be a lot more software, and so it is going to be a lot more complex, but in all honesty, on the LEM program, we had very little software and, therefore, very little software reliability.
MR. MULÉ: Actually, we did have software, but I think there was probably more software in ground support than there was –.
MR. SANDLER: Probably.
MR. MULÉ: — on the vehicle.
We talked this morning about software reliability. Actually, software quality assurance did have a leg up on us, and it was a very, very tough sell. It was one of those areas where we had to assure the designers.
We had to ask them to open up the door not only to us, but to the user.
One of the problems was that in the very early days, software seemed to be only the province of a designer and very often did not have any feedback to the user. I think that was a big thing with software quality assurance.
MR. SANDLER: And by the way, on Apollo 11, probably the first anomaly before landing was a computer overload, and they had to dump a lot of that data and reboot it, so that they could handle the data that was coming in.
But to give you some idea, there was no more than 10,000 lines of code on the Apollo program, which you have in your hand calculator now.
MODERATOR: I am also curious what personal relationships you had with the Apollo astronauts.
MR. SANDLER: Well, I had a lot.
MODERATOR: Can you talk about some of those without breaking privacy?.
MR. SANDLER: No privacy. I was very close to Fred Haise. Fred Haise eventually came to Grumman, and Fred Hayes eventually came to our committee. So I was close to Fred. I was close to Gene Cernan. We became pretty good friends over the years, and of course, I knew [Jim]McDivitt and the other fellows, and everybody in Grumman really knew the astronauts because they spent a lot of time at Grumman, and one of the major reasons they were there was not only to check out the equipment and those things, but their visibility made a big difference in everybody’s feelings about what they were doing. They realized that what they were doing meant the lives of these guys. So they took it all very seriously. So the astronaut relationship was very crucial in this whole process.
MODERATOR: Favorite memory of your interactions with those guys?.
MR. SANDLER: Fred Hayes, his love of hamburgers.
[Laughter.].
MODERATOR: I see another question over here in the front.
QUESTIONER: Marsha Freeman with 21st Century.
NASA is looking now at sending people to a part of the Moon that we haven’t been to before, and of course, on Apollo 11, people had never been there before. I was wondering — and of course, we had a number of the unmanned precursor missions and had filled in a lot of information about what the lunar surface characteristics would be, but there I am sure was a certain amount of uncertainty in terms of what the conditions would be for landing the Lunar Module.
How much margin did you have to build into it in a certain sense to accommodate how much uncertainty you felt there was and what the surface conditions would be?.
MR. SANDLER: [Inaudible] and 2 feet of dust and surface friction, but as we talked about this morning, they nor us obviously really understand the environment in the polar regions. So I am sure they will have to send some kind of robotics there to find those things out.
Do you want to comment on that? MS. HANSEN: We are just right now in the stages of trying to determine what our test program will be, and we will be looking at all sorts of options in terms of how we get data.
There is, as you know, a Lunar Reconnaissance Satellite that will be collecting data. So we will be exploring that as we proceed with the project.
MODERATOR: And I have to say we have been remiss in not introducing you. We are very glad you are here.
This is Lauri Hansen who is the head of the Lunar Lander Project Office at Johnson Space Center.
I think we have another question over here on this side.
QUESTIONER: Bob Zimmerman, freelance writer again.
Most of the discussion here has been about Apollo 11 and the first LEM that landed, but there were five other landings. I would be curious to have your reaction. Did you start to get nonchalant in later missions?.
MR. SANDLER: No, absolutely not.
Again, one of the things we talked about this morning was Apollo 14 where one of the first things that happened during descent was switching to the abort mode.
The toggle switches got shorted out when they went to the abort mode, and a lot of that software had to be rewritten while they were in flight by the crew in Houston, so that they could go back to the primary mode. While they did that, the landing radar stopped working, and they had to press the circuit breakers to get the landing radar to work.
So a lot of those kinds of things happened, quite frankly, during each mission, and a lot of the problems that were solved got resolved in real time by Mission Control, and the last mission, which was Apollo 17, which was the best of all the missions, had no anomalies. We finally reached the reliability codes at the end of the program, but if you go through every one of these missions, you can see what the anomalies were and what corrective action was taken in real time to make those missions successful.
MODERATOR: After those six missions, as the missions wore on, public interest sort of waned. How did you feel about that?.
MR. SANDLER: In all honesty, I didn’t think about that. We were so busy working each individual mission and trying to make them successful that we really didn’t think about it, at least I didn’t think about the public waning.
MODERATOR: Any of the rest of you?.
MR. SCHWARZ: I think that I brought up the point today that that is where we as a team — NASA, us, the Government, us subcontractors — failed because most of the public looked at it as we only went to the Moon to pick up some rocks and come home, and there have been so many things that occurred in this world today, everything, things that you touch every single day that the world doesn’t know was a derivative of something that started with the space program, not necessarily the lunar program, and I think a better job of marketing the accomplishments is going to attract the public interest again.
MODERATOR: A question up here in the front row of the back section.
QUESTIONER: Thank you. This is still George Hanover from still Congressman Rohrabacher’s office.
I am just wondering with the situation with kids today — and I am talking, oh, maybe K through even 8 — how do we get them motivated and interested in science and technology? You fellows were in the front line when the space program was first started, and there was a lot of interest then, but maybe nowadays, we don’t see that as much, and I think maybe your opinion would be very interesting on this issue.
MR. SANDLER: I will give you a roundabout answer, if I could. I was part of a group that was trying to build a museum on Long Island called the Long Island Museum of Science and Cradle — well, not the Cradle — the Science and Technology Museum which is part of the Cradle of Aviation, and the real issue was to make it a science museum that would attract kids, so that they would want to study science and engineering.
It is very interesting. If you look at the statistics, specifically on the Long Island area, you find less and less youngsters are studying that because they are finding that there is either more money in financial areas or whatever than there is in science and engineering, and quite frankly, when we were growing up — maybe I am speaking for myself, but when we were growing up, we thought we did good jobs at science and engineering, and most of us did.
I don’t think the kids think that today. They think they can get a good job anywhere, so why not take the easy way out and study something that is easy.
Do you want to answer that, Seymour? You teach.
So why don’t you? Seymour is a professor at Hofstra, so go ahead.
MR. BERG: Get an MBA and go.
[Laughter.].
MR. SANDLER: Well, why don’t you answer that? Come on. You must have an opinion.
MR. BERG: I answered it. That’s the answer.
MR. SANDLER: Get an MBA and go?.
MR. BERG: No. I’m a little worried about the education system today altogether, plus the fact that until very recently, I don’t think there was really any real great demand for engineers in particular, but it should be picking up with this kind of action or this kind of work, and hopefully, we can get some youngsters interested, but let’s face it. It’s difficult.
A degree in psychology, that is no problem. A degree in economics, like one of my kids got and then went on to be a good businessman, that was a cinch, but the other guy with his Ph.D. in industrial engineering and a bright and helpful computer science man, he makes a living.
What was the expression some years before about engineers? Engineers are in the middle someplace. They are not bringing in the big bucks, and that is the point you made, and there is no question about it that it is hard to get people today to go into that.
MR. SANDLER: When there are easier degrees to get.
MR. BERG: That is right, and big bucks in their jobs in business. I’m going to get myself an MBA, too.
[Laughter.].
MODERATOR: Well, this discussion sets us up really well for our next clip. Let’s take a look at that.
[Video clip presentation.]
MODERATOR: Well, when we left in 1972, we didn’t know if we would ever go back. Now we are aiming to go back and to stay there by 2020. What lessons from Apollo 11 and Apollo 13, the other Apollo missions and your work on the Lunar Module do you think apply to NASA’s new mission? Sum up maybe the top couple of lessons.
MR. SANDLER: Well, I think probably the most significant thing is what we learned in the systems integration business of how to take diverse technologies, many disciplines, many different kinds of people with their different skills, and make a system that is directed towards an individual mission, and that systems integration capability, quite frankly, doesn’t exist in a lot of engineering areas, and if you think about what we did, it really was a systems integration kind of job that put everything together in many different ways and use many different skills from design to testing to failure analysis and Mission Control, all of these things together to accomplish a mission.
MR. SCHWARZ: The one thing that I think I can point to then that is different today — and I don’t know how you are going to get back to it — is we had a seamless integration team. We were one team. There was no NASA is they and Grumman is we. We were together.
I am not so sure I see the unified grouping with contractors, not just as NASA, but the Government today, and I am sure it is created by some of the problems they have had in subcontracts, but that is key and very, very important to make a program like this work.
If there is a “we” and a “they,” you are going to have a very, very difficult time in succeeding.
MODERATOR: How about one more question up here?.
QUESTIONER: Hi. I am actually an intern at Goddard this year, and I was just wondering, what were you looking for to after the Apollo missions and stuff like that. What was the next challenge after going to the Moon as opposed to right now? Obviously, we are going back to the Moon. So that is not a very original idea, but were you thinking that we were going to go to Mars 10 years later, or did that just kind of drop that?.
MR. SANDLER: Well, in our time, the next mission was the Shuttle mission, and we bid on the Shuttle and unfortunately did not win it, and obviously, we were very disappointed.
MR. ROCAMBOLI: I think there was more to it, Gerry. There was another plan that never went into effect.
MR. SANDLER: Well, that’s right.
MR. ROCAMBOLI: After the LEM program, we were supposed to colonize the Moon. It was not just land on the Moon and that’s it. There was another plan and plans to actually have a base on the Moon, to have lunar orbiting vehicles there constantly, Space Station, and transitions from lunar orbit to Earth orbit. None of that happened.
MR. SANDLER: And the Apollo program got reduced from 20 vehicles down to 17.
MR. ROCAMBOLI: Absolutely.
But there was a very ambitious exploration of the Moon that was planned that never happened.
MR. SANDLER: Speaking personally, quite frankly, and trying to relate to you as an intern, when the space program was over, the LEM program was over, I personally saw my future in other areas, and so I went into the aircraft side of the business and eventually wound up doing those kind of things, and then eventually, I went over to the data systems side of the business and ran those kind of things.
So I think unless those things are continuous and there is a continuum of activity, people do lose interest and will find other opportunities for themselves.
MODERATOR: All right. We have got time for one last question, right over here.
QUESTIONER: Warren Leary with the New York Times.
Obviously, your group published a lot of papers and you had documentation that the current engineers have read. What do you think your discussions with them has added to that body of information they already have, and then secondly, in your discussions with them, did they ask questions of you that kind of spurred some memories and thoughts that you hadn’t thought about in a while?.
MR. SANDLER: Well, I will admit that our discussions today brought back lots of things in my mind that I didn’t even know I knew, and hopefully, I am right about what I said, but yeah, a lot of things came back from our discussions, and hopefully, that helped the NASA people as well.
I don’t know about the documentation. Do you want to answer that?
MS. HANSEN: Certainly, there is lots of documentation from the Apollo era. To me, that is not nearly the same thing as talking face to face with the people that were involved in building the hardware and learning lessons the hard way and getting the real experiences. You get a totally different flavor from sitting down and reading a pile of paper versus talking to the engineers that have really been there.
PARTICIPANT: I think I will add to that, that it is imperative that we learn from what these gentlemen accomplished. The documentation typically has the facts as they ended up, not necessarily the understanding of how they got to those.
So, as we look at developing the new Lunar Lander, we don’t want to have to iterate through the same problems and so forth that they went through. We want to learn from those experiences.
MODERATOR: Great. I just want to say thanks to all of you for being here, your willingness to come and share your experiences with us.
Doug Cooke, our Deputy Administrator in Exploration Systems, has something he would like to say to you.
MR. COOKE: Yes. I would like to add my thanks, very sincere thanks for your being here today. It has meant a lot to us. It has been an honor for you to be here and to share your experiences. They are experiences that are somewhat different than recent experiences in human space flight in terms of the work you did on a very unique spacecraft.
It is information, insights, and experience that we need to understand and build on as we build the next vehicles to go back to the Moon with hopefully more people, for longer stays, and just the beginning in exploring our solar system.
Thanks very much.
[Applause.].
MODERATOR: Again, thank you all very much. We also want to thank Northrop Grumman for their assistance with the pictures you have been seeing today and with the Lunar Module they sent us.
And that’s it. Our time is up. Thanks so much for joining us today.
[End of conference of July 20, 2007.]
