The Ares PDR Was Not As Smooth As NASA Says It Was

Editor’s note: NASA recently concluded a PDR (Preliminary Design Review) for its new Ares 1 rocket at NASA Marshall Space Flight Center in Alabama (charts).

Despite glowing comments about the success of the review and the smoothness with which it operated, many of the participants seem to have a different point of view:

“Too many people involved in the planning phase, meetings were too large”; “The integrated vechicle review did not present the element design issues (RIDs) so it was difficult to know if the parts added up to a rocket that will fly”; “The review occurred to close to the element PDRs, This did not allow for some of the element level rids to addressed or predeclared in documents”; “Much of the documentation presented for PDR was not mature enough for PDR. This limited an effective of these documents and left the impression that the PDR was rushed.”; “The RID screening rules and procedures seemed to change from day to day, like we were making it up as we went along.”; “Insufficient time was allotted to review the documents.”; “Not allowing RIDs to be written against the SRD and declaring it a finished document prior to the PDR was just arrogant and wrong. This was further evidenced and confused by the introduction of two version of the SRD, showing that it was in fact being changed behind the scenes.” etc.

Below are verbatim commments provided to an online review website by the actual PDR recipients. The deadline for adding comments was today (23 September 2008).

Ideas: What did we intend to do, and did we do it?

1. Execute an effective PDR for all phases

2. We intended to deliver a well planned and execution of the PDR Kick Off and DDP, with all technical issues worked behind the scenes, invisible to the customer.

3. To train the participants so that they would be able to function effectively. I thihnk we earned a 7 out of 10 on did we do it

4. The intent was to make sure all DDP participants knew what was expected from them with respect to presetation content, time allowed and presentation format. I would term this partially successful

5. The PDR followed the NASA PDR criteria but the presentation of the design documentation made it difficult to evalaute the Preliminary Design.

6. I thought it went very well.

7. The best was to address this is how effective we are in addressing and resoving the issues raised during the PDR by CDR.

8. 1. Training was a joke. we had 40 people in a conference room that held 20 + about 13 in the hall way.

9. 2.Who was the customer? NASA HQ, ARES PO, the elements. Invisiable from who????

10. The PDR team did not execute the plan that they somewhat trained the participants in. Their failure to adequately inform the participants of the changes in the plan damaged the adequacy and credibility of the review.

11. We intended to judge the preliminary design against the requirements. We ended up doing a reqts review for the most part.

12. The focus on most of the above comments seems to be on process rather than purpose. This also seemed to be the mindset of the PDR planning goup: Rid Training, RID processing, Kick off logistics, design presentations, team processes, etc. etc. The real purpose, evaluating the preliminary design against requirements, seems to have been lost in the minutiae of RID processing without a comprehensive evaluation of design against requirements.

13. What should have been doing is correcting design and concept defects; instead we worked very hard to kill RIDs. I guess it’s not a defect if the RID is rejected for missing or incomplete “from/to” language.

14. Because of the short review time for RID review and dispostion, RIDs were rejected based on technicalities and the underlying issue described in the RID was not addressed.

15. The process was well thought-out, however, this design review as in others is made up of participants that are not familiar with the process at all. I recommend two things to preceed any training: 1. Clear definition of roles (a list of all the roles, what does each role entail), and 2. A list of the steps from pre-RID to RID to closure. My document was effectively reviewed; just not efficiently.

16. From a Board perspective, the review went well. The Exec overview during the kickoff served is purpose. The board meeting was reasonably crisp and offered ample airing of the issues that surfaced. However, as was presented, and as shows in the comments, there were several disruptions and missteps along the way. These took extra-ordinary effort to deal with. So, from my perspective, it would be very important to review these lessons learned and incorporate corrective action in future similar reviews.

17. We intended to confirm that the design process had matured sufficiently in addressing the requirements commensurate with the Preliminary Design Review criteria. We only confirmed where we are in the design process and captured the shortcomings through stoplight metrics and actions.

Ideas: What worked well, and why?

1. Kickoff worked well, because the Project Office had staff that was experienced in conducting the kickoffs

2. The DDP worked well, because the Project Office had staff that was experienced in conducting kickoffs, and this was very similar to the kickoff.

3. Things went well because the Project office AND Engineering worked together as a team!!

4. Many of the team tabletop activities went well. The most successful ones were the ones who had a plan of how to conduct their team meetings and a schedule for getting through their activity, and also had the better facilities in building 4205.

5. The wiki site was a great source of information

6. The wiki sit helped me find the information quickly.

7. In my opinion, that is the only way the PDR could be successful.

8. The invidividual reviews were performed in a very efficient and organized manner.

9. The teams performed well, especially given the timetable and changes of course during the review.

10. Free coffee. Many thanks to the occupants of 4205 for their generosity

11. The Ares1 Project Coordinators saved the day despite the verbal abuse, neglect (the RID coordinator generally could not be found), constant change and near 100% lack of direction from the two people tasked to lead the work. If not for the professionalism of these Project Coordinators, the PDR review would have been a disaster. It would be a welcome change if they could be rewarded/acknowledged for the herculean efforts to make this review a success despite the actions of the two people tasked with organizing and running it.

12. The Kickoff was great. A large auditorium with actual space between seats and a well organized presentation. The agenda was followed. Gray Research is nice for smaller events, but could not handle the PDR

13. The commuications with the board went very well. The Wiki, the exec overview, and the board meeting communications through e-mail all met their purpose and made serving easy

14. Opportunities for stakeholders to have insight and input in the process is important and we did a good job of making those opportunities available to all.

Ideas: Based on what went well, identify what we should keep doing. Please prioritize your list. (Be sure to include which Phase and Area of that Phase you are referring to when you enter your responses)

1. Wiki

2. Face to Face Tabletops

3. Sequestering teams away from their normal work sites

4. RID Tool “Reporting”

5. 1. Face – to Face meetings

6. 1. Face-toface meetings2. WIki Page

7. sequestering, face to face meetings

8. Continue the open face to face forums, and pursue better meeting space on site.

Ideas: What didn’t work well and why?

1. Need more than one person that understands the ENTIRE process and can help with answering questions, emails, phone calls, etc.

2. Too many people involved in the planning phase, meetings were too large

3. Internal communication was not integrated

4. Internal communication was not integrated

5. The RID tool access was too limited to allow everyone who needed access to the system. Many cases only a few people were able to enter rids for an entire branch.

6. Too many people involved in the planning phase, meetings were too large

7. Need more than one person that understands the ENTIRE process and can help with answering questions, emails, phone calls, etc.

8. RID tool was cumbersome and still is as we try to address the RIDs.

9. The presenation of the design was not well laid out. A Product Breakdown Structure (PBS) may help.

10. The integrated vechicle review did not present the element design issues (RIDs) so it was difficult to know if the parts added up to a rocket that will fly

11. The review occurred to close to the element PDRs, This did not allow for some of the element level rids to addressed or predeclared in documents.

12. The table tops seemed to have no agenda and were always full of people who seemed to just be observing the meeting, making it impossible for people with technical issues to attend.

13. Time was limited for the book-mangers to work RIDs with the initiators

14. Comments made by the “screening” team, when returned to the review team, were more often than not confusing to a point that the question was often raised as to the background of the screening team and did they understand the issue being discussed on the RID. This may have been a problem with the way the RID tool was being used/implemented.

15. People assigned to perform document reviews were also assigned to the screening team, leaving no time to actually review the documentation.

16. The text fields in the MSFC RIDS tool only allow 2 or three lines of text in a narrow window to be shown. This made it difficult to review RIDs in the screening team with a projector. The RIDS tool should be modifed to allow showing all the text in any field, not just a small slice of text.

17. Building 4205 was a very inconvienent place to hold the RID screening tabletop meetings. The Gallery room in 4205 is an echo chamber and it is impossible to conduct an effective meeting there. Parking around 4205 is limited, and the Marshall cops were giving parking tickets to people for parking on the grass or on the roadways. Most people had to park at the 4200 complex and walk over to 4205. The people normally resident in 4205 were resentful of the PDR visitors and got upset when we used the copiers, coffee machines, vending machines, etc. The building has limited restroom facilities. For a major review such as the Ares I PDR, the project needs to find a better venue in Huntsville that provides adequate parking, excellent wireless access, multiple meeting rooms, well-stocked vending machines, and access to restaraunts. The Von Braun Center downtown or the Jacobs Conference Center are possibilities.

18. Building 4205 was a very inconvienent place to hold the RID screening tabletop meetings. The Gallery room in 4205 is an echo chamber and it is impossible to conduct an effective meeting there. Parking around 4205 is limited, and the Marshall cops were giving parking tickets to people for parking on the grass or on the roadways. Most people had to park at the 4200 complex and walk over to 4205. The people normally resident in 4205 were resentful of the PDR visitors and got upset when we used the copiers, coffee machines, vending machines, etc. The building has limited restroom facilities. For a major review such as the Ares I PDR, the project needs to find a better venue in Huntsville that provides adequate parking, excellent wireless access, multiple meeting rooms, well-stocked vending machines, and access to restaraunts. The Von Braun Center downtown or the Jacobs Conference Center are possibilities.

19. Much of the documentation presented for PDR was not mature enough for PDR. This limited an effective of these documents and left the impression that the PDR was rushed.

20. The RID screening rules and procedures seemed to change from day to day, like we were making it up as we went along.

21. The CSRT detailed design presentations were missing major content such as graphs and tables, probably because they were generated on a Macintosh computer and the PowerPoint files were incompatible with the Windows versions. All presenters should have used Windows versions of Powerpoint to produce their slides.

22. The RID tool does NOT have a capability to retrive RIDs lost for nay reason.

23. RID training was conducted in an incompetent fashion. We had 40+ people in a room designed to hold 15-20. We had 10-15 people sitting in the hallway outside listening in.

24. The RID tool can not recover issues lost for any reason. A daily backup would be of great benefit. One full day of work entering RIDs (by multiple people) was lost and had to be repeated.

25. Insufficient time was allotted to review the documents.

26. Bldg 4205 was a very bad choice. That building was not designed to handle such an event/meetings. The selection and use of 4205 reduced the effectiveness of the review. People who made this selection did not understand what was expected or needed by the review teams.

27. Single point failures in communication and planning impacted the effectiveness of the PDR.

28. Several presenters did not know what was in there charts or had not seen them prior to presenting

29. Definition of “Editorial Issues” not consistent between teams.

30. Developer of wiki page was not given credit for his work (see source code of actual developers name). Credit was given to someone else. Not ethical.

31. Meeting room facility needs and request ignored.

32. Request to have room cleaned ignored.

33. Planning and notices of “overbooking” of kick-off and detailed design review had effect of discouraging people to participate in PDR.

34. Informal notice that people who might be admitted to kick off could be asked to leave to make room for VIPs discouraged attendance.

35. RID tool passwords and usernames shared with others (beyond RID tool account holder) by RID coordinator. RID tool usernames/passwords not secure.

36. RID tool passwords and usernames shared with others (beyond RID tool account holder) by RID coordinator. RID tool usernames/passwords not secure.

37. ABUSIVE STAFF LEADERS

38. NO ONE WOLD TAKE RESPONSIBILITY FOR GETTING THINGS SETUP AND PROBLEMS WORKED. CRITICAL NEEDS IGNORED.

39. MANY OTHERS DID PDR PLANNING WORK ADN SUPPORT, BUT CERDIT FOR WORK COMPLETED WAS TAKEN BY SOMEONE ELSE.

40. STAFF OVERWORKED, NO DIRECTION, SOME EMAIL FROM rid CONDINATOR VERY ABUSIVE TO TEAM MEMBERS

41. Restrictive invitations to PDR presentation and RID participation greatly reduced the capability of potential participants to provide a review of the vehicle design.

42. Not allowing RIDs to be written against the SRD and declaring it a finished document prior to the PDR was just arrogant and wrong. This was further evidenced and confused by the introduction of two version of the SRD, showing that it was in fact being changed behind the scenes.

43. This one goes to both this team and those above them. It is impossible to have adequate review of parts or the integrated vehicle if the schedules for other Elements does not allow for participation.

44. Allow adequate time for issues raised in Element or sub-system reviews to be addressed and brought forward. If we are actually building an integrated vehicle, then we need to pay attention to all the parts. We were directly told in training that the results of the US and Avionics reviews didn’t matter to this review.

45. Not enough actual design documentation was available for review, many of the products were in poor shape for a pdr. Not enough actual design documentation was available for review, many of the products were in poor shape for a pdr.

46. Facility was inadequate/noisy. No place to sit quietly and review documents.

47. Documents were printed out for reviewers use but were not clearly marked-took a long time to find what you wanted and this could have been alleviated easily by labeling the notebooks with doc names.

48. I don’t care who at NASA the RID coordinator is/was sleeping with personal abuse of team members is wrong!

49. My overriding concern with the PDR is the lack of ethics displayed by the contractor RID coordinator and two NASA managers. The contractor RID coordinator was given tasks by NASA and did not do them, when timing was critical she passed the work onto others, then had the finished work sent back to her. She then uploaded the products as her own taking full credit work she did not do. This was known by the NASA PDR lead. Nothing was done due to the sexual relationship between the RID coordinator and a Sr. NASA manager. The lack of ethics and standards, the dishonesty and overt favoritism is damaging this project and will if left unchecked endanger the Ares 1 program (as it will spread) and one day Astronauts. We need to get back to building rockets, not hiring girlfriends. 1.

50. Need to define exactly what should be done/completed prior to tabletop meetings during PDR. Book managers needed to already have worked issues/comments/editorials with reviewers prior to attending PDR tabletop with the sequestered team. They needed to already have worked whether they accepted/rejected their comments before entering. They needed to check the RID tool themselves for their documents to see if any RIDs had been entered against their document. The tabletop meetings were not meant to work issues, unless a sequestered participant had a question/comment about a RID.

51. I do not believe the sequestering of the review teams worked well for non-local people supporting other projects. Complaints were voiced about lack of Orion support of the Ares PDR, imagine the outcry if Orion would have requested 3 contiguous weeks of sequestered Ares support a month or two before the Ares PDR. It was just not realistic.

52. The option to print a full RID report in the RID tool should only be available to certain users and not everyone registered to use the tool.

Ideas: Based on what did not go well, identify what we should do differently. Please prioritize your list. (Be sure to include which Phase and Area of the Phase you are referring to when you enter your responses.)

1. Individual review period was not long enough for those review team members who had to prepare charts then rework them ad naseum for the DDP

2. Not enough time was provided for a complete review of the technical design

3. Logistics and requirements should be well defined for the Review’s needs

4. There are many good leaders in this group, but one person needs to be empowered with assigning/delegating the tasks needed for successful execution…this person should be assigned my NASA management, and given the authority to act as a supervisor during this process

5. Need to define exactly what should be done/completed prior to tabletop meetings during PDR. Book managers needed to already have worked issues/comments/editorials with reviewers prior to attending PDR tabletop with the sequestered team. They needed to already have worked whether they accepted/rejected their comments before entering. They needed to check the RID tool themselves for their documents to see if any RIDs had been entered against their document. The tabletop meetings were not meant to work issues, unless a sequestered participant had a question/comment about a RID.

6. 1. Present the Integrated (all the Element) RID Story so that it is easy to follow and so that it can be referenced while reviewing VI products

7. Rework the RID tool so that it supports the RID process used (instead of driving the process). Also improve the reports.

8. Have a drawing Layout Room for CDR. There will be more drawiongs and we will need to see them.

9. Obviously the location for the RID review was undersized for the number of reviewers. Network capacity was a problem and should have been anticipated. Room location for some put review teams into areas that were not supportive of the review. Consider a larger location next time with the appropriate network capacity.

10. Since a lot of the PDR documentation was not mature enough to be considered for PDR, limiting the value of a review of these documents, the PDR entry critieria should be reconsidered. Possibly restricting documentation addressed in the PDR to 60% maturity with no TBDs or TBRs.

11. Consider using the large auditorium at the Davidson Center not just for the kickoff presentations but also for the Detailed Design Presentations. Provide good wireless network access in the Davidson Center and breakout rooms for splinter meetings.

12. The conduct of this review begs a question: was it designed to review technical content or just pass a program milestone?

13. PEOPLE RUNNING THE PDR HAD NEVER CONDUCTED A REVIEW OF THIS TYPE (OR ANY OTHER ACTUALY). NO EXPERIENCE WAS EVIDNET. PDR LEFT TO RUN IT’S SELF AS BEST TEAM MEMBERS COULD SET UP.

14. Plan and execute and open PDR of the design, not just the delivered documentation. Make every facet of the design RIDable and accept that you have invited the discipline experts to CRITICALLY review your design.

15. Plan for days or hours of fully training participants or participant leaders in the review processes and tools. One 30 minutes session and a pointer to the locations was essentially a hand-wave at what was really needed.

16. Product readiness needs to be addressed/assessed. In general, this program changes direction so much that we spend all of our time reacting instead of working on a quality product. this review was no different. Groundrules were changed during the review-i.e., how we were to handle editorials. in that case, book managers would contact a reviewer and tell them how they were going to handle their editorial comment and by mid week that was no longer true. changing the review plan during the review is just inexcusable and shows poor planning-and my example was one of the least destructive changes.

17. (Type here to submit an idea.)

18. Organizing the review teams by WBS prevented any team from obtaining a system level overview of what was going on. The result was a completely stovepiped review. The prime purpose of the review was to demonstrate that the preliminary design met requirements. In order to properly demonstate this, a review team should be given the requirements for some subset of the total design and the preliminary design soulution against these requirements. The team should be tasked to examine the adequacy of the design subset through manufacturing, assembly, test, ground operations, launch and flight. This is the only way to completely validate the preliminary design against requirments.

19. The PDR was essentially a bean counting activity driven by RIDs, the RID tool, and RID tool problems. Instead of being the focus of the review, RIDs should only be used as input problem flags to an SE&I/Operations Research activity that determines root causes and identifies and documents the larger real issues. Focus on finding the problems rather than closing RIDS.

20. The answer is obvious — Pass a MIlestone with minimum damage

21. 19. is the answer to 11.

22. Regularly scheduled Data Drops and Basic Metric by the RID Coordination Team available for all who where participating in the review might have cut down on the number of times people where performing the system stalling Data Extracts from the RID Tool

23. A clearly defined, user friendly and consistently implemented Reclama Process to insure that Issues brought up by Initiators where addressed through out the review process. Initiaors informed when their RIDs were dispositioned at all phases of the review process and given the opportunity to Reclama.

24. To accomodate participants that are not familiar with the process at all, I recommend two things to preceed any training: 1. Clear definition of roles (a list of ALL the roles, what does EACH role entail), and 2. A list of the STEPS from pre-RID to RID to closure.