GAO Report: NASA: Better Mechanisms Needed for Sharing Lessons Learned — GAO-02-195

Note: Full report online at online at GAO

Executive Summary

In the early 1990s, the National Aeronautics and Space Administration
(NASA) administrator challenged the agency to complete projects faster,
better, and cheaper. The intent was to reduce costs, become more
efficient, and increase scientific results by conducting more and smaller
missions in less time. Although NASA maintained a high success rate
under the faster, better, and cheaper strategy, a few significant mission
failures also occurred—particularly the loss of the Mars Polar Lander and
Climate Orbiter spacecraft. NASA investigations of these failures, as
well
as its review of other programs, raised concern that lessons from past
experiences were not being applied to current programs and projects.
At the request of the Chairman and Ranking Minority Member,
Subcommittee on Space and Aeronautics, House Committee on Science,
GAO assessed whether NASA has adequate mechanisms in place to ensure
that past lessons learned from mission failures are being applied.
Specifically, GAO (1) identified the policies, procedures, and systems
NASA has in place for lessons learning, (2) assessed how effectively
these
policies, procedures, and systems facilitate lessons learning, and (3)
determined whether further efforts are needed to improve lessons
learning.

Background

NASA’s procedures and guidelines require that program and project
managers review and apply lessons learned from the past throughout a
program’s or project’s life cycle and to document and submit any
significant lessons to the agency’s Lessons Learned Information System
(LLIS) in a timely manner. NASA defines a lesson learned as knowledge or
understanding gained by experience. The experience may be positive, such
as a successful test or mission, or negative, such as a mishap or failure.

A
lesson must be significant in that it has a real or assumed impact on
operations; valid in that it is factually correct; and applicable in that
it
identifies a specific design, process, or decision that reduces or
eliminates
the potential for failures and mishaps, or reinforces a positive
result.

Results in Brief

NASA recognizes the importance of learning from the past to ensure future
mission success and uses several mechanisms to capture and disseminate
lessons learned. The principal source NASA has established for the
agency-wide collection and sharing of lessons is the LLIS, a Web-based
lessons database that managers are required to review on an ongoing
basis. In addition, NASA uses training, program reviews, and periodic
revisions to agency policies and guidelines to communicate lessons.
Several NASA centers and key programs also maintain lessons learned
systems that are geared toward their own staff. Recently, NASA has taken
steps to improve the way it captures and shares information by developing
a business strategy called knowledge management. Knowledge
management can be defined as the way that organizations create, capture,
and reuse knowledge to achieve their objectives. According to NASA
officials, knowledge management has the potential to link agency staff
with the knowledge and resources they need to complete tasks faster,
better, and cheaper. In pursuit of knowledge management, NASA has
developed a strategic plan, established a management team to coordinate
knowledge management activities at NASA’s centers, and initiated several
information technology pilot projects.

Despite the processes and procedures in place to capture and share
lessons learned, there is no assurance that lessons are being applied
toward future missions success. A GAO survey of NASA program and
project managers revealed weaknesses in the collection and sharing of
lessons learned agency-wide. While some lessons learning does take place,
our survey found that lessons are not routinely identified, collected, or
shared by programs and project managers. Respondents reported that they
are unfamiliar with lessons generated by other centers and programs. In
addition, many respondents indicated that they are dissatisfied with
NASA’s lessons learned processes and systems. Managers also identified
challenges or cultural barriers to the sharing of lessons learned, such
as
the lack of time to capture or submit lessons and a perception of
intolerance for mistakes. They further offered suggestions for areas of
improvement, including enhancements to LLIS and implementing
mentoring and “storytelling,” or after-action reviews, as additional
mechanisms for lessons learning.

While NASA’s current knowledge management efforts should lead to some
improvement in the sharing of agency lessons and knowledge, they lack
ingredients that have been shown to be critical to the success of
knowledge management at leading organizations. Cultural resistance to
sharing knowledge and the lack of strong support from agency leaders
often make it difficult to implement an effective lessons learning and
knowledge sharing environment. We found that successful industry and
government organizations have overcome barriers by making a strong
management commitment to knowledge sharing, developing a welldefined
business plan for implementing knowledge management,
providing incentives to encourage knowledge sharing, and building
technology systems to facilitate easier access to information. The
application of these principles could increase opportunities for NASA to
perform its basic mission of exploring space faster, better, and cheaper
more successfully.

Principal Findings

NASA’s Policies and
Procedures for Lessons
Learning

NASA uses various mechanisms to communicate lessons garnered from
past programs and projects. Policies and guidelines, programmatic and
technical reviews, mentoring and training programs, the Academy of
Program and Project Leadership, and LLIS are the mechanisms employed
by NASA for capturing and sharing lessons learned. LLIS is the “official”
agency-wide repository for such lessons. Lessons entered in the LLIS
database are screened for relevance and to ensure that they do not
contain
sensitive or proprietary information. Initial reviews of lessons are
usually
conducted by the centers, with a final review by the Office of Safety and
Mission Assurance. After a lesson is entered into the system, it remains
in
the database indefinitely and is not reviewed for currency or relevance.
Currently, the system contains over 900 lessons on topics ranging from
program management to technical cause of failure.

In response to the Mars Program failures and the recommendations of
agency reviews of program and project execution, NASA has recently
taken action to improve its policies and practices for capturing and
sharing knowledge by developing a business strategy referred to as
knowledge management. Implementation of knowledge management can
lead to increased productivity, collaboration, and innovation in the
workplace. To coordinate and guide its efforts, NASA recently formed a
knowledge management team, which developed a strategic plan that laid
out broad goals and objectives for knowledge management. In addition,
several pilot projects are underway at various NASA centers to enhance
knowledge sharing.

Fundamental Weaknesses
Exist in the Collection and
Sharing of Lessons
Learned

A survey we conducted of all NASA program and project managers
revealed fundamental weaknesses in the collection and sharing of lessons
learned agency-wide. Although NASA’s processes and procedures require
that program and project managers review and apply lessons learned
throughout a program’s or project’s life cycle, our survey found that
managers do not routinely identify, collect, or share lessons.

Respondents
indicated that LLIS, NASA’s primary method for disseminating lessons
learned agency-wide, is not the primary source for lessons learning.
Instead, managers identified program reviews and informal discussions
with colleagues as their principal sources for lessons learned. One
reason
LLIS is not widely used, according to one center official, is because its
lessons cover so many topics that it is difficult to search for an
applicable
lesson. Another respondent indicated that it is difficult to weed through
all
the irrelevant lessons to get to the few “jewels” that you need to find.
Respondents also identified challenges or cultural barriers to the
sharing
of lessons learned as well as areas of improvement. Managers noted that
there is a reluctance to share negative lessons for fear that they might
not
be viewed as good project managers, and there is a lack of time for
lessons
learning to take place. One manager stated, “Until we can adopt a culture
that admits frankly to what really worked and didn’t work, I find many of
these tools to be suspect.” Managers suggested that NASA could improve
lessons learning by implementing mentoring and “storytelling” activities,
and it could enhance LLIS by increasing its search functions, including
more positive lessons, and developing a mechanism to disseminate key
lessons to users.

In discussions with NASA officials, we found there was general agreement
with the results of our survey as well as suggested improvements for
lessons learning. Officials indicated that lessons learning has taken on
greater importance in recent years due to the implementation of more
programs and projects under the faster, better, cheaper strategy and the
continuing loss of agency expertise due to attrition. They acknowledge
that LLIS has not been an effective mechanism for agency-wide sharing of
lessons. Although the system is viewed as providing a useful repository
for
storing lessons, officials agreed with managers’ concerns about the
difficulties involved in searching the system and finding relevant
lessons,
the inconsistent quality of information contained in the system, and the
lack of lessons about positive project experiences. However, while
program and project managers’ suggested improvements would help
increase the usability of LLIS, they have not targeted some of the more
fundamental problems hampering NASA’s ability to share lessons, such as
persistent cultural barriers.

Creating an Environment
for Lessons Learning
through Knowledge
Management

Leading organizations are discovering that actively managing knowledge
creates value by increasing productivity and fostering innovation.
Likewise, NASA’s paramount concern should be about capturing and
sharing organizational knowledge and using it to perform its basic
mission
of exploring space faster, better, and cheaper. Although NASA has
recently
taken action to improve the way in which the agency captures, organizes,
and shares knowledge, these efforts do not fully address the fundamental
weaknesses in lessons learning identified by our survey: namely, cultural
resistance to sharing knowledge and the lack of an effective strategic
framework and management attention for overcoming such resistance.
NASA has made a reasonable start by developing a strategic plan for
knowledge management, but the agency has not made a good business
case for how it will implement and use knowledge management within the
organization. In addition, while successful industry and government
organizations have made a firm commitment to making knowledge
management practices work, NASA has not provided the leadership,
support, and resources needed for effective knowledge management to
take place. Furthermore, knowledge management organizations have
employed incentives, processes, and systems designed to address cultural
barriers to continuous lessons learning and knowledge sharing. For
example, organizations that value knowledge sharing have encouraged
employees to spend time sharing knowledge, helped facilitate
communities of practice based around common interests, and provided
rewards when knowledge has been shared and applied. NASA has not
done so on an agency-wide basis.

Recommendations

NASA needs to strengthen its lessons learning in the context of its
overall
efforts to develop and implement an effective knowledge management
program. Improvement of NASA’s lessons learning processes and systems
can help to ensure that knowledge is gained from past experiences and
applied to future missions.
We recommend that the NASA administrator strengthen the agency’s
lessons learning processes and systems by

Agency Comments
and Our Evaluation

In written comments on a draft of this report, NASA generally concurred
with our recommendations for improving the agency’s lessons learned
processes and systems. NASA stated that it must do a better job of
communicating the various lessons learned sources to employees,
improving mechanisms to link these sources, and ensuring appropriate
training for employees in order to maximize lessons learning.