Space Shuttle: Need to Sustain Launch Risk Assessment Process
Improvements (Chapter Report, 03/26/96, GAO/NSIAD-96-73).
Pursuant to a congressional request, GAO reviewed the National
Aeronautics and Space Administration's (NASA) management of risk
associated with space shuttle flights, focusing on NASA attempts to: (1)
increase the flow and communication of risk information; and (2) use
quantitative methods for assessing risk.
GAO found that: (1) NASA has successfully created numerous formal and
informal communication channels and an open organizational culture that
encourages people to discuss safety concerns and to elevate unaddressed
concerns to higher management levels; (2) while most personnel agreed
that the current culture encourages discussions of safety concerns,
there was not universal agreement about the kinds of risk information
needed for final launch decisions; (3) some personnel expressed concerns
about the effects of pending cost reductions and program changes on
shuttle safety; (4) NASA primarily relies on qualitative methods to
assess and prioritize significant shuttle risk; (5) costs, lack of
expertise, and lack of data have hindered NASA progress in increasing
its use of quantitative methods to assess shuttle safety risks; and (6)
NASA databases do not always provide timely, accessible, accurate, and
complete information to facilitate quantitative assessment or
decisionmaking.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: NSIAD-96-73
TITLE: Space Shuttle: Need to Sustain Launch Risk Assessment
Process Improvements
DATE: 03/26/96
SUBJECT: Space exploration
Budget cuts
Statistical methods
Safety
Information dissemination operations
Systems evaluation
Information gathering operations
Data bases
Aerospace research
IDENTIFIER: NASA Program Compliance Assurance and Status System
NASA Problem Reporting and Corrective Action System
Space Shuttle
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** <info@www.gao.gov> **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Honorable
James A. Hayes, House of Representatives
March 1996
SPACE SHUTTLE - NEED TO SUSTAIN
LAUNCH RISK ASSESSMENT PROCESS
IMPROVEMENTS
GAO/NSIAD-96-73
Space Shuttle
(709066)
Abbreviations
=============================================================== ABBREV
GAO - General Accounting Office
NASA - National Aeronautics and Space Administration
Letter
=============================================================== LETTER
B-260150
March 26, 1996
The Honorable James A. Hayes
House of Representatives
Dear Mr. Hayes:
As requested, we reviewed the National Aeronautics and Space
Administration's (NASA) management of risk associated with flying the
space shuttle. This report addresses NASA's efforts to create an
environment conducive to the free flow of needed risk information and
adopt quantitative methods for assessing risk.
We are sending copies of the report to interested congressional
committees, the Administrator of NASA, and the Director of the Office
of Management and Budget. We will also provide copies to others on
request.
Please contact me at (202) 512-8412 if you or your staff have any
questions concerning this report. Other major contributors are
listed in appendix V.
Sincerely yours,
David R. Warren, Director
Defense Management Issues
EXECUTIVE SUMMARY
============================================================ Chapter 0
PURPOSE
---------------------------------------------------------- Chapter 0:1
The 1986 space shuttle Challenger accident brought into sharp focus
the risks involved in human space flight. The Presidential
Commission that investigated the accident found that it was caused by
a poor design of the joints holding the solid rocket motors together,
but the Commission also cited inadequacies in the National
Aeronautics and Space Administration's (NASA) processes for
identifying, assessing, and managing risk as contributing factors.
The former Chairman, Subcommittee on Investigations and Oversight,
House Committee on Science, Space, and Technology, asked GAO to
review NASA's management of risk associated with flying the shuttle.
Specifically, GAO reviewed the actions NASA has taken to improve the
free flow of information in the launch decision process and the
progress NASA has made in adopting quantitative methods for assessing
risk.
BACKGROUND
---------------------------------------------------------- Chapter 0:2
Space systems are inherently risky because of the technology involved
and the complexity of the activity. For example, thousands of people
perform about 1.2 million separate procedures to process a shuttle
for flight. While the risks cannot be completely eliminated, they
must be identified and managed to the extent possible.
Although the Presidential Commission determined that a faulty solid
rocket motor joint design caused the accident, it identified other
contributing factors. The Commission concluded that there were
serious flaws in the decision-making process leading up to launch.
It reported management isolation and communication failures as
contributing causes to the accident. The Commission cited the
propensity of some NASA managers to attempt to resolve potentially
serious problems internally rather than tell higher management
levels.
Following the Commission's investigation, the National Research
Council reviewed NASA's approach to conducting shuttle risk
assessments. The Council found that NASA was placing too much
reliance on qualitative risk assessments and recommended greater use
of quantitative methods, such as probabilistic risk assessments.\1
GAO assessed the current communications environment, in part, by
observing the launch decision process and by interviewing shuttle
managers, representatives of NASA's safety organization, and managers
and working-level engineers at three shuttle contractors. GAO
initially interviewed officials in groups. The group interviews
enabled participants to exchange their perspectives on communications
within the shuttle program, provided GAO with an understanding of
these complex areas, and produced concrete illustrations. GAO
followed up the interviews of NASA officials with a structured survey
to more precisely measure views of communications issues that emerged
from the group discussions.
--------------------
\1 Probabilistic risk assessment is a method of systematically
examining complex technical systems to measure both the likelihood
that an accident will occur (probability) and the level of damage or
loss that will result (consequences).
RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3
NASA has been successful in creating an environment conducive to the
free flow of needed risk information. NASA managers and safety
representatives responsible for shuttle operations reported that they
believe conditions governing the flow of information and decision
processes are appropriate. However, some viewed the management
information systems as needing improvement. GAO's own analysis of
communication flows in NASA's assessment of a recently identified
shuttle hardware problem illustrated significant improvements in
communication of risk information.
Some discussion group participants expressed concern about impending
budget reductions and the transition of shuttle operations to a prime
contractor. The challenge for NASA will be to maintain the
principles of effective communications it has in place now as it
continues to reduce shuttle funding and transfers management of
shuttle operations to a single contractor.
NASA still relies primarily on qualitative risk assessments, but has
made limited progress in adopting quantitative methods, such as
probabilistic risk assessments, for assessing significant shuttle
risks. However, there is not a consensus among shuttle managers and
safety representatives on increasing the use of these methods. NASA
lacks an overall strategy on when to use such methods to supplement
engineering judgments. Officials cited limited resources and a lack
of personnel with expertise in these methods as barriers to their
implementation. Officials told GAO that another reason quantitative
methods are not used more routinely is that needed data is not always
available in a readily usable form.
PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4
NASA HAS IMPROVED THE FLOW
OF INFORMATION IN THE
SHUTTLE PROGRAM
-------------------------------------------------------- Chapter 0:4.1
GAO asked NASA and contractor managers, safety representatives, and
contractor engineers to assess conditions related to the flow and
quality of information to management. Based on GAO's interviews and
survey responses, the program's organizational culture encourages
people to discuss safety concerns and to elevate concerns to higher
management if they believe the issues were not adequately addressed
at lower levels.
A variety of communication forums help ensure that NASA and
contractor managers and the safety community are continually apprised
of safety problems and issues that arise during shuttle processing.
These forums include the certification of flight readiness, daily
telephone conferences, and weekly meetings.
NASA managers at the three field centers with primary responsibility
for the shuttle program and at headquarters reported having taken
steps to create an organizational environment that encourages
personnel at all levels to voice their views on safety issues to
management. For example, managers encourage debate at readiness
reviews and other meetings and invite individuals to meet with them
at other times about safety concerns.
Although the current program culture encourages open discussion of
safety issues, there was not complete agreement on the kind or level
of detail of information to be discussed at the flight readiness
review. NASA managers widely endorsed 7 of the 15 types of safety
issues GAO asked about as needing to be discussed in detail; however,
opinions were divided in other areas such as whether or not
information about hazards and waivers should always be briefed.
GAO compared how NASA addressed a problem in a solid rocket motor
joint that occurred in 1995 with its handling of the joint problem
that caused the 1986 accident. Based on this comparison and
observations, NASA was much more open in dealing with the more recent
problem. For example, shuttle program managers were kept informed
and involved in resolving the problem and NASA held weekly press
meetings to discuss its progress.
FUNDING REDUCTIONS,
DOWNSIZING, AND
RESTRUCTURING
-------------------------------------------------------- Chapter 0:4.2
Although NASA has made substantial improvement in the flow of
communications, some managers expressed concern about the impact of
funding reductions particularly with respect to staffing and
organizational restructuring. NASA must further reduce shuttle
operation costs to meet expected declining budgets. Because of this,
NASA plans significant changes in the way the shuttle program is
managed.
Future funding reductions, downsizing, and program restructuring will
be challenging because the program also must maintain the capability
to meet the demanding international space station launch schedule.
NASA must reduce shuttle budgets by an additional $2.5 billion in
fiscal years 1996 through 2000, while implementing a very compressed
launch schedule.
To help meet the cost reduction and schedule challenges, NASA is
planning to turn shuttle operations over to a single prime
contractor. The agency will reduce its involvement in day-to-day
operations but will retain responsibility for launch decisions.
Although not enough information is available about the plan to assess
all of its implications, some of the shuttle program managers and
safety representatives GAO interviewed expressed concern about
continued funding reductions and the transition from the current way
of doing business to the future management of the program.
GAO's work shows that NASA follows certain management principles in
its communications processes. These principles include priorities
that place safety above cost or schedule; an environment that
encourages timely, open debate; a culture that encourages people to
elevate their safety concerns; NASA and contractor working
relationships that ensure agency managers obtain continual knowledge
of problems and issues; and an organizational relationship that
enables managers to carry out their responsibility to certify
readiness for flight. Survey respondents generally agreed that these
principles should be followed in the future.
NASA STILL RELIES PRIMARILY
ON QUALITATIVE RISK
ASSESSMENTS
-------------------------------------------------------- Chapter 0:4.3
NASA still relies primarily on its engineers' judgment to assess and
prioritize significant shuttle program risks. It has made some use
of quantitative methods, such as probabilistic risk assessments, but
has no overall strategy on when these methods should be used in
shuttle decision-making. Past quantitative risk assessments have
included proof-of-concept studies, assessments of specific shuttle
systems, and assessments of accident probabilities for launches
involving radioactive material. NASA had a contractor develop a
probabilistic risk assessment model for use in the shuttle program
but has not developed a plan for incorporating this tool into its
shuttle program management.
The contractor that developed the risk assessment model cited
potentially beneficial uses, such as establishing cost objectives for
redesigning the highest risk components. However, some shuttle
managers told us that NASA lacks an overall strategy and specific
employee skills to efficiently and effectively utilize methods such
as probabilistic risk assessments. Some officials stated there is a
lack of trust in probabilistic risk assessments because people do not
understand the methodology. Therefore, acceptance of this risk
assessment method as a supplement to existing qualitative methods is
not NASA-wide, and there is much skepticism about the cost and
benefits of using probabilistic risk assessments.
DATA NOT ALWAYS AVAILABLE IN
READILY USABLE FORM
-------------------------------------------------------- Chapter 0:4.4
NASA has developed two automated database systems to provide shuttle
data for use in decision-making--the Program Compliance Assurance and
Status System and the Problem Reporting and Corrective Action System.
However, some officials told GAO that information from these
databases is not always timely or reliable, and the systems are
cumbersome to use.
The Program Compliance Assurance and Status System is based on older
technology, trend and other data are not centralized in the system,
and software needed to convert contractor data to NASA database
format has not been developed. A January 1995 internal study found
important information missing from thousands of entries. Officials
also said that the Problem Reporting and Corrective Action system
records are often not reliable and lack uniformity in categorizing
problems.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5
GAO recommends that the Administrator of NASA
identify guiding principles of good risk management, such as those
described in this report, and ensure that terms and conditions
of the planned shuttle operations contract reflect these
principles;
take steps to ensure that flight readiness review participants
understand and agree on the minimum issues that should always be
discussed at the review and the level of detail that should be
provided;
establish a strategy, including specific milestones, for deciding
whether and how quantitative methods might be used as a
supplemental tool to assess shuttle risk; and
assess the shuttle program's centralized database, as well as other
databases, to insure that data required to conduct risk
assessments and inform decisionmakers, is accessible, timely,
accurate, and complete.
AGENCY COMMENTS
---------------------------------------------------------- Chapter 0:6
In commenting on a draft of this report, NASA concurred with GAO's
four recommendations and stated that the agency is already taking
action to implement them. GAO made additional changes to the report,
where appropriate, based on NASA's technical comments. NASA's
comments are in appendix IV.
INTRODUCTION
============================================================ Chapter 1
The first space shuttle launch occurred on April 12, 1981. During
the 25th launch on January 28, 1986, the shuttle Challenger was
destroyed shortly after liftoff from Kennedy Space Center. Shuttle
flights were suspended while the accident was investigated by the
Presidential Commission. The shuttle returned to flight on September
29, 1988. Since that time, it has flown successfully about 50 times.
The Presidential Commission determined that the 1986 accident was
caused by a faulty seal in one of the solid rocket motor joints. The
Commission also found other contributing causes to the accident, such
as management isolation, communications failures, and lack of a
properly staffed, supported, and robust safety organization.
According to the Commission's June 6, 1986, report, the decision to
launch the Challenger was based on incomplete and sometimes
misleading information, a conflict between engineering data and
management judgments, and a the National Aeronautics and Space
Administration (NASA) management structure that permitted internal
flight safety problems to bypass key shuttle managers.\1 Officials
who made the launch decision were unaware of a recent history of
problems with the defective solid rocket motor joint and of the motor
contractor's initial recommendation against launching. According to
the Commission, if the decisionmakers had known all of the facts, it
is highly unlikely that they would have decided to launch.
--------------------
\1 Report of the Presidential Commission on the Space Shuttle
Challenger Accident, June 6, 1986.
RISK OF SPACE FLIGHT
---------------------------------------------------------- Chapter 1:1
Space flight can never be made risk free because it involves complex
hardware and software systems, harsh operating environments, and the
possibility of human error. A 1995 study by a NASA contractor, for
example, placed the median estimate of a catastrophic shuttle failure
at 1 in 145 launches.
According to the advisory committee on the Future of the U. S.
Space Program, "there can be no acceptable objective among those who
would challenge the vastness of space other than perfection."\2
Unfortunately, as the Committee's report points out, the objective of
perfection is not readily met, especially since space missions are
fundamentally difficult and demand undertakings that depend upon some
of the world's most advanced technology and there are many
opportunities for error.
The shuttle is an extremely complex system. The program employs
thousands of people and launching a shuttle requires that 1.2 million
separate procedures be accomplished correctly. Also, NASA has
identified over 5,000 critical system components whose failure,
either singularly or in combination, could cause loss of the vehicle
or crew. Because these risks cannot be completely eliminated, they
must be identified and properly managed.
--------------------
\2 Report of the Advisory Committee on the Future of the U.S. Space
Program, December 1990.
NASA'S RISK MANAGEMENT PROGRAM
---------------------------------------------------------- Chapter 1:2
NASA's risk management policy requires that program and project
management communicate to NASA management and all program/project
personnel the significance of assessed risks and the decisions made
with respect to them. At NASA, risk management includes identifying
the primary risk drivers and estimating the likelihood of occurrence,
identifying the ensuing consequences, and determining the cost and
schedule impact.
NASA policy regarding safety is to
avoid loss of life, injury of personnel, damage, and property loss;
instill safety awareness in all NASA employees and contractors;
assure that an organized and systematic approach is utilized to
identify safety hazards and that safety is fully considered from
conception to completion of all agency activities; and
review and evaluate contractors' and NASA's plans, systems, and
activities related to establishing and meeting safety
requirements to ensure that desired objectives are effectively
achieved.
Failure modes and effects analyses\3 are conducted for all flight
hardware elements and ground support equipment. This analysis starts
with the identification of all potential failure modes and evaluation
of "worst case" effects. NASA places potential effects of failures
into the general categories shown in table 1.1.
Table 1.1
Potential Effects of Failures of Shuttle
Hardware Components
Criticality Potential effect of failure
------------- -------------------------------------------------------
1 Single failure that could result in loss of life or
vehicle.
1R Redundant item(s), all of which failed, could cause
loss of life or vehicle.
2 Single failure that could result in loss of mission.
2R Redundant item(s), all of which failed, could cause
loss of mission.
3 All others.
----------------------------------------------------------------------
Hazard analyses\4 are conducted to identify potential safety hazards
and means for minimizing the hazards. NASA's actions to minimize
hazards follow the sequence of (1) system designs that minimize
potential hazards, (2) use of safety devices if the design does not
eliminate a potential safety hazard, (3) use of warning devices to
alert the flight or ground crew to potential hazards, and (4) use of
special procedures.
--------------------
\3 The failure mode and effects analysis is a systematic evaluation
of each component of the shuttle system to identify hardware items
that are critical to the performance and safety of the vehicle and
mission. The evaluation includes identifying all system components,
determining the potential modes of failure for each component, and
recommending corrective action. A critical items list is developed
as a result of the failure modes and effects analysis. The list
includes all system components that could cause loss of life,
vehicle, or mission.
\4 Hazard analysis is to determine potential sources of danger that
could develop while operating and maintaining the system hardware and
software. Hazard analysis also identifies the presence of other
potential risks caused by the environment, crew-machine interfaces,
and mission activities.
RISK ASSESSMENT APPROACHES
-------------------------------------------------------- Chapter 1:2.1
Approaches for assessing risk can be either quantitative or
qualitative, depending on whether statistical probabilities are
assigned to a risk element. All risk assessment approaches require
experts to make subjective judgments about the risk elements as well
as the likelihood of their occurrence.
Quantitative approaches, such as probabilistic risk assessments, can
be used to assess both the likelihood that an accident will occur
(probability) and the level of damage or loss that will result
(consequences). Quantitative assessment methods mathematically
quantify risk on the basis of engineering judgment, calculated
probabilities of component reliability, analysis of potential human
failures, and whether they occur singly or in combination. A
probabilistic risk assessment, for example, addresses three basic
questions: (1) What could go wrong? (2) How likely is it that this
will happen? and (3) What are the consequences?
Qualitative assessments, on the other hand, assess risk through
descriptive information, identifying the nature and components of
risk or an ordinal scale, such as high, medium, and low. Qualitative
ratings are usually based on the judgments of experts after they
consider such things as test and operational experience, analytical
results, trends, and other reported data.
CERTIFYING THE SHUTTLE FOR
FLIGHT
---------------------------------------------------------- Chapter 1:3
NASA follows a formal review process in certifying the shuttle for
flight. The certification of flight readiness process is a
step-by-step activity designed to certify the readiness of all
components of the vehicle assembly and all aspects of mission
support.
The flight preparation process begins with project milestone reviews
including (1) element acceptance, (2) payload readiness, (3) software
readiness, and (4) project preflight readiness reviews. These
reviews are chaired by NASA project managers and the contractors
formally certify the flight readiness of the hardware and software.
The next step in the process is the program milestone reviews. These
reviews are held to assess the readiness for mating the external tank
and solid rocket booster, orbiter and external tank, and ferrying the
orbiter atop the shuttle carrier aircraft when required. These
reviews are chaired by the manager of launch integration and each
shuttle element manager certifies that it has satisfactorily
completed the manufacture, assembly, test, and checkout of the
elements, including the contractor's certification that design and
performance are up to standard.
The final step in the flight preparation process is the flight
readiness review. This review is held about 2 weeks prior to launch
and is chaired by the Associate Administrator for Space
Flight.<su5,6>
All shuttle elements, safety and mission assurance, center directors,
and senior representatives from the major contractors participate in
this review. At the end of the flight readiness review, all
organizations must certify that the mission is ready for launch. The
Associate Administrator for Safety and Mission Assurance\7
is also an active participant.
The safety and mission assurance organization holds parallel reviews
to assess safety issues related to the planned launch. The safety
and mission assurance organization participates in all phases of the
flight preparation process.
Two days before a scheduled launch, a mission management team holds a
review to assess flight readiness. Its agenda includes close out of
any open work, close out of any flight readiness review action items,
discussion of new or continuing anomalies,\8 and an updated briefing
on anticipated weather conditions at the launch site and at abort
landing sites in different parts of the world. The mission
management team meets every day after the launch -2 day review up to
the conclusion of the mission. Figure 1.1 illustrates NASA's flight
preparation process.
Figure 1.1: Flight Preparation
Process
(See figure in printed
edition.)
NASA's safety organization provides an independent channel for
assessing shuttle flight safety. Each center's safety organization
participates in the element acceptance reviews as well as the flight
readiness review and the mission management team. Participation in
these reviews provides the opportunity for NASA's safety organization
to express any residual concerns about the safety of an upcoming
mission. The organization also holds independent prelaunch
assessment reviews. In addition, the Associate Administrator for
Safety and Mission Assurance attends the flight readiness review and
has a direct communications link to the NASA Administrator.
Other program briefings and reviews are also a part of the
certification of flight readiness process. For example, the program
manager holds an early morning telephone conference with the shuttle
centers and headquarters each day to discuss the status of progress
and problems. Likewise, about midday the working level shuttle
managers hold a telephone conference to provide updated information.
Safety and mission assurance personnel attend all of the shuttle
program and project meetings and contribute their independent views.
--------------------
\5 The Associate Administrator for space Flight is responsible for
providing leadership and programmatic direction to accomplish the
NASA human space flight program, including space shuttle, space
station, spacelab, cooperative U.S./Russian human space flight
programs, and other related space flight activities.
\6 As a result of organizational changes announced subsequent to the
completion of our review, the Director of the Johnson Space Center
will chair future flight readiness reviews.
\7 The Associate Administrator for Safety and Mission Assurance is
responsible for providing leadership, policy direction, functional
management, and coordination for the safety, reliability,
maintainability, and quality assurance for all NASA programs.
\8 Anomalies are unexpected events; hardware or software damage; a
departure from established procedures or performance; or a deviation
of system, subsystem, and/or hardware or software performance outside
certified design/performance specification limits.
OBJECTIVES, SCOPE, AND
METHODOLOGY
---------------------------------------------------------- Chapter 1:4
The former Chairman, Subcommittee on Investigations and Oversight,
House Committee on Science, Space, and Technology, asked us to review
NASA's management of risk associated with flying the space shuttle.
Specifically, we reviewed the actions NASA has taken to improve the
free flow of information in the launch decision process and the
progress NASA has made in adopting quantitative methods for assessing
risk.
To assess the communications environment, we reviewed policies,
procedures, and practices related to management of the shuttle
program used by the agency in making launch decisions; we observed
various shuttle processing reviews, including a shuttle launch; and
discussed various aspects of the program with those responsible for
its management.
We also conducted discussions of these topics with groups of shuttle
and safety managers at NASA Headquarters, and the Johnson, Marshall,
and Kennedy field centers. Together these individuals represented
almost all of the top NASA officials responsible for shuttle launch
decisions and management of most shuttle manufacturing and processing
work. To understand the flow of risk information within shuttle
contractor organizations and between NASA and its shuttle
contractors, we also held discussions with groups of program and
safety managers and working-level engineers at three of NASA's prime
shuttle contractors. We chose the three contractors because the work
is among the more complex and highest risk in the program.
Group discussions are very useful for exploring the various facets of
communications issues and processes. However, they did not enable us
to determine how many participants held a particular view or the
intensity of their views. Therefore, to more precisely measure the
themes that emerged from the group discussions, we sent a structured
questionnaire to the NASA interview participants and some safety
representatives who did not participate in the group interviews.
Appendixes I through III contain a more detailed discussion of our
group interview and survey methodology.
To evaluate NASA's use of quantitative risk assessment methodologies,
we reviewed policies, procedures, and practices related to NASA's
shuttle risk management program and held discussions with senior
shuttle managers and NASA's safety and mission assurance
organization. We also discussed the use of quantitative risk
assessment methodologies with other federal agencies that are
responsible for managing complex systems to establish a benchmark for
the use of such methods within the federal government. This work
included the Nuclear Regulatory Commission and the Federal Aviation
Administration. We also obtained information on the Environmental
Protection Agency's use of quantitative risk assessment in the
management of superfund cleanup sites. In addition, we consulted
outside experts to obtain their views on the usefulness of
quantitative risk assessments to NASA.
We conducted our review primarily at NASA Headquarters, Washington,
D.C.; Marshall Space Flight Center, Alabama; Johnson Space Center,
Texas; Kennedy Space Center, Florida; Thiokol Corporation, Ogden,
Utah; and Rocketdyne Division of Rockwell International, Canoga Park,
California.
We conducted our review between June 1994 and December 1995 in
accordance with generally accepted government auditing standards.
NASA HAS IMPROVED THE SHUTTLE
COMMUNICATIONS ENVIRONMENT AND
NEEDS TO SUSTAIN IMPROVEMENT IN
THE NEW MANAGEMENT ENVIRONMENT
============================================================ Chapter 2
Good communications is one of the keys to effective risk management.
Without adequate information about risks, launch decisions may be
flawed as they were in the case of the Challenger accident.
Interviews with key shuttle program officials, survey data, and our
observations indicate that NASA has been successful in creating
communication channels and an organizational culture that encourages
people to discuss safety concerns and to bring those concerns to
higher management if necessary.
NASA has announced plans to make fundamental changes in the way it
manages the shuttle program--turning day-to-day management over to a
single prime contractor and reducing direct NASA involvement. Some
managers expressed concern about the potential impact of this change,
particularly with respect to staffing and organizational
restructuring. NASA's challenge will be to ensure adherence to the
communications principles that are essential to promoting shuttle
safety.
NASA HAS MADE CHANGES TO
STRENGTHEN SHUTTLE RISK
MANAGEMENT
---------------------------------------------------------- Chapter 2:1
According to the Presidential Commission, prior to the Challenger
accident, project managers for the various elements of the shuttle
program felt more accountable to their center management than to the
shuttle program organization. As a result, vital program information
frequently bypassed the program manager, who was located at the
Johnson Space Center. The Commission recommended that NASA give the
program manager authority over all program funding and work. In
response, NASA centralized program management in a shuttle program
director at headquarters with overall responsibility for shuttle
operations and budgets. Also, the program manager at the Johnson
Space Center was made a headquarters employee in order to minimize
center-to-center communications problems. Effective January 31,
1996, however, shuttle program management responsibility was
transferred from the headquarters director to the Johnson Space
Center director. Because NASA has not yet prepared a detailed plan
for implementing this change, we could not fully evaluate its
implications. However, according to NASA officials in the Office of
Human Space Flight, the Johnson Center director will have full
authority over the shuttle resources and work at all participating
centers and will report directly to the NASA administrator. NASA has
also given astronauts a role in certifying the shuttle for launch and
encouraged them to move into shuttle management positions, as
recommended by the Presidential Commission.
NASA also established the Headquarters Office of Safety and Mission
Assurance\1 under the direction of an associate administrator
reporting directly to the NASA administrator. The agency
strengthened the safety organizations at its shuttle field centers so
that each director of safety and mission assurance reports to a
center director rather than the engineering organization. NASA also
increased the number of people assigned to the safety organization.
In addition, NASA established a safety reporting system to provide an
avenue for NASA and contractor personnel to confidentially report
problems to safety and program management officials that could result
in loss of life or mission capability, injury, or property damage.
--------------------
\1 When established, this organization was the Office of Safety,
Reliability, Maintainability, and Quality Assurance.
NASA HAS IMPROVED THE
COMMUNICATIONS ENVIRONMENT
---------------------------------------------------------- Chapter 2:2
Participants in our discussion groups--both within NASA and in the
contractor organizations--described a communication environment that
is more open than the one that existed at the time of the accident.
Respondents in our follow-up survey portrayed the culture as
encouraging contractors and employees to discuss and, if necessary,
elevate safety concerns. Discussion groups also identified multiple
channels, both formal and informal, for communicating flight safety
information. In some cases, these communication channels represent
independent, parallel paths for assessing risk. Our own observations
and analysis of NASA's approach to dealing with a recent problem
illustrated the openness with which agency officials address safety
issues.
CURRENT CULTURE ENCOURAGES
OPEN DISCUSSION OF SAFETY
ISSUES
-------------------------------------------------------- Chapter 2:2.1
In group discussions with key NASA and contractor shuttle managers
and contractor working-level engineers, we asked them to assess
conditions related to the flow of safety information to top
management. All of the groups reported that the shuttle program's
organizational culture encourages people to discuss safety concerns
and bring concerns to higher management if they believe the issues
were not adequately addressed at lower levels. As one manager noted,
because of the complexity of the shuttle program, open communication,
group discussions, and the sharing of information are essential to
flight and work place safety.
NASA managers at the three field centers with primary responsibility
for managing shuttle elements and at NASA headquarters reported
having taken steps to create an organizational environment that
encourages personnel at all levels to voice their views on safety to
management. One manager noted that people are not afraid to surface
their mistakes to management when they discover mistakes have
occurred. Another manager said, "If . . . I got the idea that I
had a manager in the system who wasn't allowing their people to feel
comfortable in bringing [up] things, probably that's the time I think
I would change that person's job because . . . our people need to
feel that they can come without attribution and talk about what they
need to talk about."
Managers in each group we interviewed cited various techniques they
use to create an organizational environment that encourages personnel
at all levels to voice their professional viewpoints on safety issues
to management, even if dissenting. For example, managers invite
people to express their concerns by
trying to keep every line of communication open and telling people
that bringing up a problem does not reflect poor performance;
holding extensive dialogue over shuttle safety issues, beginning
early in the problem identification stage, so that everyone
fully understands the issues;
encouraging people to come in or call their managers if they want
to talk about a safety concern, no matter how small the issue;
and
not only encouraging, but expecting, open expression of
professional differences at all levels.
The contractor managers also described a working relationship with
NASA that they believe encourages open communication and the
elevation of safety concerns. They described the flow of information
between NASA and shuttle contractors as continual, open, and
comprehensive. From their perspective, daily contact between
contractor and NASA working-level personnel contributes to the
exchange of information. Contractor support to and participation in
flight readiness reviews and other shuttle processing meetings, and
their reporting of safety information directly into NASA's
centralized information systems are among the other mechanisms that
achieve that exchange.
One manager noted that the Challenger accident prompted a change in
his contractor's management approach. Before the accident, company
meetings were closed to the NASA site representatives. Since the
accident, NASA representatives attend all technical meetings.
Managers from two other contractors said that they would not hesitate
to go to the highest levels of NASA management to ensure that safety
issues received appropriate attention.
Contractor working-level engineers portrayed their organizations as
supportive of engineers elevating shuttle safety issues and concerns
to management. For example, at one contractor facility, program
teams are structured so that minority opinions about the handling of
safety problems can be elevated to a higher level board. At another
contractor facility, the work environment was described as one that
encourages debate, discussion, and never keeping a safety concern
quiet. At the third contractor plant, the formal reporting process
ensures that NASA and contractor managers are continually apprised of
issues, review how issues are resolved, and can request more work if
they do not agree with the resolution of a safety issue.
The managers and safety representatives who responded to our survey
also gave very favorable ratings to NASA's current communications
culture. For example, 90 percent of those responding to the survey
said that to a great or very great extent NASA's organizational
culture encourages civil service employees to discuss safety concerns
with management.
As shown in figure 2.1, more than 80 percent of the respondents to
our survey rated the following current shuttle communications and
information flow conditions very favorably.
Figure 2.1: Characteristics of
the Current Shuttle
Communications Environment
(See figure in printed
edition.)
Note: The chart presents the percentage of 39 respondents rating
each characteristic as present to a great or very great extent.
As part of our review, we attended numerous certification of flight
readiness and prelaunch assessment reviews for shuttle mission
STS-64, including the flight readiness review and launch. We
observed open and candid discussions, debate of issues, and a
structure that required the recording and follow-up of unresolved
issues. At most reviews, presentations appeared thorough and
participants asked many probing questions to ensure they had an
adequate understanding of the issues being briefed. If participants
did not believe they adequately
understood an issue or additional work was required to resolve an
issue, it was listed as an open item to be resolved prior to launch.
NUMEROUS COMMUNICATIONS
PATHS ARE AVAILABLE
-------------------------------------------------------- Chapter 2:2.2
Managers, safety personnel, and working-level engineers described
shuttle program and contractor procedures and structures that provide
multiple avenues for continual communication with contractors, across
centers, and with headquarters to discuss safety issues. These
avenues include the certification of flight readiness process, daily
telephone conferences, and weekly meetings. In response to our
survey, almost all NASA program managers and safety representatives
believe the opportunities to discuss and communicate shuttle issues
and concerns meet, or even exceed, the needs of the program in terms
of the number of forums held and the types and levels of expertise
represented.
The certification of flight readiness process requires the
involvement of all centers and projects on issues that could affect
safety or mission success. In preparation for a launch, NASA relies
on a number of reviews to ensure that the shuttle is safe for flight.
These reviews are designed to ensure compliance with requirements,
that prior problems/failures have been corrected, planned work has
been completed, and operational support is in place for the mission.
Managers also reported other, sometimes less formal, channels for
communicating safety information. For example, the shuttle program
manager holds an early morning telephone conference daily, enabling
NASA managers at headquarters and the centers to discuss problems and
draw upon the experience of others. The manager of launch
integration also conduct a daily "noon board" telephone conference to
discuss shuttle issues, status, and required changes related to
vehicle processing at the Kennedy Space Center. Project
representatives from the various shuttle centers participate if the
issue involves their shuttle element. Also, NASA's shuttle program
manager chairs a weekly Program Requirements Control Board meeting
that is the controlling authority for all changes to the shuttle
program baseline. Safety and mission assurance engineers participate
in all of these meetings. Further, NASA safety and project
representatives at contractor plants help ensure a continual flow of
information on contractor issues. In addition, the NASA Safety
Reporting System (an anonymous reporting system) provides another
opportunity for people to report safety concerns.
In addition to taking part in all of the program and project reviews
for the certification of flight readiness, NASA's Office of Safety
and Mission Assurance conducts prelaunch assessment reviews of all
major shuttle elements. The office's System Safety Review Panel also
conducts several reviews, including a review of in-flight anomalies
from previous missions. These safety office reviews are conducted
independently of the project offices responsible for the various
shuttle elements. Results of the safety office reviews are presented
at the flight readiness review. The safety organization continues to
monitor shuttle missions up to and during launch. Figure 2.2
illustrates the parallel assessments by safety and mission assurance
and the shuttle program and project offices.
Figure 2.2: Safety and Mission
Assurance Parallel Assessments
(See figure in printed
edition.)
We asked contractor working-level engineers what avenues are open to
them to communicate their views in the event that they disagree with
a safety decision made at higher levels of management, either within
their organization or within NASA. A variety of communication routes
were cited: a company ombudsman, the firm's safety manager, NASA
counterparts, or higher levels of management within the contractor's
organization and the NASA Safety Reporting System.
NOT COMPLETE AGREEMENT ON
TYPE AND AMOUNT OF
INFORMATION NEEDED
-------------------------------------------------------- Chapter 2:2.3
While there was a high level of agreement that the current culture
encourages and enables contractors and employees to discuss safety
issues and concerns, there was not universal agreement about the
kinds of risk information needed for final launch decisions. We
asked NASA managers and safety representatives to designate the types
of safety issues that should always be briefed in detail to
corporate-level management at the final flight readiness review.
Seven of the 15 types of issues we asked about were widely endorsed
as needing the board's review; however, opinions were divided in
other areas. For example, the views of the board members tended to
differ from those of the other managers and safety representatives
regarding whether hazards and new waivers should always be briefed in
detail. Opinions were also divided about the level of detail that
should be provided when there are changes that affect procedures or
processes involving the flight crew, operations, software, or shuttle
hardware.
We also observed differences in the amount of detail provided during
two flight readiness reviews. At the first review, we observed that
the review board's chairman required less detail about issues and
concerns than at the second review. The second review meeting we
observed was chaired by a different official. This official
requested a greater level of detail about issues being discussed.
Thus, the change in personnel caused some initial confusion about the
type and amount of information needed to make corporate-level launch
decisions.
NASA DEMONSTRATED OPENNESS
IN DEALING WITH RECENT MOTOR
JOINT ISSUE
-------------------------------------------------------- Chapter 2:2.4
To provide a better understanding of the cultural and communication
path changes within NASA, we compared NASA's approach to handling the
motor joint issue at the time of Challenger with a recent issue
concerning another joint in the solid rocket motor. On two
successive flights in 1995, hot gas penetrated beyond the joint's
sealer compound and made very small singe marks on the joint's
primary o-ring. NASA was more cautious in its approach to handling
the latest motor joint problem. For example, NASA immediately halted
shuttle launches and publicly aired the problem. NASA held weekly
press meetings to discuss the problem and progress in correcting it.
Shuttle and contractor managers at all organizational levels were
heavily involved in the issue and the safety organization provided an
independent assessment of the problem. NASA did not resume shuttle
launches until it was confident that the problem was understood and
corrected. Table 2.1 describes our observations.
Table 2.1
Differences in NASA's Approach to the
Challenger Motor Joint Problem and the
Recent Motor Joint Problem
Challenger problem Recent problem
---------------------------------- ----------------------------------
Nature of the problem
----------------------------------------------------------------------
Design flaw. NASA did not make Process problem. Process
timely attempt to develop and enhancements were initiated as
verify a new seal after the soon as gas paths were detected.
initial design was shown to be Improvements are continuing.
defective.
Hot gas penetrated past primary o- Hot gas made very small singe
ring to secondary o-ring on STS marks on primary o-ring on two
51-C (Jan. 24, 1985) prior to successive flights.
Challenger.
Information flow
----------------------------------------------------------------------
All program managers were not All program managers were heavily
informed of problem prior to the involved in the issue.
launch of Challenger.
NASA solid rocket motor manager All levels were aware of and
waived constraint to launch for understood the problem. NASA
six consecutive launches prior to stopped shuttle launches until the
Challenger. He was required to anomaly was resolved and repairs
notify higher levels but did not. made.
Differing views at lower levels of Relevant information was raised to
management were not raised to the appropriate levels of contractor
appropriate levels. and NASA management. NASA and the
hardware contractor reached a
consensus on cause and corrective
actions.
Top decisionmakers were not aware Top decisionmakers were fully
of all the facts; so, flight was informed, understood the facts,
allowed to proceed. and stopped shuttle flights.
NASA culture
----------------------------------------------------------------------
Culture not conducive to airing Organizations and management
problems. encourage people to elevate
problems and concerns.
Inadequate public airing of Held weekly press meetings to
problems. discuss problems and progress.
Safety organization oversight
----------------------------------------------------------------------
Safety and Mission Assurance Safety and mission assurance
reporting channels varied among organizations are independent of
centers. engineering and project management
throughout NASA.
No formal process to facilitate NASA Safety Reporting System
confidential reporting of safety allows confidential reporting of
concerns. safety concerns.
Little or no trend analysis was Engineering organizations compile
performed on motor joint and hot and, along with safety and mission
gas blow-by problems. assurance, evaluate trend data.
Failure mode and effects analysis/ Indepth failure mode and effects
critical items list and hazard analysis/critical items list and
analyses were minimal. hazard analyses were used during
the investigation and repair
planning.
Postflight inspections included Indepth case and nozzle postflight
case and case to nozzle joints and inspection performed, including
seals, but only limited nozzle all joints. Formal and timely
inspection. Nozzle joints were not reporting of all identified
inspected. Delayed reporting of discrepancies.
inspections results.
----------------------------------------------------------------------
SOME MANAGERS CONCERNED ABOUT
FUTURE CHANGES
---------------------------------------------------------- Chapter 2:3
Some discussion group participants told us they are concerned about
the impacts of continued cost reductions and planned program changes.
Over the next 5 years, plans call for NASA to make significant
additional reductions in shuttle costs while maintaining the
capability to meet the demanding schedule for international space
station assembly and support. Although final decisions have not been
made, NASA has initiated a number of actions to further reduce
shuttle operation costs, including turning shuttle operations over to
a single prime contractor. Some participants in our discussion
groups expressed concern about the effect of continued cost
reductions and the transition to contractor management of the
program.
In July 1995, we reported on the schedule pressures created by the
International Space Station assembly requirements.\2 Based on our own
analysis and internal NASA studies, we concluded that the shuttle's
ability to meet station launch requirements appeared questionable.\3
To meet the station's "assembly complete" milestone, shuttle
officials had designed a very compressed launch schedule. During
certain periods of the station assembly, clusters of shuttle flights
are scheduled to be launched within very short time frames. For
example, the schedule calls for five launches within a 6-month period
in fiscal year 2000 and seven launches during a 9-month period in
fiscal year 2002. Because the schedule is so compressed at times,
there is very little margin for error. There is little flexibility
in the schedule to meet major contingencies, such as late delivery of
station hardware, or technical problems with the orbiter.
We reported in June 1995\4 that NASA had reduced shuttle operations
funding requirements by a cumulative amount of $2.9 billion between
fiscal years 1992 and 1995 when the fiscal year 1992 budget request
is compared to the fiscal year 1995 request. In our survey, we asked
NASA managers and safety representatives what actions had been taken
to accommodate the funding reductions and whether these actions, in
their opinion, had enhanced, degraded, or had little or no effect on
the accuracy, completeness, and timeliness of shuttle safety-related
information. Generally, their assessment was that the actions either
had little or no effect on quality or somewhat degraded quality. For
example, of nine respondents who reported funding reductions
accomplished by delaying safety improvements, six said the delay
somewhat degraded the quality of safety-related information.
However, some respondents reported actions taken to cut costs
actually enhanced the quality of information.
Just over 75 percent of NASA managers and safety representatives we
surveyed believed that NASA emphasized safety over shuttle schedule
to a great or very great extent. Figure 2.3 illustrates NASA
managers and safety representative responses to our survey question
on the extent to which program priorities place greater importance on
safety than on meeting schedule.
Figure 2.3: Respondents Views
on Extent to Which Program
Priorities Place Greater
Importance on Safety Than on
Meeting Schedule
(See figure in printed
edition.)
Note: The chart presents the responses of 39 NASA shuttle managers
and safety officials. No respondents designated "little or no
extent."
Just over 60 percent of NASA managers and safety representatives we
surveyed believe that to a great or very great extent NASA emphasizes
safety over reducing cost. Figure 2.4 illustrates responses to our
survey question on the extent to which program priorities place
greater importance on safety than on cost reduction.
Figure 2.4: Respondents Views
on Extent to Which Program
Priorities Place Greater
Importance on Safety Than on
Cost Reduction
(See figure in printed
edition.)
Note: Percents do not add to 100 due to rounding. The chart
presents the responses of 39 NASA shuttle managers and safety
officials.
Contractor managers and working-level engineers also told us that
past funding reductions had not affected the quality of
safety-related information they develop. According to the contractor
managers, reductions in the shuttle flight rate and various
contractor productivity enhancements have enabled them to accommodate
past personnel cuts without, they believe, sacrificing the quality of
shuttle information they develop.
Some working-level engineers in the group interviews cited a variety
of concerns about the effects of funding reductions. For example,
the engineers said (1) investigations of lower priority issues take
longer to complete because there is not enough time to devote to
them, (2) keeping people with the required skill level is a concern,
and (3) there is a lack of storage in automated databases to archive
safety information. In addition, some engineers told us that the
funding reductions have adversely impacted employee morale because
people are being asked to accomplish more with fewer resources and
some employees fear losing their job. Some engineers said, however,
that although morale was lower, they did not believe it adversely
affected flight safety.
In November 1995, the Associate Administrator for Space Flight
testified that NASA plans an additional $2.5 billion cumulative
reduction from total shuttle funding requirements in fiscal years
1996 through 2000 against the fiscal year 1996 budget request.\5
According to the Associate Administrator, the program will achieve
the budget reductions through restructuring and other workforce and
content reductions.
Both NASA and contractor managers in our discussion groups expressed
concerns about how they would cope with additional funding cuts. For
example, the project managers for two contractors said that workforce
reductions can impact their timeliness in responding to situations
that arise. One contractor manager noted that while the company
measures various indexes such as "first time quality" and overtime,
it is difficult to specify the point at which additional program
changes to accommodate funding cuts might reduce quality. Another
contractor manager noted that at some point, funding reductions could
translate into not having enough people, so that maintaining the
required quality will mean continual schedule delays--a signal to the
contractor that their program cannot be reduced further.
Although firm estimates are not available, NASA expects to achieve
significant cost savings by turning shuttle operations over to a
prime contractor. The contractor would be responsible for shuttle
processing and launch, but NASA will retain the responsibility for
making the final launch decision. The single prime contractor would
combine many of the tasks now performed under 28 separate shuttle
program contracts. Savings are expected to accrue because shuttle
operations would be more efficient and require fewer civil service
employees. Current plans are to award the contract by fiscal year
1997.
During our discussion groups, some NASA managers expressed concern
about the transition of shuttle operations to a single prime
contractor. They feel that over the years NASA has assembled an
expert shuttle operations team and there are many unknowns about
making a transition to a new way of doing business. For example, the
safety and mission assurance organization maintains independent
oversight of shuttle operations. NASA's projections are that the
quality assurance oversight role will be reduced under the single
prime contractor concept of operations. Although managers expressed
concern about transitioning to a single operations contractor, in
response to our survey, 76 percent of the managers and safety
representatives said that quality assurance inspections and reviews
should be decreased. According to NASA, there will continue to be
independent oversight and the agency has plans to assure that the
oversight/insight will be properly focused with the reduced level of
resources expected.
NASA will retain decision authority and direct oversight over work
that is considered out-of-family (those events/activities that may
contain a level of risk beyond the known and accepted level). In
addition, NASA will retain the developmental effort for new hardware.
This work will transition to the single prime contractor, but only
after all the unknowns are understood by NASA. Further, NASA will
return to an oversight mode when there is an indication that there is
an increase in the understood level of risk for any reason. The
single prime contractor will be required to propose a process for
performing risk assessment and to demonstrate that they are able to
institute and properly manage the process. This includes the process
for keeping NASA informed of issues that have the potential for
increasing risk.
--------------------
\2 Space Shuttle: Declining Budget and Tight Schedule Could
Jeopardize Space Station Support (GAO/NSIAD-95-171, July 28, 1995).
\3 In commenting on a draft of the aforementioned report on June 23,
1995, NASA stated that "Although our space station assembly schedule
is demanding and funding is tight, we are currently on schedule and
within budget. We are committed to achieving the space shuttle
enhancements and launches required to assemble a productive station
on time for ourselves and our international partners."
\4 Space Shuttle: NASA Must Reduce Costs Further to Operate Within
Future Projected Funds (GAO/NSIAD-95-118, June 15, 1995).
\5 Statement of Dr. J. Wayne Littles, Associate Administrator for
Space Flight, National Aeronautics and Space Administration, before
the Subcommittee on Space and Aeronautics, Committee on Science,
House of Representatives, Nov. 9, 1995.
PRINCIPLES GUIDING THE
COMMUNICATIONS PROCESS
---------------------------------------------------------- Chapter 2:4
Through our discussion groups, individual interviews, and
observations, we identified several management principles related to
communication and information flow that appear to guide shuttle
communications. We also identified additional management principles
that we believe are essential to promoting shuttle safety in the
future. In our survey, we listed these principles and asked NASA
managers and safety representatives to identify those guiding
principles that they believe are essential to promoting shuttle
program safety as NASA deals with budget constraints, associated
downsizing, and restructuring in the near term, and with continuation
of shuttle flights in the long term. A large percentage of managers
and safety representatives we surveyed agreed that the following
principles are essential to promoting shuttle safety.
The organizational environment and structures for both contractor
and NASA personnel encourage timely, open discussion and debate
to ensure managers have the benefit of all relevant knowledge of
shuttle program issues.
Managers (civil service and contractors) stress safety over
schedule and cost and those managers foster these values among
employees.
The organizational environment encourages people (civil service and
contractor) to elevate concerns to higher management if they
believe the issues were not adequately addressed at lower
levels.
The working arrangement between NASA and contractors ensures agency
managers obtain continual knowledge of problems and issues so
that appropriate decisions can be made.
Organizational mechanisms enable NASA corporate-level managers to
carry out their decision-making responsibilities for certifying
readiness for flight.
NASA uses the most appropriate analytic and quantitative methods
available to assess shuttle risks and conduct sufficient
assessments and reviews to carry out the agency's oversight of
shuttle work processes.
Management information systems, including databases, are
accessible, accurate, complete, and timely for shuttle program
oversight and decision-making.
The NASA environment is a self-evaluative one that monitors its
effectiveness in communication and information flow and seeks
ways to improve it.
In addition to the principles previously listed, some NASA managers
provided additional principles that they believe are essential to
promoting shuttle safety as NASA deals with budget constraints,
downsizing, and restructuring.
Management of changes in the program receives adequate attention
and time to ensure that (1) program priorities are adhered to,
(2) government and contractor responsibilities for the reporting
and resolution of safety-related issues are clearly defined, and
(3) changes to the shuttle program are appropriately evaluated
before implementation.
Appropriate training is conducted to ensure that personnel can
effectively and efficiently carry out their work when changes in
program operations, processes, and staffing occur.
Morale and the working environment of employees are considered key
elements in assuring a safe and quality program.
Prime contractor management methods ensure quality of subcontractor
work.
CONCLUSIONS
---------------------------------------------------------- Chapter 2:5
NASA has created an organizational culture that encourages shuttle
program and contractor employees at all levels to bring safety
concerns to the attention of NASA's top management. NASA has also
established policies and procedures to ensure the free flow of needed
safety-related information. However, in response to our survey, some
shuttle program personnel expressed concern about whether NASA might
be emphasizing cost reductions over flight safety as planned budget
reductions and operational changes occur. Also, in response to our
survey, several types of issues were endorsed as always needing the
flight readiness review board's attention. However, opinions were
divided in other areas, suggesting that managers and safety
representatives may not be clear on each other's expectations about
the issues that should always be briefed. If, as is likely, the
planned shuttle operations contractor assumes more of the burden of
providing information to the flight readiness review, it will be
important to clearly specify the type and level of detail of
information to be provided.
NASA has adopted certain management principles that help guide the
shuttle launch decision process. These include such steps as
stressing safety over schedule and cost and developing an
organizational culture that encourages both contractor and NASA
personnel to elevate concerns to higher management if they believe
the issues were not adequately addressed at lower levels.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 2:6
We recommend that the Administrator of NASA identify guiding
principles of good risk management, such as those contained in this
chapter, and ensure that terms and conditions of the planned shuttle
operations contract reflect these principles.
We also recommend that the Administrator take steps to ensure that
flight readiness review participants understand and agree on the
minimum issues that should always be discussed at the review and the
level of detail that should be provided.
AGENCY COMMENTS
---------------------------------------------------------- Chapter 2:7
In commenting on a draft of this report, NASA agreed with our first
recommendation and stated that the agency is taking steps to
implement it. According to NASA, the shuttle flight operations
contract request for proposal and statement of work have been
carefully reviewed and these documents reflect the principles of good
risk management described in this report. NASA said that it will
ensure that the contract terms and conditions are compatible with
these principles.
Regarding the second recommendation, NASA said that it is appropriate
and the agency has recently completed an activity to update and
clarify the roles and responsibilities of each program element and
organization relative to the flight readiness review. The new
procedure is to be fully implemented in support of shuttle flight
STS-78 in June 1996.
We made additional changes to the report, where appropriate, based on
NASA's technical comments.
NASA HAS NOT DEVELOPED AN OVERALL
STRATEGY FOR USING QUANTITATIVE
RISK ASSESSMENT METHODS IN THE
SHUTTLE PROGRAM
============================================================ Chapter 3
The National Research Council recommended in 1988 that NASA apply
quantitative risk assessments to the shuttle program. However, NASA
still relies primarily on qualitative methods to assess and
prioritize significant shuttle risk. This approach relies heavily on
the judgment of shuttle engineers to identify significant risk items
that could cause loss of a shuttle or crew. Although NASA awarded a
contract for development of a quantitative method model known as a
probabilistic risk assessment\1 for the shuttle program, NASA has not
fully assessed the potential benefits of using the tool in routine
shuttle decision-making. The agency also has not developed an
overall strategy for assuring use of this method where it is
appropriate. In addition, databases are not always timely, complete,
accessible, or reliable enough to be used in these type analyses.
--------------------
\1 Probabilistic risk assessment is a systematic methodology for
evaluating the probability that an event will occur and predicting
the consequences should the event occur.
NATIONAL RESEARCH COUNCIL
RECOMMENDED QUANTITATIVE
APPROACH
---------------------------------------------------------- Chapter 3:1
The National Research Council investigation of NASA's risk assessment
approach following the Challenger accident found that quantitative
assessment methods had not been used to directly support NASA
decision-making related to the space shuttle. The Council
recommended that probabilistic risk assessment approaches be applied
to the shuttle at the earliest possible date. They also recommended
that databases be expanded to support probabilistic risk assessments,
trend analysis, and other quantitative analysis and that NASA develop
a statistical sciences capability to perform necessary risk
assessments.
QUANTITATIVE METHODS USED IN
OTHER HIGH-RISK AREAS
-------------------------------------------------------- Chapter 3:1.1
Quantitative methods, such as probabilistic risk assessments, have
been used in the decision-making process by other federal agencies
involved in high-risk ventures. For example, the Nuclear Regulatory
Commission uses probabilistic assessments in its regulation and
oversight of nuclear power plants. These techniques are used to
assess the safety of operating reactor events and as an integral part
of the design certification review process for advanced reactor
designs. Commission officials stated they have found probabilistic
risk assessments to be an effective tool for making plant-by-plant
examinations to determine areas needing more emphasis, such as how
long it takes a utility to respond to problems. Commission officials
told us that, in their experience, probabilistic risk assessments can
help identify and focus their attention on risk areas that require
the most resources.
The Environmental Protection Agency uses quantitative risk
assessments to determine the health risks posed by superfund
hazardous waste sites. The agency reviews contaminated sites for
investigation and cleanup. One element of the investigation is a
baseline risk assessment--an evaluation of current or potential
threat to human health. The evaluation establishes probabilities
that are used to decide whether a site requires cleanup. For
example, if the risk of humans developing cancer from site chemicals
is greater than 1 in 10,000, Environmental Protection Agency policy
requires that the site be cleaned.
NASA pointed out that it is important to make a clear distinction
between quantitative risk assessments in general and the specific
probabilistic risk assessment method when determining the value of
applying these methods to space hardware issues. NASA said it
recognized that probabilistic risk assessments had proven valuable at
the Nuclear Regulatory Commission and the Environmental Protection
Agency. However, this method did not have comparable utility at
NASA. Reactor design and certification risk assessments are based on
failure rates compiled from hundreds of plants and facilities while
the shuttle has significantly less hard data available to quantify
risk. In addition, NASA said the public health risk posed by nuclear
power plant accidents or toxic waste sites argues for a multimillion
dollar investment in risk assessment that can span years of analysis.
In contrast, according to NASA most shuttle risk issues must be
resolved in a shorter time frame.
NASA'S RESPONSE TO THE
COUNCIL'S RECOMMENDATION
---------------------------------------------------------- Chapter 3:2
In response to the Council's interim report, NASA began taking
tentative steps toward the use of probabilistic analysis by
initiating contractor trial probabilistic risk assessments of some
shuttle elements. In parallel with this, NASA began developing a
procedure to prioritize the shuttle's highest risk elements. This
proposed technique would lend itself to the incorporation of
quantitative measures of risk and probabilities of occurrence as
these measures were developed. NASA planned to assess the benefits
and applicability of this method to the shuttle risk management
process based on the results of the contractor studies. A former
Associate Administrator for Safety and Mission Assurance indicated
that he would personally evaluate the probabilistic risk assessment
technique and develop a strategy for introducing it throughout NASA.
However, the strategy has not yet been developed.
Regarding its databases, NASA responded by developing a centralized
database designed to improve the quality of information by providing
an integrated view of the status of shuttle problems in near real
time. The Council recommended that development of this system be
given a high priority. NASA developed a database to provide
information, but, as discussed in other sections of this chapter,
that database has limitations.
NASA officials told us that while some progress has been made, the
use of probabilistic methods have not reached a mature state at NASA.
NASA has made limited use of probabilistic risk assessments of the
shuttle, including proof-of-concept studies, assessment of some
specific shuttle systems, and required assessments of accident
probabilities for launches involving radioactive material. A 1994
survey of probabilistic methods used in structural design, which
included some shuttle projects, found that there is no agreed-upon
approach across centers for preferred methods, practices, or software
and that the various quantitative tools have not been fully examined,
evaluated, and accepted by NASA centers.
In early 1994, the NASA Administrator and the Office of Space Flight
concluded that a probabilistic risk assessment of shuttle risk was
needed to guide safety improvement decision-making. According to a
safety official, NASA contracted with Science Applications
International Corporation in January 1994 to conduct a probabilistic
risk assessment of the space shuttle. This was the first assessment
to include a complete shuttle mission. The contractor was required
to develop and apply a risk model of the shuttle during flight and to
quantify in-flight safety risk. The analysis was to identify,
quantify, and prioritize risk contributors for the shuttle.
According to the model's author, secondary objectives were to provide
a vehicle for introducing and transferring probabilistic risk
assessment technology to NASA, and to demonstrate the value of the
technology. The model was completed in April 1995.
According to the contractor who developed the probabilistic risk
assessment model, the model could be a useful tool in NASA's
management of shuttle risks. For example, the model might be used to
establish realistic cost objectives for redesigning the high risk
components, helping to assure that limited resources are focused
toward solving those problems that will have the most impact on
safety. The National Research Council also noted that a detailed
quantitative risk assessment provides decision makers with a better
basis for managing risk.
An internal shuttle program survey of managers, safety experts, and
senior engineers revealed mixed reactions to the model. Although
generally positive, respondents cited some concerns. For example,
some respondents commented that more use of actual failure data would
have benefited the analysis and that some assumptions used were
debatable. Some found fault with the excess use of expert opinion
and the lack of thoroughness in delineating certain assumptions.
Following the survey, the Deputy Associate Administrator for Space
Flight informed shuttle and safety managers that they should feel
free to use the report and model as a "limited tool in the risk
management tool box."
According to some NASA safety officials, the model has not been
routinely used by NASA personnel as a risk assessment tool because
officials are still evaluating the utility of the model and barriers
exist to its use by NASA employees. For example, there is no
instruction manual for using the model and it requires use of
contractor owned software. According to safety officials, NASA does
not have current copies of the required software and older inadequate
versions are limited. According to these officials, only one NASA
employee has been able to use the model on a NASA computer using the
older software. In addition, no firm decisions have been made
regarding maintenance and update of the model to reflect shuttle
changes, such as the super light weight external tank. Safety
officials stated they are continuing to assess the model to determine
its utility within NASA.\2
NASA project and safety officials compile a list of significant
shuttle risk issues for each project to target resources and manage
risk reduction efforts. Only risks that can be reduced by
incorporating hardware or procedure modifications are included in the
assessment. According to NASA's April 1995 shuttle safety risk
ranking methodology guidance, the source of risk information
currently used in the rankings is qualitative and the process ranks
catastrophic events by judgmentally derived prioritization matrices.
The guidelines state that many comparisons of catastrophic events
could be made but are sometimes subjective, emotional, and rely on
different techniques. A complete probabilistic risk assessment would
be the most desirable analysis, according to the guidelines, but
probabilistic analyses are labor-intensive efforts that require many
system experts, a complete understanding of the methodology, and
proper management of the effort.
--------------------
\2 After our field work was completed, NASA informed us that the
Office of Safety and Mission Assurance and the Office of Space Flight
were working together to procure the necessary software to operate
the shuttle probabilistic risk assessment model. The software is
being procured to (1) maintain the currency of the model by the
addition of new flight and test data as they become available; (2)
modify the model, as appropriate, to reflect the most current shuttle
design configuration; and (3) permit possible "reverse engineering"
of the model to enable the use of its major components, both
separately and together, using less expensive and more commonly used
software applications.
LIMITED USE OF QUANTITATIVE
RISK ASSESSMENTS
-------------------------------------------------------- Chapter 3:2.1
NASA has made limited progress in adopting the National Research
Council's recommendations that the agency assess risk with
quantitative methods, such as probabilistic risk assessments. NASA
uses a variety of methods to assess shuttle risk issues, and efforts
are underway to increase the use of quantitative methods.
Qualitative methods are still widely used when risk issues are
thought to be well understood. NASA has made limited use of the
classical probabilistic risk assessment method of analysis. Cost,
lack of specific expertise, and lack of data are the reasons cited
for limited use.
According to shuttle and safety managers, lack of a strategy for
incorporating the methods into decision-making processes has impeded
NASA's progress in adopting the National Research Council's
recommendations on risk assessments. Also, insufficient expertise
exists at NASA to conduct specific quantitative analyses, such as
probabilistic risk assessments.
NASA project and safety officials told us that progress in
implementing quantitative risk assessment methods has been impeded
because NASA does not have a working strategy for formalizing these
methods for the shuttle program. Such a strategy would include clear
and measurable goals, resource requirements, assessments of current
utilization and skills within NASA, and training needs, including the
need to learn by doing selected projects. Without this focus,
projects and safety organizations are skeptical about the cost and
benefits of using the probabilistic risk assessment model.
Project and safety officials at several centers expressed concerns
about the applicability of probabilistic risk assessments to the
shuttle program. While officials stated they recognize probabilistic
risk assessments could be used as an effective additional tool to
assess risk, they see a need for more training on the methodology and
the need to learn by doing selected projects. Several stated they do
not have the resources needed for this type analysis but are
stretched just to operate their programs. Several officials stated
they believe there is a lack of trust in the probabilistic risk
assessment method because people do not understand it. Many
officials expressed concern about the complexity of the shuttle
probabilistic risk assessment model, the lack of good data, and the
dependence upon the contractor to make needed changes to the model.
Several officials commented that NASA needs a "champion" at
headquarters to provide a focused effort to emphasize use of these
tools when appropriate.
NASA headquarters safety and mission quality officials stated they
have not developed a master plan for formalizing quantitative
techniques within NASA or made the progress they would like in this
area. However, steps are being taken to address several of the
concerns expressed by project and safety officials at the centers.
For example, training courses in risk management and assessment are
being planned that will be offered to safety and other NASA
personnel. Reference manuals on sources for data and techniques on
risk assessments are under contract. According to NASA safety
officials, the first effort to develop these type documents began in
1989 but was unsuccessful and the documents were not published.
However, NASA has established a coordination committee to develop a
standard, comprehensive approach to introduce structural design
methods that can be used in the shuttle program.
NASA is also trying to give this issue visibility as the agency plans
to move to a single prime contractor and to assure that the statement
of work contains provisions that the contractor use quantitative risk
assessment techniques where appropriate.
According to the National Research Council, decisionmakers within
NASA must be supported by people skilled in the statistical sciences
to aid in the transformation of complex data into useful information.
The Council recommended that NASA develop a staff of experts in these
areas to provide improved analytical support for risk management.
NASA officials at several centers and at NASA Headquarters told us
they lack sufficient personnel with these skills, and in one case, a
center lost needed contractor skills that caused the delay or
termination of a planned analytical project. A 1994 NASA survey of
probabilistic methods used in structural design work found that a
wide variance of knowledge exists at the centers and that a majority
of working-level engineers are not familiar with and do not use
probabilistic methods.
NASA'S DATABASES DO NOT PROVIDE
SUFFICIENT INFORMATION FOR
QUANTITATIVE ANALYSIS
---------------------------------------------------------- Chapter 3:3
Another factor that has hindered development of quantitative methods
of risk assessment is that NASA's databases do not always provide
timely, accessible, accurate, and complete information. A large
percentage of managers and safety representatives we surveyed believe
that NASA should provide management information systems, including
databases that are accessible, accurate, complete, and timely for
shuttle program oversight and decision-making. However, more than
half assessed NASA's current management information systems as
needing improvement.
NASA has developed automated database systems to provide shuttle data
used in decision-making. One system, called the Program Compliance
Assurance and Status System, is a central database designed to
integrate existing data, such as in-flight anomalies, from various
sources in the program. Another system, the Problem Reporting and
Corrective Action system, provides data to the central system and is
designed to document and track problems in the program.
According to NASA officials, the Program Compliance Assurance and
Status System is neither timely nor fully utilized. The system is
cumbersome to use because it is based on older technology, some trend
and other data is not centralized in the system, and software needed
to convert contractor data to NASA database format has not been
developed. Program officials told us they maintain trends on some
aspects of the shuttle program, but have found the centralized system
to be difficult to use and not compatible with other existing
databases. The officials stated that the required conversion
programs have never been developed to input some contractor data into
the system. In some cases, safety officials must obtain data
directly from contractors to conduct quantitative risk assessments.
Because the system is hard to use in real-time and the data is not
always current, some officials stated they are using a different
software program with faster computers to access and correlate data
more rapidly.
A January 1995 internal report on shuttle problem reporting system
data integrity at two centers found missing criticality codes on
thousands of entries. Blank entries could, therefore, be interpreted
as either not applicable or inadvertently omitted. A NASA
Headquarters official was not aware of any corrective action on this
matter. Officials told us that the Problem Reporting and Corrective
Action System records are often not reliable, lack data needed for
quantitative risk assessments, and lack uniformity in categorizing
problems. The system also contains entries that may not meet the
definition of a "real problem." NASA safety officials acknowledged
that the system needs improvement but stated no firm decision has
been made regarding the extent of improvements pending the transition
to a single prime contractor.
CONCLUSIONS
---------------------------------------------------------- Chapter 3:4
NASA has made limited progress in adopting the National Research
Council's recommendation that the agency assess risk with
quantitative methods, such as probabilistic risk assessments. NASA
officials, for the most part, rely on qualitative methods for
assessing risk in the shuttle program when they believe risk issues
are well understood. Although some progress has been made, NASA
lacks an overall strategy with focused management emphasis to
incorporate methods, such as probabilistic risk assessments into the
shuttle program, when appropriate. Resource constraints and specific
expertise are cited as barriers to increased use of these methods.
In addition, NASA's databases need improvement and are not fully
utilized by decisionmakers nor are they adequate to support the use
of quantitative risk assessment methodologies.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 3:5
We recommend that the Administrator of NASA establish a strategy, to
include specific milestones, for deciding whether and how
quantitative methods, such as probabilistic risk assessments, might
be used as a supplemental tool to assess shuttle risk.
We also recommend that the Administrator assess the shuttle program's
centralized database, as well as other databases, to insure that data
required to conduct risk assessments and inform decisionmakers is
accessible, timely, accurate, and complete.
AGENCY COMMENTS
---------------------------------------------------------- Chapter 3:6
NASA agreed with the need to establish a strategy, with milestones,
for incorporation of quantitative risk assessment methods into the
shuttle's risk management program. According to NASA, the agency
will establish a team to develop the strategy.
NASA also agreed that the shuttle program's centralized databases
need to be assessed. In this regard, NASA will form a team of
engineers to thoroughly examine the Program Compliance Assurance and
Status system. The team will be tasked to determine the adequacy of
what presently exists and make recommendations for improvements as
necessary. The assessment team will report to the shuttle program
manager. In addition, the Problem Reporting and Corrective Action
System is being examined at each center by a reengineering team.
This team is searching out deficiencies and will recommend needed
improvements that must be implemented by the shuttle flight
operations contractor.
We made additional changes to the report, where appropriate, based on
NASA's technical comments.
METHODOLOGY FOR GROUP INTERVIEWS
AND SURVEY ASSESSING THE FLOW AND
QUALITY OF SHUTTLE SAFETY-RELATED
INFORMATION
=========================================================== Appendix I
OVERVIEW
--------------------------------------------------------- Appendix I:1
This appendix describes the methodology we used to study the flow and
quality of safety-related information in the shuttle program.
Appendix II shows the questions and results of our survey of shuttle
program officials and appendix III provides the questions used in our
group interviews.
We conducted group interviews with National Aeronautics and Space
Administration (NASA) managers located at the three NASA field
centers with primary responsibility for managing shuttle program
elements and at the program's headquarters. The group interviews
enabled participants to exchange their perspectives on communication
within the shuttle program, provided us with an understanding of
these complex areas, and produced concrete illustrations.
While the interviews provided insights that may only arise in a group
setting, we also sent a brief survey to these managers and to safety
officials responsible for shuttle hardware components to obtain more
precise measures of the themes that emerged in the group discussions.
We included personnel in the shuttle program's safety and mission
assurance and engineering organizations in our interviews and survey
as well as project and program managers in order to obtain a full
range of perspectives on communication. The objectivity and accuracy
of our interpretation of the transcribed group discussions were
verified through several approaches.
Another component of our design was to interview personnel with three
shuttle program contractors. We selected the contractors responsible
for the solid rocket motor and the shuttle main engine because these
systems are complex, high-risk elements. We also selected the
contractor responsible for processing the shuttle for launch because
it is a labor-intensive effort. We interviewed the program managers
and senior safety officials for shuttle program contractors in order
to understand the flow of information between NASA and contractors
from the contractor's perspective. We held separate group interviews
with working-level engineers with each of the three contractors in
order to understand the flow of information within the contractor's
organization. The viewpoints expressed in the contractor interviews
cannot be generalized to other shuttle program contractors.
NASA GROUP INTERVIEWS
--------------------------------------------------------- Appendix I:2
We included shuttle project and program managers (or their designated
alternates) located at three centers and at headquarters in the NASA
group interviews. We held a group interview composed of 6 to 14
managers at each of the following locations: Johnson Space Center,
Kennedy Space Center, Marshall Space Flight Center, and the shuttle
program's headquarters organization. In total, 40 individuals in the
following positions or alternates designated by NASA participated:
shuttle program and project managers and senior managers in the
safety and mission assurance and the program's engineering
organizations. The group interviews were conducted between April and
June 1995.
The main questions in our NASA group interviews focused on
information conveyed at the Level I flight readiness review, the
extent to which various conditions ensure that serious issues come to
the attention of management, and funding reductions and restructuring
as they relate to morale and the transmittal of high quality
information for safety assessment.
We moderated the interviews that were audio-recorded. A co-moderator
took notes in the event that audio-recordings were not complete or
clear. The transcription of each interview was systematically
analyzed by the moderator to extract the discussion themes and
illustrations of these themes.
We ensured the adequacy of our analysis and interpretation of the
group interviews through several steps. A summary of each group
discussion with NASA personnel was developed by our staff and
independently audited by another staff member who traced each
statement in the summary back to the portion of the transcribed text
which supported the statement. The summaries were then reviewed by a
NASA official with in-depth knowledge of the shuttle program who had
not participated in the group interviews. The reviewer was asked to
assess the summaries for their technical correctness and objectivity.
As a final verification step, each participant in the group
interviews received the summary of the interview he or she
participated in, along with a copy of the transcription of the
discussion. The participants were asked to verify that the summary
accurately reflected his or her input and the communication themes
that emerged. The suggested clarifications from the NASA reviewer
and the NASA group interview participants were incorporated in the
summaries.
CONTRACTOR INTERVIEWS
--------------------------------------------------------- Appendix I:3
We held interviews with the project manager and senior safety
official for the solid rocket motor, shuttle main engine, and shuttle
processing contractors during May and July 1995. The main questions
in these interviews focused on NASA reporting requirements,
information conveyed at Level III flight readiness reviews and other
reviews, the flow of shuttle-related information within the
contractor's organization and with NASA, and funding reductions and
restructuring as they relate to morale and the transmittal of
high-quality information for safety assessment.
We also held a group interview composed of 9 to 12 working-level
engineers at the three shuttle program contractors in our study. The
interviews were conducted in May and July 1995. We attempted to
include engineers from each of the contractor's major work areas,
including contractor safety organizations. The main questions in
these group interviews focused on the types of safety issues and
concerns the engineers brief to their management, the flow of
information within their organizations, and funding reductions and
restructuring as they relate to morale and the transmittal of high
quality information for safety assessment.
The contractor interviews were audio-recorded. The transcription of
each interview was systematically analyzed by our staff to extract
the discussion themes and illustrations of these themes. A summary
of each interview was developed by a staff member and independently
audited by another staff member. As a final verification step, each
interviewee received a copy of the summary and the transcribed
interview and was asked to verify that the summary accurately
reflected the discussion and the major themes that emerged. The
suggested clarifications from the contractor group interview
participants were incorporated into the summaries.
SURVEY METHODOLOGY
--------------------------------------------------------- Appendix I:4
We pretested a questionnaire on communications with managers at the
three space centers included in the group interview portion of our
study and at NASA Headquarters. Headquarters officials also
performed a technical review of the survey questions. The
questionnaire was distributed in August 1995, to each participant in
the NASA group interviews, except for two managers who had retired
from the program at the time of the survey. We also sent
questionnaires to the safety officials responsible for shuttle
hardware components. Survey recipients were told that the survey
results would be reported in summary form in our report and that any
discussion of individual answers would omit information identifying
the respondent. Of the 44 surveys we distributed, we received 39
responses, representing a response rate of 89 percent.
SURVEY QUESTIONS AND RESPONSES
========================================================== Appendix II
This appendix provides the exact text of the survey questions.
Transitional phrases used to guide the respondents from one topic to
another are not included. The appendix provides the combined number
of managers and safety representatives endorsing the response options
accompanying each survey question as well as the number of
respondents eligible to answer an item who did not answer.
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
(See figure in printed
edition.)
Several managers and safety representatives identified additional
guiding principles necessary for safe operation as they undergo
organizational and funding changes:
The guiding principles that:
Management of changes in the program receives adequate attention
and time to ensure that (1) program priorities are adhered to,
(2) government and contractor responsibilities for the reporting
and resolution of safety-related issues are clearly defined, and
(3) changes to the Shuttle Program are appropriately evaluated
before implementation.
Appropriate training is conducted to ensure that personnel can
effectively and efficiently carry out their work when changes in
program operations, processes, and staffing occur.
Morale and the working environment of employees are considered key
elements in assuring a safe and quality program.
Prime contractor management methods ensure the quality of
subcontractor work.
GROUP INTERVIEW QUESTIONS
========================================================= Appendix III
This appendix presents the questions used in our group interviews.
We used different sets of main questions for the interviews we held
with NASA personnel, program managers and safety officials with
Shuttle Program contractors, and contractor engineers at the working
level. Different sets of probes, not listed below, were prepared for
each set of main questions.
NASA GROUP INTERVIEW
QUESTIONS
--------------------------------------------------- Appendix III:0.0.1
1. In order for the Board to provide a corporate decision on launch,
what types of safety issues do you think always need to be briefed in
detail at the FRR--regardless of what was done in the past?
2. Unintentional filtering can occur if information is lost or
important details are missing as messages are transmitted within an
organization. Several conditions can minimize the opportunity for
such unintentional filtering to occur in the Shuttle Program. For
example,
An organizational culture that encourages people to discuss safety
concerns with management and to elevate concerns to higher
management if they believe the issues were not adequately
addressed at lower levels.
A culture that encourages contractors to raise safety issues with
their government counterparts.
A parallel assessment and/or review function, such as SR&QA, with
an independent reporting chain to the Associate Administrator
for Safety and Mission Quality.
Program priorities which do not tempt managers to override safety
considerations in order to meet schedule or cut costs.
Databases that permit timely retrieval of complete and accurate
information relevant to shuttle risk assessment and
decision-making.
The question is: To what extent, if at all, does the Shuttle Program
have these or other conditions that ensure that management is
informed of serious safety issues? Please discuss your position on
each of these conditions.
3. (Headquarters only) What impact, if any, have funding reductions
during the last 3 years had on the morale of civil service employees
at Headquarters and at each of your Centers?
4. (Centers only) During the last 3 years, what impact, if any, have
funding reductions or related restructuring or downsizing had on the
morale of civil service employees that you work with?
5. (Headquarters only) What impact, if any, do you think the funding
reductions have had on the quality of safety-related information you
develop or receive?
6. (Centers only) Are there any functions, processes, or tasks that
have been changed or eliminated during the last 3 years because of
these factors? (If yes) What effect, if any, have these changes or
cuts had on the quality of safety-related information you develop?
7. (Centers only) While changes or cuts may not have a noticeable
effect on the quality of safety-related information, they could
increase the risk of degraded quality beyond what is acceptable.
What techniques were used to ensure that these changes or cuts did
not increase the risk beyond what is acceptable?
INTERVIEWS WITH
CONTRACTOR PROGRAM
MANAGERS AND SAFETY
OFFICIALS
--------------------------------------------------- Appendix III:0.0.2
1. What kinds of problems, hazards, or other safety issues or
concerns does NASA require you to report to them? How does NASA
require that you document these safety issues or concerns? Are there
any types of issues or concerns that are not documented?
2. Over about the last 2 years, what types of safety issues did your
program office discuss in detail at your reviews that prepare for the
Level III FRRs?
3. If there is disagreement among contractor personnel about whether
a safety issue should be brought forward to NASA, how is it handled?
4. What do you see as your role and responsibilities at the Level
III FRR or in support of this review? How about the COFR 6 review?
5. Over about the last 2 years, what types of safety issues or
concerns did your office brief NASA on in detail at the Level III
FRR? How about at COFR 6 reviews?
6. Finally, as a contractor, what types of safety issues do you
think your office should always brief in detail at NASA reviews,
regardless of what was done in the past?
7. If you had a safety concern that in your view did not get
surfaced at the appropriate NASA review level or if you disagreed
with a safety decision made by NASA, what avenues are open to you to
communicate your concerns?
8. See item 2, NASA Group Interview Questions. The identical
question was asked of contractor officials except for the framing of
the last part of the question, as shown below.
The question is: To what extent, if at all, does the Shuttle Program
and your organization have these or other conditions that ensure that
management is informed of serious safety issues? Please discuss your
position on each of the conditions. Please consider conditions first
in NASA and then within your own organization.
9. Has your shuttle work had any funding reductions or related
restructuring or downsizing during the last 3 years? (If yes) What
impact, if any, have these factors had on the morale of contractor
employees you work with?
10. Were any functions, processes, or tasks related to your work on
the shuttle changed or eliminated during the last 3 years because of
reductions or associated downsizing or restructuring? (If yes) What
effect, if any, did these have on the quality of shuttle information
you develop?
GROUP INTERVIEWS WITH
WORKING-LEVEL ENGINEERS
--------------------------------------------------- Appendix III:0.0.3
1. Over about the last 2 years, did you brief your management in
detail on any safety issues or concerns, and if so, what types of
safety issues or concerns were these?
2. If there is a disagreement among contractor personnel about
whether a safety issue should be brought forward to your higher
levels of management, how is it handled?
3. If you had a safety concern that did not get surfaced at the
appropriate level, or if you disagreed with a safety decision made at
higher levels, either within your contractor's plant or NASA, what
avenues are open to you to communicate your views?
4. Given the organizational culture within your organization, how
acceptable or unacceptable do you believe it is to voice safety
concerns?
5. Has your shuttle work had any funding reductions or related
restructuring or downsizing during the last three years? (If yes)
During the last three years, what impact, if any, have these factors
had on the morale of contractor employees you work with?
6. Are there any functions, processes, or tasks related to your work
on the shuttle that have been changed or eliminated during the last 3
years because of funding reductions or associated downsizing or
restructuring? (If yes) What effect, if any, have these changes or
cuts had on the quality of shuttle information you develop?
(See figure in printed edition.)Appendix IV
COMMENTS FROM THE NATIONAL
AERONAUTICS AND SPACE
ADMINISTRATION
========================================================= Appendix III
enclosures included suggestions for technical changes, which we
incorporated where appropriate.
(See figure in printed edition.)
(See figure in printed edition.)
MAJOR CONTRIBUTORS TO THIS REPORT
=========================================================== Appendix V
NATIONAL SECURITY AND
INTERNATIONAL AFFAIRS DIVISION,
WASHINGTON, D.C.
--------------------------------------------------------- Appendix V:1
Lee Edwards
James Beard
Fred Felder
Richard Irving
Julia Kennon
Marilyn Mauch
Terry Wyatt
*** End of document. ***
|
NEWSLETTER
|
|
Join the GlobalSecurity.org mailing list
|
|
|
