Find a Security Clearance Job!

Space

Space Station: Plans to Expand Research Community Do Not Match Available
Resources (Letter Report, 11/22/94, GAO/NSIAD-95-33).
The National Aeronautics and Space Administration (NASA) plans to
launch, assemble, and operate an earth-orbiting microgravity and life
sciences research laboratory--International Space Station
Alpha--starting in 1997.  A robust research community is needed to help
carry out the proposed space station's scientific mission of studying
the effects of weightlessness on various materials and life forms.
However, burgeoning costs associated with building the space station
could stymie efforts to develop a research program to support the space
station's scientific agenda because NASA may not have the money to pay
researchers to conduct needed experiments.  This report reviews (1) what
NASA is doing to assess the required size of the research community
needed for the space station and to ensure that such a community will be
available, (2) how NASA will ensure that the research selected for the
space station will be the best possible, and (3) whether a recently
canceled shuttle research flight harmed NASA's ability to assemble a
research community for the  space station.
--------------------------- Indexing Terms -----------------------------
 REPORTNUM:  NSIAD-95-33
     TITLE:  Space Station: Plans to Expand Research Community Do Not 
             Match Available Resources
      DATE:  11/22/94
   SUBJECT:  Future budget projections
             Aerospace research
             Scientific research
             Human resources utilization
             Cost control
             Research and development costs
             Budget decision units
             Research program management
IDENTIFIER:  NASA International Space Station Alpha Program
             NASA Space Acceleration Measurement System
             NASA Extended Duration Orbiter Medical Program
**************************************************************************
* This file contains an ASCII representation of the text of a GAO        *
* report.  Delineations within the text indicating chapter titles,       *
* headings, and bullets are preserved.  Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are           *
* identified by double and single lines.  The numbers on the right end   *
* of these lines indicate the position of each of the subsections in the *
* document outline.  These numbers do NOT correspond with the page       *
* numbers of the printed product.                                        *
*                                                                        *
* No attempt has been made to display graphic images, although figure    *
* captions are reproduced. Tables are included, but may not resemble     *
* those in the printed version.                                          *
*                                                                        *
* A printed copy of this report may be obtained from the GAO Document    *
* Distribution Facility by calling (202) 512-6000, by faxing your        *
* request to (301) 258-4066, or by writing to P.O. Box 6015,             *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time.                                    *
**************************************************************************
Cover
================================================================ COVER
Report to the Ranking Minority Member, Subcommittee on Oversight of
Government Management, Committee on Governmental Affairs, U.S. 
Senate
November 1994
SPACE STATION - PLANS TO EXPAND
RESEARCH COMMUNITY DO NOT MATCH
AVAILABLE RESOURCES
GAO/NSIAD-95-33
Space Station
Abbreviations
=============================================================== ABBREV
  EDOMP - Extended Duration Orbiter Medical Program
  IML - International Microgravity Laboratory
  NASA - National Aeronautics and Space Administration
  NIH - National Institutes of Health
  SAMS - Space Acceleration Measurement System
  SLS - Spacelab Life Sciences
Letter
=============================================================== LETTER
B-259020
November 22, 1994
The Honorable William S.  Cohen
Ranking Minority Member
Subcommittee on Oversight of Government Management
Committee on Governmental Affairs
United States Senate
Dear Senator Cohen: 
The National Aeronautics and Space Administration's (NASA) goal is to
launch, assemble, and operate an earth-orbiting microgravity\1
and life sciences research laboratory--International Space Station
Alpha--starting in 1997.  You requested that we review NASA's efforts
to develop a robust life and microgravity sciences research community
for the space station.  Specifically, we reviewed (1) what NASA is
doing to assess the required size of the research community needed
for the space station and to ensure that such a community will be
available, (2) how NASA will ensure that the research selected for
the space station will be the best possible, and (3) whether a
recently canceled Shuttle research flight adversely affected NASA's
efforts to develop a research community for the space station. 
In June 1994, we reported to you on the impact of the expanded
Russian role on space station funding and research.  We stated that
Russian participation in the space station would substantially
increase overall station research resources.  However, the degree to
which the U.S.  research community will benefit from these increased
resources has yet to be determined.\2
--------------------
\1 Microgravity is a condition of free-fall within a gravitational
field in which the weight of an object is significantly reduced
compared to its weight at rest on Earth.  When orbiting Earth, a
spacecraft is in continuous free-fall and, thus, in microgravity. 
Microgravity is also a low acceleration environment where the
acceleration imparted to an object is one-millionth of that measured
at the earth's surface.  Following NASA usage, the term "microgravity
science" in this report refers to the study of chemical and physical
phenomena in a low acceleration environment. 
\2 Space Station:  Impact of the Expanded Russian Role on Funding and
Research (GAO/NSIAD-94-220, June 21, 1994). 
   RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
NASA has taken an initial step to assess the size of the space
station's research community, but it is not intending to develop this
community by directly soliciting proposals to do research on the
space station.  Instead, NASA is focusing on developing a
comprehensive research program that emphasizes more ground-based
research and uses space flight only for research efforts that require
a microgravity environment in space.  To accomplish this program,
NASA wants to greatly increase the number of ground-based
investigators.  This science-oriented approach is reasonable, but
funding levels could jeopardize it unless NASA adjusts its funding
priorities.  To achieve its goal, NASA would need increased funding
for its life and microgravity sciences research and analysis over
fiscal years 1995 to 1999--the formative years for the initial
development of space station-related research.  However, NASA expects
such funding to remain constant.  The result could be a smaller than
desired number of investigators in the ground-based research program
from which station-based research will be selected. 
The existing process of using peer review panels to judge the
scientific merit and microgravity-related relevance of experiments
for the space station will continue.  The peer review scores for 285
life and microgravity sciences research proposals show that NASA's
funding decisions were generally consistent with the findings of the
peer review panels:  most top-rated proposals and none of the lowest
rated ones were funded.  But peer review panels and NASA sometimes
disagreed on the scientific merit and relevance of these proposals. 
For example, 4 of the 15 U.S.  experiments selected by NASA for a
July 1994 Space Shuttle research flight--the International
Microgravity Laboratory--were not among those rated highest in the
peer review process. 
NASA's efforts to increase the size of its life and microgravity
sciences research community are not likely to be adversely affected
by the February 1994 cancellation of the third Spacelab Life Sciences
flight (SLS-3).  The U.S.  principal investigators on that flight
stated that, although they were concerned about NASA's lack of
communication about the flight cancellation, most of them have been
accommodated on other space flights and they generally will be able
to meet their experiment objectives.  All of the investigators with
whom we spoke also plan to continue submitting proposals for future
NASA research opportunities. 
   BACKGROUND
------------------------------------------------------------ Letter :2
As part of its mandate to guide the nation's civil space program,
NASA is to preserve U.S.  preeminence in critical aspects of space
science, technology, and applications.  The goal of life and
microgravity sciences is to study gravity-dependent physical
phenomena and those phenomena obscured by the effects of gravity in
biological, chemical, and physical systems.  Research is conducted in
biotechnology (e.g., protein crystal growth), combustion science,
fluid physics, life and biomedical sciences, and materials science. 
Life science research in space biology studies the effects of gravity
on living systems by using acceleration environments across the
"gravity continuum"--micro, earth-normal, and hypergravity.\3
NASA's Office of Life and Microgravity Sciences and Applications,
which was formed in 1993, funds this type of research.  Between
fiscal years 1989 and 1994, the annual budget authority for life and
microgravity research increased by 114 percent, from $222.4 million
to $476.3 million.\4
Not all aspects of life and microgravity sciences research require a
space-based environment.  Short duration, low acceleration
environments can be created in drop towers (2 to 5 seconds of free
fall),\5 aircraft flying a distinctively curved flight path (up to 23
seconds of low gravity), and suborbital rockets (over 300 seconds). 
Hypergravity can be created by a centrifuge.\6
Space-based research is principally conducted in pressurized and
nonpressurized facilities on the Space Shuttle.  The centerpiece for
this research is a 23-foot by 16-foot pressurized
module--Spacelab--that fits in the Space Shuttle payload bay. 
Spacelab was developed by the European Space Agency and contains
utilities, computers, work areas, and instrument racks for
experiments.  An exterior cutaway view of Spacelab is shown in figure
1. 
(See figure in printed edition.)Figure 1:  Spacelab in Space Shuttle
Payload Bay
Source:  NASA. 
The most recent Spacelab flight was the second International
Microgravity Laboratory (IML-2), which ended a 15-day mission on the
Space Shuttle on July 23, 1994.  IML-2 was a collaborative effort by
NASA; the European Space Agency; and the national space agencies of
Canada, France, Germany, and Japan.  According to NASA officials,
IML-2 provided a preview of the science operations to
The IML flights, which began in January 1992, gave the U.S scientific
community access to foreign-developed flight hardware while providing
the international research community with access to the Space
Shuttle/Spacelab.  Approximately 80 investigations were performed on
IML-2, including 15 U.S.  experiments--11 in the biotechnology, fluid
physics, and materials science and 4 in the life sciences.\7 IML-2
was the last flight of this international series of spacelabs before
the station era begins in 1997.\8 An interior view of IML-2 is shown
in figure 2. 
(See figure in printed edition.)Figure 2:  IML-2
Source:  NASA. 
NASA publicly solicits research proposals from investigators in the
life and microgravity research communities.  The funding decision is
principally based on an evaluation of the project's scientific merit
by a peer review panel.  NASA's peer review and other quality
assurance procedures are outlined in appendix I. 
--------------------
\3 Other life science research areas are space physiology and
countermeasures, space radiation health, space human factors
engineering, and advanced life support. 
\4 Budget authority is the authority provided by law to enter into
financial obligations that will result in immediate or future outlays
of federal government funds.  The basic forms are appropriations,
borrowing authority, and contract authority. 
\5 For example, experiments dropped from the 89-foot tower at NASA's
Lewis Research Center are in free fall for 2.2 seconds. 
\6 A centrifuge is a rotating device designed to provide a high
acceleration environment.  For example, the 58-foot diameter
centrifuge at NASA's Ames Research Center, California, can expose
humans to centrifugal forces up to 20 times Earth-normal gravity
(20g).  Another Ames centrifuge can expose incubated cells to
centrifugal forces up to 6g. 
\7 The number of U.S.  experiments excludes the Space Acceleration
Measurement System (SAMS) and Extended Duration Orbiter Medical
Program (EDOMP).  SAMS is designed to measure and record low-gravity
accelerations at three experiment sites simultaneously.  EDOMP
gathers data on the effects of long duration exposure to microgravity
on human physiology. 
\8 Three more pressurized Spacelabs are scheduled to fly on the
Shuttle before the start of the station era:  U.  S.  Microgravity
Laboratory-2, 1995; a recently announced Life and Microgravity
Spacelab, 1996; and Microgravity Science Laboratory-1, 1997.  Another
Spacelab--Neurolab--is scheduled to fly in 1998. 
   DEVELOPING A LARGER RESEARCH
   COMMUNITY WILL REQUIRE
   ADJUSTING FUNDING PRIORITIES
------------------------------------------------------------ Letter :3
NASA intends to build a space station-era research community from the
ground up.  To do so, a larger cadre of ground-based researchers than
currently available will be needed to adequately support U.S. 
research on the station.  A NASA official estimates that the number
of ground-based microgravity researchers needs to increase from 73 to
240 between fiscal years 1992 and 1998.\9 NASA officials have not
made comparable estimates for life science researchers.  To
accomplish this goal, NASA has abandoned its tradition--principally
associated with life science research--of soliciting research
proposals for general and specific space flight opportunities. 
Although this approach appears reasonable, the planned funding levels
do not match the program's objective, and funding priorities may need
to be reassessed if the number of life and microgravity ground-based
investigators is to be significantly increased.\10
--------------------
\9 The estimate assumes a requirement for two flight investigators
for each research discipline each year, and four ground-based
investigators for each flight investigator. 
\10 NASA is not necessarily the only source of federal funds for
station-era research.  For example, NASA and several institutes of
the National Institutes of Health have agreed to establish
collaborative science planning and joint funding of research. 
      SELECTING IML-2
      INVESTIGATORS ILLUSTRATED
      TWO BASIC APPROACHES FOR
      DEVELOPING RESEARCH
      COMMUNITY
---------------------------------------------------------- Letter :3.1
In recent years, NASA has used two approaches for developing life and
microgravity science research communities--"select for flight" or
"select for science." In the select-for-flight approach, all of the
U.S.  life science and most of the U.S.  microgravity investigators
on IML-2 were selected from proposals submitted in response to
flight-related announcements.  In the select-for-science approach,
two IML-2 microgravity investigators were selected from researchers
who submitted proposals in response to two 1991 discipline-related
("fundamental science" and "biotechnology") research announcements. 
A NASA program scientist considers the IML-2 flight to have been a
programmatic success and, in some respects, a model for the
international space station.  According to a NASA official, one
indication of the flight's success was the amount of good research
generated from the many proposals submitted in response to a mix of
science and flight-related research announcements.  Additionally, two
of NASA's recent research announcements were in the select-for-flight
tradition:  its July 1993 announcement soliciting proposals for
research on a 1998 space life sciences flight (Neurolab) and its
February 1994 announcement soliciting proposals for life science
research on the Russian space station Mir from 1995 to 1997. 
Presumably then, one effective way to develop a research community
would be to solicit specific proposals for research that are directly
related to the space station.  NASA, however, has chosen to move
toward exclusive use of "select for science," as discussed below. 
      "SELECT FOR SCIENCE" HAS
      BECOME PREFERRED APPROACH
---------------------------------------------------------- Letter :3.2
Although NASA recently solicited proposals specifically for research
on Shuttle flights to Mir, NASA's life science office changed from
the widespread use of the select-for-flight approach in December
1993.  At that time, it solicited proposals for ground-based research
in space biology focused on the hypergravity effects that can be
induced by NASA's centrifuges.  NASA's shift to ground-based research
did not stifle competition for funding:  it received 650 responses to
the December 1993 announcement.  Although "select for science" is
relatively new to life science research, all microgravity research
announcements since 1990 have focused on research opportunities in
one or more science disciplines.\11 And, as if to emphasize the
independence of microgravity research from space station development,
NASA changed the fiscal year 1992 goals for the microgravity program. 
The previous goals referred to developing and using the space
station, whereas the current, more general goal is to "enable
[microgravity] research .  .  .  by choosing the carrier most
appropriate for the experiment."
Physical events, unlike biological processes, can be meaningfully
observed under the short-duration microgravity conditions afforded by
ground-based facilities, aircraft, and suborbital rockets. 
Consequently, a ground-based microgravity research investigator does
not always have to conduct experiments in a space environment, and
many do not.\12 For example, of the 51 principal investigators who
conducted such research at NASA's Lewis Research Center from fiscal
years 1989 through 1993, only 7 have been principal investigators on
space-based experiments, including a microgravity Spacelab flight in
September 1995. 
In June 1994, NASA's life and microgravity sciences advisory
committee stated that it supports the life and microgravity sciences
programs "in terms of their scientific contributions independent of
the type of flight platform." And the advisory committee specifically
recommended that
     "NASA establish a vigorous ground based research program
     focussing on gravitational biology in which centrifuge
     facilities at NASA centers are utilized for exploring science
     programs aimed at forces greater than 1g [Earth-normal
     gravity]."
--------------------
\11 A June 1991 solicitation for research in gravitational biology
was in the select-for-science tradition. 
\12 An advantage of a ground-based versus space-based research
program is that ground-based researchers can more readily repeat
their experiments even though the time an experiment is in a low
acceleration environment is comparatively brief.  For example, at
NASA's Lewis Research Center, over 1,000 drops per year were made in
the 2.2-second drop tower in 1989 and 1990 before operations were
reduced because the facility was modernized. 
      SELECT-FOR-SCIENCE APPROACH
      REASONABLE
---------------------------------------------------------- Letter :3.3
NASA's strategy for using the select-for-science approach to further
develop a life science research community in the station-era appears
reasonable based on the experience of the microgravity sciences
community. 
First, the microgravity sciences research community has been growing. 
Principal investigators funded for microgravity sciences research
increased by 120 percent--from 89 in calendar year 1989 to 196 in
fiscal year 1993.  The budget authority for microgravity sciences
increased by
130 percent, from $75.6 million to $173.9 million during this period. 
The number of proposals submitted in response to research
announcements also generally increased during this period.  For
example, although proposals submitted in response to materials,
fluids, and fundamental (benchmark) physics research announcements
decreased from 397 in 1991 to 217 in 1993, those responding to
combustion physics announcements increased from 65 in 1989 to 98 in
1993,
biotechnology research announcements increased from 94 in 1991 to 141
in 1994, and
materials and fluids research announcements increased from 69 in 1990
to 346 in 1991. 
Second, the microgravity research community is stable but not
stagnant.  Fifty-five percent of all microgravity sciences
investigators that were funded in 1989 were also funded in fiscal
year 1993.  This core group represents 25 percent of the
investigators funded in fiscal year 1993.  On the other hand, 44
percent of the investigators funded in 1993 were not funded in 1992. 
Third, NASA is attracting new investigators to its microgravity
sciences program.  The decline in proposals (from 397 to 217)
submitted in response to the 1993 materials, fluids, and fundamental
physics announcement may have been partly caused by NASA's stated
purpose of encouraging new investigators, and most of the 55
investigators funded from this announcement were new to the program. 
Only 15 of them had been previously funded by NASA.  In June 1994,
NASA's life sciences advisory subcommittee recommended that NASA use
this approach and establish appropriate categories within life
science research announcements that recognize and encourage new
investigators. 
Finally, for those proposals we reviewed, the select-for-science
approach produced relatively fewer low peer review scores than the
select-for-flight approach.\13 Figure 3 shows that 8 percent of the
select-for-science proposals received peer review scores in the
bottom category, while 32 percent of the select-for-flight proposals
received scores in the bottom category.\14
   Figure 3:  Research Proposals-
   Comparison of Selection
   Approaches
   (See figure in printed
   edition.)
\a Refer to selection approach. 
\b Categories based on peer review. 
Similarly, as figure 4 shows, of all the proposals in the bottom
category, only 16 percent were in the select-for-science tradition,
while 84 percent were in the select-for-flight tradition. 
   Figure 4:  Research Proposals-
   Comparison of Categories Based
   on Peer Review Scores
   (See figure in printed
   edition.)
\a Categories based on peer review. 
\b Refer to selection approach. 
Many of the proposals submitted in response to NASA's research
announcements were not considered scientifically meritorious.  For
example, peer review panels gave 129, or 44 percent, of the 290
proposals we reviewed relatively low scientific merit scores. 
--------------------
\13 We examined peer review scores of 290 proposals submitted in
response to 4 NASA research announcements issued in 1988, 1989, and
1991.  The U.S.  experiments on IML-2 were selected from these
proposals.  As such, they are not a random sample of all proposals
made from 1988 to 1991 and are not necessarily representative of
proposals made in other years. 
\14 The top, middle, and bottom score categories are described in
appendix II. 
      EXPECTED FUNDING WILL NOT
      SUPPORT NASA'S APPROACH
---------------------------------------------------------- Letter :3.4
NASA's plans to expand its ground-based research program are not
realistic based on planned funding.  A NASA microgravity research
official estimates that NASA will need to fund about 240 ground-based
investigators to support a station-based microgravity sciences
research program.  In fiscal year 1992, NASA funded 73 ground-based
investigators in microgravity sciences, only about 30 percent of the
future need.  Ground-based research is funded from NASA's research
and analysis budget.  However, NASA does not anticipate that this
budget will increase for fiscal years 1995 through 1999.  Annual life
science research and analysis appropriations are estimated to be
about $51 million and microgravity sciences at $21.7 million.  To
deal with this potential mismatch between plans and resources, NASA's
microgravity sciences office has proposed that the research and
analysis budget be augmented by research and development funds used
to support NASA's space-based research program.  The proposed amounts
are $4.7 million for fiscal year 1996, $12.2 million for 1997, and
$22.2 million for 1998.  According to NASA, shifting resources in
this way would not increase its overall budget authority. 
NASA's life and microgravity sciences advisory committee concurred
with this approach, stating in June 1994 that the research and
analysis budgets are the "seed which provides for a successful flight
program" and recommended that they be increased
     "where they are proven to be inadequate to support the
     intellectual underpinning of the flight program, even if this
     means a transfer from the [research and development] budget so
     as to comply with overall budget constraints."
   SELECTING QUALITY EXPERIMENTS
------------------------------------------------------------ Letter :4
NASA's quality assurance procedures start with a series of external
and internal reviews designed to evaluate the merits of research
proposals.  Peer review is a crucial part of this consensus-building
process.  The process starts with individual reviewers independently
evaluating each proposal assigned to peer review panels.  The
reviewers then resolve any differences by consensus within the peer
review panel.  The panel's final determinations are not binding on
NASA's selection officials, and NASA can choose proposals other than
those highly recommended by the panel. 
In June 1994, we reported that the peer review processes at the
National Institutes of Health (NIH), National Science Foundation, and
National Endowment for the Humanities appear to be working well and
that intrinsic qualities of a proposal (e.g., research design), and
not characteristics of reviewers or applicants (e.g., applicant's
region, academic rank, or employing academic department's prestige)
were important factors in reviewers scoring.\15
In 1993, the Senate Committee on Appropriations directed NASA to
model its peer review standards after NIH.  Based on the Committee's
direction, NASA requires that
all research proposals be reviewed by peers for scientific merit and
relevance (previously, some life science research conducted by NASA
scientists was not subject to peer review);
all research be reviewed by peers at least every 3 years;
all research be reviewed for progress annually and for the
performance of its objectives at least every 3 years;
peer review be performed by the best-qualified individuals available
in the field reviewed; and
peer review scores provided by external peer review groups be
critical factors determining the priority for initial and continued
funding of research projects and programs. 
--------------------
\15 Peer Review:  Reforms Needed to Ensure Fairness in Federal Agency
Grant Selection (GAO/PEMD-94-1, June 24, 1994). 
      INDIVIDUAL PEER REVIEWERS'
      SCORES ARE USUALLY
      CONSISTENT
---------------------------------------------------------- Letter :4.1
The logic of peer review, in our opinion, rests, in part, on the
assumption that two or more peers can independently agree on a
research experiment's scientific merits.  For example, they should
agree on the testability of the proposed hypothesis and the relevance
and appropriateness of the experimental design.  As such, peers'
scores for scientific merit of any given proposal ought to be the
same or similar.\16
Peers agreed on the scientific merit of 73 percent of the proposals
that we reviewed, including all but 1 of the 15 U.S.  experiments
selected for IML-2.  Table 1 shows the distribution of the reviewers'
scores.  Peer reviewers were better able to agree on proposals having
top scientific merit scores than on proposals having middle or bottom
scientific merit scores.  Peers gave only 11 percent (11 of 99) of
the top proposals dissimilar scores; in contrast, they gave 35
percent (44 of 126) of proposals dissimilar scores in the middle
category and 38 percent (23 of 60) scores in the bottom category. 
                                     Table 1
                      Distribution of Dissimilar Peer Review
                           Scores for Scientific Merit
                                  Microgravi
                                          ty
                                  fundamenta  Microgravity  Microgravi
Peer review score       sciences   l science  biotechnolog   ty flight
category\a              1989            1991        y 1991        1988     Total
----------------------  --------  ----------  ------------  ----------  --------
Top                     23 (1)        26 (2)        33 (4)      17 (4)   99 (11)
Middle                  33 (7)        15 (6)       47 (17)     31 (14)  126 (44)
Bottom                  11 (3)         2 (2)         8 (2)     39 (16)   60 (23)
================================================================================
Total                   67 (11)      43 (10)       88 (23)     87 (34)       285
                                                                          (78)\b
--------------------------------------------------------------------------------
\a Top, middle, and bottom categories and similar/dissimilar scores
are described in appendix II. 
\b Investigators submitted 319 proposals in response to these
announcements, but 29 proposals were considered unresponsive to the
announcements' objectives.  Some of the unresponsive proposals
received panel scores and some did not.  To be consistent, we did not
include any of them in this table.  Individual peer review scores
could not be located for five other life science-related proposals
and, thus, could not be included in the table. 
--------------------
\16 Scientific merit was defined in a variety of ways for the
proposals we reviewed.  For the life science proposals, scientific
merit included the experiment's hypothesis, experimental design,
significance, investigator's professional experience, and adequacy of
facilities.  For the microgravity science proposals, criteria such as
relevance and institutional resources were separately rated from
scientific merit. 
      NASA SELECTED PROPOSALS
      DESPITE PEER REVIEW CONCERNS
---------------------------------------------------------- Letter :4.2
Table 2 shows that NASA's selecting officials' funding decisions were
generally congruent with the findings of the peer review panel.  Of
the
84 proposals funded, 73, or 87 percent, were in the top category for
scientific merit scores, and the other 11 proposals funded were in
the middle category. 
                                     Table 2
                      Distribution of NASA Funding Decisions
                                  Microgravi
                                          ty
                                  fundamenta  Microgravity  Microgravi
Peer review score       sciences   l science  biotechnolog   ty flight
category                1989            1991        y 1991        1988     Total
----------------------  --------  ----------  ------------  ----------  --------
Top                     25 (18)      26 (17)       33 (22)     17 (16)   101(73)
Middle                  35 (2)        15 (0)        47 (2)      31 (7)  128 (11)
Bottom                  12 (0)         2 (0)         8 (0)      39 (0)    61 (0)
================================================================================
Total                   72 (20)      43 (17)       88 (24)     87 (23)  290 (84)
--------------------------------------------------------------------------------
Determinations of peer review panels are not binding on NASA's
selection officials.  For example, NASA selected four proposals that
received mid-level scores by the peer review panel.\17 Based on an
average of peers' individual scores, three of them would have been in
the top category.  However, in subsequent deliberations, the peer
review panel members placed three proposals in the middle category
because the
need for the microgravity environment of space was not compelling,
experiment-related issues could be resolved using the ground-based
program, and
appropriateness of analytical techniques was questionable. 
No peer review panel was convened for the fourth proposal because the
number of proposals in the specific area of investigation was too
small.  Generally, the peer reviewers found the proposal to be of
high quality, but they also noted that the research objectives,
although compatible with the life science program, were inconsistent
with the microgravity science program.  In this case, the
investigator did not propose to use microgravity to study phenomena
whose understanding is obscured on earth by the presence of gravity. 
After the peer review panel completed its deliberations, a NASA
categorization committee made category assignments that were
forwarded to a steering committee.  The categorization committee
determined that the four proposals were, in the words of a NASA
official, "sound but not exceptional science"--the second highest of
four possible categories.\18 The steering committee assessed these
categorizations and recommended funding the proposals, but committee
members noted that one investigation resembled a "fishing
expedition," another had "similar weaknesses" to proposals that were
rejected, a third would require too much time to conduct on a
Spacelab mission, and a fourth should only be partly funded.  Despite
these views, these four proposals were funded for IML-2. 
--------------------
\17 The four proposals represent 36 percent of the U.S.  microgravity
experiments flown on IML-2. 
\18 Categorizations are formally described in the Federal Acquisition
Regulation (48 C.F.R.  1870.103).  Informally, the highest category
means "best science, with no technical risk." Proposals in categories
I and II are "recommended for acceptance" according to this part of
the regulation.  Categorizations apply to proposals submitted in
response to an "Announcement of Opportunity" to participate in
specific NASA programs such as Space Shuttle/Spacelab flights (48
C.F.R.  1870.1), and not to a "NASA Research Announcement" used to
solicit proposals in areas that NASA has special research interests
(48 C.F.R.  1870.2). 
   ASSESSING THE IMPACT OF
   CANCELED SPACELAB FLIGHT ON THE
   RESEARCH COMMUNITY
------------------------------------------------------------ Letter :5
NASA's efforts to develop a research community are not likely to be
adversely affected by the February 1994 cancellation of SLS-3.  The
U.S.  principal investigators on the Spacelab flight stated that they
will be able to meet their experiment objectives on other missions,
including multiple Shuttle flights to Mir.  They plan to submit
proposals for future NASA research opportunities. 
      NASA ADVISORY COMMITTEE
      CONCERNED ABOUT CANCELLATION
      OF SPACELAB PROGRAM AND
      USEFULNESS OF RUSSIAN SPACE
      STATION
---------------------------------------------------------- Letter :5.1
NASA planned to fly a collaborative U.S.-French SLS-3 mission in
February 1996.  The purpose of the mission was to study the effects
of microgravity on the musculoskeletal system of humans, Rhesus
monkeys, and rats.  The French were responsible for developing the
Rhesus research facility.  Planning for the mission began in the late
1970s.  On February 18, 1994, however, the NASA Administrator
notified his French counterpart that the flight was canceled because
of budget limitations and NASA's commitment to the international
space station.  In November 1993, as part of the agreement between
NASA and the Russian Space Agency to bring Russia into the space
station program, the United States and Russia agreed to fly up to 10
Space Shuttle flights to the Russian space station Mir. 
This agreement raised the concern that multiple Shuttle flights to
Mir would displace future Spacelab flights to the detriment of NASA's
life and microgravity sciences program.  Eventually, these concerns
were formally expressed by the Space Studies Board of the National
Research Council in a February 25, 1994, letter to the NASA
Administrator.  The Board discussed broad issues of science selection
and management in the space station program, including termination of
Spacelab missions because these laboratories "can provide more
high-quality science than can Mir .  .  .  ." The Board also noted
that
     "cancellation of [a future life science Spacelab] mission or
     substitution of [Shuttle] middeck experiments for a dedicated
     Spacelab mission would have serious consequences for .  .  . 
     the continued participation of the mainstream life sciences
     community that NASA seeks to attract."
NASA responded to these concerns in April 1994, stating that all
experiments from SLS-3 have been accommodated on other missions,
including Shuttle flights to Mir.  NASA also noted that although Mir
is not a substitute for Spacelab, it will augment and enhance
on-orbit science capabilities because experiments requiring more than
30 days of microgravity cannot be performed on Spacelab. 
In late 1994, NASA announced a new life and microgravity sciences
spacelab mission for July or August 1996.  This mission will provide
a flight opportunity for some experiments that were scheduled to fly
on SLS-3. 
      U.S.  INVESTIGATORS NOT
      ADVERSELY AFFECTED BY
      SPACELAB CANCELLATION
---------------------------------------------------------- Letter :5.2
We discussed the cancellation of SLS-3 with 13 of the 15 U.S. 
investigators who were scheduled to fly experiments on this
flight.\19 Their views are summarized, as follows: 
Nine investigators were generally satisfied with the way NASA handled
the cancellation.  NASA never formally notified investigators about
its decision to cancel the flight.  Consequently, most investigators
learned of the cancellation from rumors or other informal
communication.  One investigator said investigators should have been
consulted before NASA canceled the mission.  One investigator
questioned why SLS-3, a relatively near-term mission, was canceled
rather than a later one such as the Neurolab flight in 1998. 
Eleven investigators said that their experiments will be accommodated
on other missions, including the Russian Biosatellite, another
Spacelab mission, or Space Shuttle flights to Mir.\20 Two
investigators said they have not been assigned to specific missions. 
Ten investigators currently scheduled for other missions said they
will be able to meet their basic experiment objectives.  However,
three of them said they will not necessarily be able to obtain the
same amount of information as they would have on SLS-3.  Their
experiments involved the use of Rhesus monkeys, and even though they
will fly on the Biosatellite, in-flight biological measurements
cannot be done.  Three other investigators said that their
experiments on a substitute mission would be adversely affected by
hardware limitations or the loss of opportunities to efficiently
collaborate with other investigators. 
All 13 investigators said they will continue to submit proposals for
future NASA research opportunities, and at least 6 have already done
so. 
--------------------
\19 Two investigators did not reply to our inquiries. 
\20 The Biosatellite is part of the Russian Cosmos series of
satellites.  Collaboration between U.S.  and Russian scientists on
Biosatellite-related experiments dates from 1971 and included eight
flights starting in 1975.  U.S.  investigators started flying primate
experiments on this satellite in 1983. 
   SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :6
To accomplish our objectives, we obtained documents from and
interviewed officials at NASA headquarters in Washington, D.C., and
at NASA's Lewis, Johnson, and Marshall field centers in Cleveland,
Ohio; Houston, Texas; and Huntsville, Alabama, respectively.  In May
1994, we attended the IML-2 mission simulation and science review
conference and observed crew training exercises prior to launch. 
To review the further development of NASA's life and microgravity
sciences research community, we obtained information on research
announcements issued between 1988 and 1994 and on the principal
investigators who conducted ground- and space-based experiments.  We
examined peer review-related information on 319 proposals submitted
in response to the 4 research announcements related to IML-2.  We
categorized the scores of all proposals that peer review panels
considered responsive to the objectives of the announcements, as
shown in appendix II. 
To assess the possible impact of the cancellation of the SLS-3
Spacelab flight on the further development of NASA's research
community, we interviewed 13 of the 15 U.S.  principal investigators
on that mission. 
We performed our work between November 1993 and September 1994 in
accordance with generally accepted government auditing standards.  As
requested, we did not obtain agency comments on this report. 
However, we discussed the issues in this report with NASA officials
and incorporated their comments where appropriate. 
---------------------------------------------------------- Letter :6.1
As agreed with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 15 days
from its issue date.  At that time, we will send copies to the NASA
Administrator and other appropriate congressional committees.  Copies
will also be made available to other interested parties on request. 
Please contact me on (202) 512-8412 if you or your staff have any
questions concerning this report.  Major contributors to this report
are listed in appendix III. 
Sincerely yours,
Donna M.  Heivilin
Director, Defense Management
 and NASA Issues
EXPERIMENT QUALITY ASSURANCE
PROCEDURES
=========================================================== Appendix I
The National Aeronautics and Space Administration's (NASA) guiding
principle for quality assurance is periodic review over the lifetime
of an experiment.  Figure I.1 depicts the major science and
engineering review milestones.\1
   Figure I.1:  NASA Flight
   Experiment Definition and
   Development Process
   (See figure in printed
   edition.)
The steps or actions involved throughout this process are outlined
below. 
--------------------
\1 The figure illustrates a generic process, but it most closely
matches the one used for microgravity-related experiments. 
      PREPARING RESEARCH
      ANNOUNCEMENTS AND SOLICITING
      RESEARCH PROPOSALS
------------------------------------------------------- Appendix I:0.1
NASA's Discipline Working Group(s) evaluates research program's
strengths and weaknesses and makes recommendations to the program
scientist, who defines areas of investigation for forthcoming
announcement.\2
NASA conducts workshops for prospective investigators from the
scientific community to develop interest in forthcoming announcement. 
NASA solicits research proposals by issuing announcement. 
--------------------
\2 The program scientist is also responsible for defining a mission's
science and application objectives and ensuring that these objectives
are met. 
      ASSESSING RESEARCH PROPOSALS
------------------------------------------------------- Appendix I:0.2
Peers are selected by contractor\3 (life sciences) or NASA
(microgravity sciences) to evaluate proposals' scientific merit. 
Peers should be leading researchers in their field and free from
conflicts of interest (e.g., a current or recent professional
collaboration with an applicant), and not currently receiving
research funds from NASA. 
For proposals receiving strong science reviews, the appropriate NASA
field center assesses a proposal's estimated cost and engineering
feasibility.  For example, Lewis Research Center is a "center of
excellence" for two microgravity science disciplines:  combustion
science and fluid physics. 
NASA program scientist recommends principal investigators'\4
proposals for funding to senior NASA management. 
--------------------
\3 Currently, the American Institute for Biological Science convenes
peer review panels. 
\4 The principal investigator conceives an investigation and is
responsible for carrying it out and reporting the results. 
      REASSESSING EXPERIMENTS
------------------------------------------------------- Appendix I:0.3
         SCIENCE CONCEPT REVIEW
----------------------------------------------------- Appendix I:0.3.1
Principal investigator and project scientist\5 describe science scope
and feasibility for evaluation by Science Review Board. 
Project manager\6 describes conceptual design of experiment-related
hardware for evaluation by Engineering Panel. 
--------------------
\5 The project scientist is responsible for ensuring that (1) the
principal investigator adequately defines investigation's science
requirements and (2) needed experiment-related hardware will
accommodate required science. 
\6 The project manager is responsible for the design, development,
fabrication, and test of an experiment. 
         REQUIREMENTS DEFINITION
         REVIEW
----------------------------------------------------- Appendix I:0.3.2
Project manager describes cost and schedule estimates. 
Engineering panel assesses design of hardware. 
Science panel assesses compatibility of science requirements with
design of hardware. 
      DEVELOPING FLIGHT
      EXPERIMENTS
------------------------------------------------------- Appendix I:0.4
Preliminary design review assesses the compatibility of science
requirements with a preliminary engineering model ("breadboard") of
hardware. 
Critical design review assesses complete engineering model of
hardware. 
Preshipment review consists of experiment simulations, integration
with hardware, and testing prior to sending the hardware to the
launch site. 
PEER REVIEW PANEL SCORES FOR
SELECTED PROJECTS
========================================================== Appendix II
To determine the similarity/dissimilarity of peers' perception of a
proposal's scientific merit, we defined similar scores on the
five-point scales as same or adjacent scores (for example:  "3" and
"3", or "3" and "4"); and on the nine-point scales as same, adjacent,
and next scores (for example:  a "3" and "3", or "3" and "4"; or "3",
"4", and "5"). 
                          Table II.1
           Peer Review Panel Scores by Announcement
                    and Relative Category.
Announcement                         Top    Middle    Bottom
------------------------------  --------  --------  --------
Life sciences, 1989\a              1.0 -     2.1 -     4.0 -
                                     2.0       3.9       5.0
Microgravity fundamental           9.0 -     6.9 -     3.9 -
 science, 1991\b                     7.0       4.0       1.0
Microgravity biotechnology,        9.0 -     6.9 -     3.9 -
 1991\b                              7.0       4.0       1.0
Microgravity flight, 1988\c        5.0 -     3.9 -     2.0 -
                                     4.0       2.1       1.0
------------------------------------------------------------
\a Proposals receiving scores greater than 2.5 were not further
considered by NASA. 
\b Proposals receiving scores less than 7.0 were not further
considered by NASA. 
\c Proposals receiving scores less than 3.0 were not further
considered by NASA. 
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================= Appendix III
NATIONAL SECURITY AND
INTERNATIONAL AFFAIRS DIVISION,
WASHINGTON, D.C. 
Dave Warren
Frank Degnan
Thomas Mills
James Berry
Kimberly Carson
Jeffrey Knott
Larry Kiser



NEWSLETTER
Join the GlobalSecurity.org mailing list