Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes (Chapter Report, 07/30/1999, GAO/NSIAD-99-162)
The Pentagon plans to boost its investment in new weapons to about $60
billion in fiscal year 2001--a 40-percent increase over fiscal year
1997. The military has high expectations for this investment: that new
weapons will be better and less expensive than their predecessors and
will be developed in half the time. However, the Defense Department
(DOD) will not meet these expectations using its traditional management
approach. Leading commercial firms have changed their practices for
developing products and have achieved the kinds of results DOD seeks.
Maturing new technology before it is included in products is one of the
main determinants of these firms' successes. This practice holds promise
for DOD, for immature technologies have been a main source of problems
on weapon systems. This report assesses (1) the impact of technology
maturity on product outcomes, (2) best practices for managing new
technologies and incorporating them into products, and (3) ways DOD can
adapt these practices to get better outcomes on weapon system programs.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: NSIAD-99-162
TITLE: Best Practices: Better Management of Technology
Development Can Improve Weapon System Outcomes
DATE: 07/30/1999
SUBJECT: Private sector practices
Defense procurement
Comparative analysis
Defense capabilities
Weapons research and development
Weapons systems
Testing
IDENTIFIER: Comanche Helicopter
DOD/NASA Integrated High Performance Turbine Engine
Technology
DOD Defense Acquisition Pilot Program
Defense Reform Initiative
Brilliant Anti-Armor Submunition
DOD Airborne Laser Program
DOD Advanced Amphibious Assault Vehicle Program
Forward Looking Infrared System
HS 702 Satellite
Seawolf Attack Submarine
Army Future Scout and Cavalry System
F-15 Aircraft
F-117 Aircraft
F-22 Aircraft
DOD Advanced Concept Technology Demonstration Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. This text was extracted from a PDF file. **
** Delineations within the text indicating chapter titles, **
** headings, and bullets have not been preserved, and in some **
** cases heading text has been incorrectly merged into **
** body text in the adjacent column. Graphic images have **
** not been reproduced, but figure captions are included. **
** Tables are included, but column deliniations have not been **
** preserved. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** <info@www.gao.gov> **
** **
** with the message 'info' in the body. **
******************************************************************
United States General Accounting Office GAO Report
to the Chairman and Ranking Minority Member, Subcommittee on
Readiness and Management Support, Committee on Armed Services,
U.S. Senate July 1999 BEST PRACTICES Better Management
of Technology Development Can Improve Weapon System Outcomes
GAO/NSIAD-99-162 United States General Accounting Office
National Security and Washington, D.C. 20548
International Affairs Division B-280233
Letter July 30, 1999 The Honorable James Inhofe Chairman The
Honorable Charles Robb Ranking Minority Member Subcommittee on
Readiness and Management Support Committee on Armed Services
United States Senate As you requested, this report assesses how
best practices offer improvements to the way the Department of
Defense (DOD) incorporates new technology into weapon system
programs. It also assesses the factors that can make it difficult
to mature technologies before they are included on weapon system
programs and what can be done about them. We make recommendations
to the Secretary of Defense on how advanced technologies can be
better managed so they pose less risk when they are included in
weapon system designs. We are sending copies of this report to the
Honorable William S. Cohen, Secretary of Defense; the Honorable
Louis Caldera, Secretary of the Army; the Honorable Richard
Danzig, Secretary of the Navy; the Honorable F. Whitten Peters,
Acting Secretary of the Air Force; the Honorable Jacob J. Lew,
Director, Office of Management and Budget; and to interested
congressional committees. We will also make copies available to
others upon request. If you have any questions regarding this
report, please call me at (202) 512-4841. Other key contacts are
listed in appendix III. Katherine V. Schinasi Associate Director
Defense Acquisitions Issues Letter Executive Summary Purpose
The Department of Defense (DOD) plans to increase its investment
in new weapons to about $60 billion in fiscal year 2001-a 40-
percent increase over fiscal year 1997. DOD has high expectations
from this investment: that new weapons will be better and less
expensive than their predecessors and will be developed in half
the time. With its traditional management approach-which has
produced superior weapons, but at much greater cost and time than
planned-DOD will not meet these expectations. Leading commercial
firms have changed their practices for developing products and
have achieved the kinds of results DOD seeks. Maturing new
technology before it is included in products is one of the main
determinants of these firms' successes. This practice holds
promise for DOD, for immature technologies have been a main source
of problems on weapon systems. In response to a request from the
Chairman and the Ranking Minority Member, Subcommittee on
Readiness and Management Support, Senate Committee on Armed
Services, GAO assessed (1) the impact of technology maturity on
product outcomes, (2) best practices for managing new technologies
and incorporating them into products, and (3) ways DOD can adapt
these practices to get better outcomes on weapon system programs.
Background GAO reviewed commercial and DOD
experiences in incorporating 23 different technologies into new
product and weapon system designs. The technologies were drawn
from (1) six commercial firms recognized for their success in
developing technically advanced products more quickly than the
products' predecessors and (2) five DOD weapon system programs
that incorporated advanced technologies, including some that did
not encounter problems and some that did. GAO asked the managers
of these technologies to assess the maturity of the technologies
at the point they were included in product development by applying
a tool, referred to as technology readiness levels (TRLs). The
National Aeronautics and Space Administration and the Air Force
Research Laboratory use TRLs to determine the readiness of
technologies to be incorporated into a weapon or another type of
system. Readiness levels are measured along a scale of one to
nine, starting with paper studies of the basic concept, proceeding
with laboratory demonstrations, and ending with a technology that
has proven itself on the intended product. The Air Force Research
Laboratory considers TRL 6 an acceptable risk for a weapon system
entering the program definition stage, the point at which DOD
launches its weapon programs, and TRL 7 an acceptable risk for the
Letter Page 2
GAO/NSIAD-99-162 Best Practices Executive Summary engineering and
manufacturing development stage. This is an important distinction
because leading commercial firms launch a new product later than
DOD, after technology development is complete. They refer to this
point as the beginning of product development, the point at which
they commit to developing and manufacturing the product.
Typically, technology is still being developed when weapon system
programs are launched; the point at which a weapon system is far
enough along to compare to a commercial product development is
likely to be at or after the start of engineering and
manufacturing development. Results in Brief The experiences
of DOD and commercial technology development cases GAO reviewed
indicate that demonstrating a high level of maturity before new
technologies are incorporated into product development programs
puts those programs in a better position to succeed. The TRLs, as
applied to the 23 technologies, reconciled the different maturity
levels with subsequent product development experiences. They also
revealed when gaps occurred between a technology's maturity and
the intended product's requirements. For technologies that were
successfully incorporated into a product, the gap was recognized
and closed before product development began, improving the chances
for successful cost and schedule outcomes. The closing of the gap
was a managed result. It is a rare program that can proceed with a
gap between product requirements and the maturity of key
technologies and still be delivered on time and within costs. Two
conditions were critical to closing the maturity gap. First, the
right environment for maturing technologies existed. Key to this
environment was making a science and technology organization,
rather than the program or product development manager,
responsible for maturing technologies to a high TRL. When a
maturity gap persisted, managers were given the flexibility to
take the time to mature the technology or decrease product
requirements so that they could use another, already mature
technology. Second, both technology and product managers were
supported with the disciplined processes, readily available
information, readiness standards, and authority to ensure
technology was ready for products. This support enabled these
managers to safeguard product development from undue technology
risks. On the other hand, immature technologies were sometimes
incorporated into products for reasons such as inflexible
performance requirements, increasing the likelihood of cost
overruns and delays in product development. Product managers had
little choice but to accept the technologies and hope that they
would mature Letter Page 3
GAO/NSIAD-99-162 Best Practices Executive Summary successfully.
However, the pressures of product development made for an
environment less conducive to maturing technology. For several
reasons, DOD is likely to move technologies to product development
programs before they are mature. Science and technology
organizations, which traditionally operate within fixed budget
levels, do not necessarily have the funds to mature technology to
the higher TRLs. Programs are more able to command the large
budgets necessary for reaching these levels. The pressures exerted
on new programs to offer unique performance at low cost encourage
acceptance of unproven technologies. The technologies GAO reviewed
indicate these conditions can be overcome on individual cases. DOD
has several initiatives underway, such as advanced technology
demonstrations, that could make it more feasible for science and
technology organizations to mature technology before it is moved
to product development programs. The challenge will be whether the
lessons learned from these cases and initiatives offer an approach
that has a DOD-wide application. GAO makes recommendations to the
Secretary of Defense on ways to pursue advanced technologies while
lessening their potential for causing problems on weapon system
programs. Principal Findings Maturity of Technology at The 23
technologies GAO reviewed spanned a wide range of readiness
Program Start Is an levels-from a low of TRL 2 to a high
of TRL 9-when they were included Important Determinant of in
product development programs. Programs with key technologies at
Success readiness levels 6 to 8 at the time
of program launch met or were meeting cost, schedule, and
performance requirements. All of the commercial technologies and a
few of the DOD technologies fell into this category. For example,
Ford managed its voice-activated control technology to TRL 8-a 10-
year effort-before introducing it on the 1999 Jaguar. Similarly,
the Defense Advanced Research Projects Agency matured a
revolutionary periscope technology to TRL 9 before it was included
on the Virginia class attack submarine. DOD programs that accepted
technologies at a readiness level of 5 or less experienced
significant cost and schedule increases due, in part, to problems
with the technologies. DOD's acceptance of technologies at level 4
or lower was not unusual. For example, the key technologies for
the Army's brilliant antiarmor submunition were at levels 2 Page 4
GAO/NSIAD-99-162 Best Practices Executive Summary and 3 when
weapon system development began. At these levels, DOD had a
significant gap in technology maturity at the start of the
program. The gap was not closed until well into the development
program, and problems with the technologies were a main
contributor to the program's 88-percent cost growth and 62-percent
slip in schedule. Controllable Conditions Closing the
technology development gap before beginning product Affect How
Well a development was the result of good
technology maturation practices and Technology's Inclusion on a
sound methods for moving technologies to products. The more
successful Product Can Be Managed of the 23
technologies were managed by science and technology organizations
until they reached at least TRL 6 and more, often TRL 8 or higher.
This environment was an important condition for successfully
maturing technologies, as it allowed room for unexpected results
such as test "failures," which are considered normal events in
developing technologies. To match technology maturity and product
requirements, managers also had the option of waiting until
technologies matured or changing product requirements so that an
already mature technology could be used. For example, Hughes
deferred the development of the HS-702 satellite until critical
solar cell technology had matured-a process that took over 10
years. Also, Navy managers accepted an existing weapon ejection
system on the Virginia class attack submarine when technology
failed to mature as expected. In contrast, performance
requirements for the Comanche helicopter were inflexible;
requirements mandated the inclusion of advanced sensors and
avionics technologies, despite their immaturity. The Comanche
program has experienced cost growth and schedule delays, partly
attributable to the inclusion of these technologies. In the more
successful cases, technology and product managers were given the
authority and tools to move technology only when it was at high
readiness levels. Disciplined processes provided managers credible
information on the status of technologies and high standards for
assessing readiness. Science and technology managers developed
technologies to standards acceptable to product managers who could
reject those technologies that fell short. For example, Ford's
science and technology managers use agreed-upon standards for
judging technology readiness, and all new technologies follow the
same maturation process. Ford's product managers are also
empowered to say no when technologies are not deemed mature.
Recently, the Jaguar vehicle team rejected night vision technology
at TRL 8 because it did not meet cost objectives. DOD program
managers that had to accept immature technologies had less
information available to guide them. For example, key technologies
for the brilliant antiarmor Page 5
GAO/NSIAD-99-162 Best Practices Executive Summary submunition
program bypassed Army science and technology organizations,
forcing the program manager to accept the technologies with little
information about their readiness. Often, the tools used to assess
the technologies' status failed to identify high risks; the TRLs
indicate that risks on the problematic technologies were often
high. Also, the greater pressures to meet cost and schedule goals
in product development provided a less forgiving environment for
fledgling technologies. Impediments to Adopting Leading
commercial firms have put the organizations, tools, and other Best
Practices for practices in place to foster technology
development and improve the Technology Inclusion in outcomes
of product developments as a matter of necessity. The large DOD
Are Surmountable investment required for a new product and
the risks to that investment if the product does not meet customer
needs reinforce these practices. The DOD cases that followed a
similar approach-the Advanced Amphibious Assault Vehicle and the
Virginia class attack submarine-have so far avoided problems with
key technologies. Yet these cases are not the norm for DOD
programs. DOD programs operate under conditions that make it more
difficult-and less rewarding-to separate technology from product
development and to allow technology to reach high maturity before
being included in an acquisition program. It is easier for weapon
system programs to fund technology development at higher readiness
levels because they attract much bigger budgets than science and
technology projects. DOD typically does not fund science and
technology organizations to take technology past the feasibility
stage- TRL 5. As a practical matter, it is often necessary to move
immature technology to a weapon system program to get needed funds
and management support. New programs are pressured to include
immature technologies that offer significant performance gains.
These pressures come from the user's perception of the threat,
technologists that see the program as an opportunity to apply a
new technology, and funding competition that rewards weapon
systems with unique features. DOD and the services have several
initiatives for improving the technology development process and
reducing weapon system cycle times. These include defense
technology objectives, advanced technology demonstrations,
advanced concept technology demonstrations, the Army's new
scout/cavalry vehicle, and the Air Force's Integrated High
Performance Turbine Engine Technology Program. These initiatives
are aimed at putting the science and technology organizations and
funding in Page 6
GAO/NSIAD-99-162 Best Practices Executive Summary place to bring
technologies to higher readiness levels before they are included
in weapon system programs. Recommendations GAO recommends that
the Secretary of Defense adopt a disciplined and knowledge-based
approach of assessing technology maturity, such as TRLs, DOD-wide,
and establish the point at which a match is achieved between key
technologies and weapon system requirements as the proper point
for committing to the development and production of a weapon
system. GAO also recommends that the Secretary (1) require that
technologies needed to meet a weapon's requirements reach a high
readiness level (analogous to TRL 7) before making that
commitment, (2) extract lessons from successful technology
inclusion cases for application to future technology inclusion
efforts, and (3) empower program managers to refuse to accept key
technologies with low levels of maturity by making decisions on
individual programs that reinforce a best practice approach to
technology maturation and inclusion. These recommendations appear
in full in chapter 5. Agency Comments DOD generally agreed with
the report and its recommendations. A detailed discussion of DOD's
comments appear in appendix I. Page 7
GAO/NSIAD-99-162 Best Practices Contents Letter Executive Summary
2 Chapter 1 Separating Technology Development
From Product Development Introduction Is a Best
Practice 12
Technology and Product Development Conducted at the Same Time
Within DOD
15 Shorter Acquisition Cycle Times Are Needed
17 Objectives, Scope, and Methodology
18 Chapter 2 Technology Maturity Can Be Measured
and Its Consequences for Maturity of Technology Products Can
Be Forecast 22
Technologies With High Readiness Levels at Launch Were Better at
Program Start Is an Able to Meet Product Objectives
25 Important Determinant of Success Chapter 3
Providing the Right Environment Is Critical to the Successful
Controllable Maturation of Technology
35 Good Technology Handoff Decisions Depend on the Tools and
Conditions Affect How Authority Given to Managers
41 Well a Technology's Inclusion on a Product Can Be Managed
Chapter 4 Several Factors Make It Difficult to
Mature Technologies Before Impediments to They Are
Included on Weapon Systems 50
Services Encouraged to Use Best Practices
54 Adopting Best Two Unique DOD Projects May Provide
Lessons on How to Enable Practices for S&T
Organizations to Manage Technology Further
56 Technology Inclusion in DOD Are Surmountable Page 8
GAO/NSIAD-99-162 Best Practices Contents Chapter 5
Conclusions 61 Conclusions and Recommendations 63 Agency
Comments and Our Evaluation
65 Recommendations Appendixes Appendix I:
Technology Readiness Levels and Their Definitions 68
Appendix II: Comments From the Department of Defense
69 Appendix III: GAO Contacts and Staff Acknowledgments
73 Related GAO Products
76 Tables Table 2.1: Cost and Schedule
Experiences on Product Developments
27 Table 3.1: TRLs of Technologies Managed by S&T Organizations
36 Page 9 GAO/NSIAD-99-162
Best Practices Contents Figures Figure 1.1: Cycle for Providing
Users a Product With Better Capabilities
13 Figure 1.2: DOD's Weapon System Acquisition Cycle
16 Figure 1.3: Allocation of DOD's Fiscal Year 1999 Research and
Development Funds
17 Figure 2.1: Using TRLs to Match Technology With Product Launch
Requirements
24 Figure 2.2: Readiness Levels of Technologies at the Time They
Were Included in Product Designs
26 Figure 2.3: Time Line for Ford's Development of Voice Activated
Controls Technology
28 Figure 2.4: Jaguar
29 Figure 2.5: Virginia Class Attack Submarine
30 Figure 2.6: Brilliant Anti-Armor Submunition
32 Figure 2.7: Comanche Helicopter
33 Figure 3.1: Hughes Solar Cell Arrays
38 Figure 3.2: Integrated Avionics for Comanche Helicopter
40 Figure 3.3: Process for Closing the Gap Between the Readiness
of Adaptive Cruise Control Technology and Jaguar Requirements
43 Figure 3.4: Process for Closing the Gap Between the Readiness
of Propulsion Technologies and AAAV Requirements
46 Figure 3.5: AAAV
47 Figure 3.6: Assimilation of New Technology Into the BAT Program
48 Figure 4.1: Airborne Laser
52 Figure 4.2: Comparison of Traditional Technology Development
Process With the Army's Fast Track Approach
57 Figure 4.3: Future Scout and Cavalry System
59 Page 10 GAO/NSIAD-99-
162 Best Practices Abbreviations AAAV Advanced Amphibious
Assault Vehicle ABL Airborne Laser ACTD Advanced
Concept Technology Demonstration ATD Advanced Technology
Demonstration BAT Brilliant Anti-Armor Subminition DARPA
Defense Advanced Research Projects Agency DEAL Deliverables
Agreement Log DOD Department of Defense DTO defense
technology objective FLIR forward-looking infrared NASA
National Aeronautics and Space Administration S&T science
and technology TRL technology readiness level Page 11
GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction
Chapter 1 A central piece of the National Military Strategy is the
military capability represented by advanced weaponry. The
Department of Defense (DOD) plans to increase its annual
investment in new weapons to about $60 billion by fiscal year
2001-a 40-percent increase over fiscal year 1997. DOD has high
expectations from this investment: that new weapons will be better
and less expensive than their predecessors and will be developed
in half the time. These expectations frame a great challenge for
managers of programs. The traditional management approach-which
has produced superior weapons but at much greater cost and time
than planned-will not meet these expectations. Cycle times-the
time to develop a new weapon-can be so long that the technology a
weapon is designed with becomes obsolete before it can be
produced. Costs of new weapons have reached the point that
significantly fewer can be bought than planned. These are not new
issues, but they have become more pressing as the pace and
sophistication of foreign and commercial technology have
increased, complicating a national security environment of unknown
threats. Leading commercial firms have changed the way they
develop products and have achieved the kinds of results DOD seeks,
often yielding more sophisticated products in half the time
formerly needed. Industry experts estimate that resolving
technology problems before product development begins results in
10 times the savings compared to correcting problems afterward. In
this sense, technology maturity breeds product success. The
practices leading firms use to mature and transition technology to
products hold promise for DOD, for immature technologies have been
main sources of problems on weapon systems. We have previously
reported on the different elements of knowledge firms insist on to
get better products to market faster. Of these, no element is more
important than having technology, advanced enough to meet
requirements but also mature enough to be predictably managed,
available at the start of the product development cycle. Maturing
new technology before it is included on a product is perhaps the
most important determinant of the success of the eventual product-
or weapon system. It is the topic of this report. Separating
Technology The cycle for placing better capabilities in the hands
of users-both Development From military and
commercial-can be described as consisting of technology Product
Development Is a Best Practice Page 12
GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction
development, product development, and production. In a 1998
report,1 we characterized the knowledge needed on a new product as
consisting of three knowledge points: when a match is made between
a customer's requirements and the available technology; when the
product's design is determined to be capable of meeting
performance requirements; and when the product is determined to be
producible within cost, schedule, and quality targets (see fig.
1.1). We found that this knowledge, when obtained at the right
time and in the right sequence-technology, design, and
manufacturing-was a best practice. This practice lowered product
development risks, reduced cycle times and costs, and resulted in
smoother production programs. Figure 1.1: Cycle for Providing
Users a Product With Better Capabilities Technology
Product Development Development
Production Product launch Knowledge Knowledge
Knowledge Point 1 Point 2
Point 3 Technology Design
Control of readiness maturity
manufacturing process Leading commercial firms recognize a
distinct difference between technology development and product
development; accordingly, they develop technology before
introducing it into product development programs. They minimize
risk, improve cost and schedule outcomes, reduce cycle time, and
improve quality during product development by 1Best Practices:
Successful Application to Weapon Acquisitions Requires Changes in
DOD's Environment (GAO/NSIAD-98-56, Feb. 24, 1998). Page 13
GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction gaining
significant knowledge about a technology before launching the
product development. Scientists and technologists-different people
than those that manage product developments-manage the development
of technology until it is ready to be included in the design of a
product. Program launch is the point at which a firm defines a
product's performance, cost, and schedule estimates and begins
making a large investment in human capital, facilities, and
materials-an investment that increases continuously as the product
approaches the point of manufacture. It includes a commitment to
manufacture the product. Therefore, program launch and the start
of product development are synonymous within commercial firms.
Protecting this investment provides a strong incentive for firms
to minimize the potential for technology development problems
during the product phase and cause delays. Confining delays in
maturing technology to a time prior to launch-in an environment
where small teams of technologists work in laboratories and are
dedicated to perfecting the technology-is critical to saving time
and money. If delays occur during product development, when a
large engineering force is in place to design and manufacture the
product, they would be much more costly. In fact, industry experts
estimate that identifying and resolving a problem before product
development can reap a 10-fold savings compared to correcting the
problem after launch and that correcting the same problem in the
manufacturing stage would be even more costly. Leading commercial
firms have found that managing technology development separately
from and before product development is a major reason they have
been able to reduce product cycle times. As a whole, 50 to 70
percent reductions in cycle times are not unrealistic achievements
by leading commercial firms. For instance, leaders in the
automobile industry have reduced cycle times from 7 years to 2
years, or by about 70 percent. The consumer electronics industry
has recently reduced its cycle time from 2 years to 6 months, and
the commercial aircraft industry has achieved reductions of 50
percent. Leading commercial firms have found that reducing the
product development cycle time brings products to market faster,
results in an increased market share, and helps to keep products
from becoming technologically obsolete. Page 14
GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction Technology
and DOD's process for developing and
manufacturing weapon systems is Product Development described as
a cycle consisting of phases. These phases are concept
exploration, program definition and risk reduction, engineering
and Conducted at the Same manufacturing development, and
production and fielding. The basic Time Within DOD
process of gathering knowledge about technology, design, and
manufacturing is followed, but in practice, the DOD cycle does not
make a clear distinction between technology development and
product development. The launch of a program in DOD usually takes
place several years before the beginning of product development
does in leading commercial firms. In fact, a new weapon system
program is normally launched at the start of the program
definition and risk reduction phase, which is often in the midst
of technology development, while most product development
activities do not begin until the engineering and manufacturing
development phase. Consequently, technology, design, and
manufacturing knowledge is attained concurrently--in the higher
cost environment that characterizes product development--
throughout the weapon system phases. In our February 1998 report,
we noted that such technology development problems are a major
cause of cost increases and schedule delays on DOD weapon system
programs. The phases in DOD's weapon system acquisition cycle and
the knowledge gathering process, as it is typically followed, are
shown in figure 1.2. Page 15
GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction Figure 1.2:
DOD's Weapon System Acquisition Cycle Concept
Program Engineering and Production
exploration definition and
manufacturing risk reduction development
and fielding Program launch Begin product
development Technology maturity Knowledge
Design maturity attainment Manufacturing processes controlled
DOD's process also has organizational and budgetary implications.
Activities accomplished in the first three phases of the
acquisition cycle use research and development funds whereas
production programs use procurement funds. Generally, DOD's
science and technology (S&T) community is responsible for basic
research, applied research, and advanced technology development to
produce generic, rather than weapon-specific, technologies. Its
goal is to conduct research, develop technology, and farm these
efforts for potential military application, such as a weapon
system. The S&T community also uses research and development
funds, but its work generally precedes the acquisition cycle.
Weapon system program managers, who receive most of DOD's research
and development budget, apply generic technologies to specific
weapon systems. However, they often become responsible for
completing development of generic technologies as well. The
allocation of DOD's fiscal year 1999 research and development
funds to these categories is shown in figure 1.3. Page 16
GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction Figure 1.3:
Allocation of DOD's Fiscal Year 1999 Research and Development
Funds 3% Basic research - $1.1 billion 9% Applied research - $3.2
billion 9% Advanced technology development - $3.5 billion 79%
Weapon specific development - $29.6 billion Source: DOD S&T
officials stated their role is to show that technology is feasible
through laboratory experiments or demonstrations. It is often at
this point that the technology's military potential will be
identified and the technology will be harvested for inclusion on a
weapon system. Because the technology is still not mature, its
development will be completed as part of the weapon system's
design and development, under the authority of the weapon system
manager and apart from the S&T community. Shorter Acquisition
DOD's weapon acquisition cycle times average between 10 to 15
years-far Cycle Times Are longer
than the cycle time for commercial products. To an extent, DOD's
cycle times are longer because they start earlier than commercial
cycles Needed and often
entail more complex products. Compounding the length of the weapon
system development cycle is its unpredictability. Over the years,
we have issued numerous reports highlighting cost overruns and
schedule delays during the product development cycle, for which
technology development problems were a major cause. These problems
require additional technology development activities to take place
at a time when Page 17
GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction the product
should be undergoing design and manufacturing development. As a
result, the pace of technology advances outruns the time to
develop a weapon system and some of the more mature components
designed into a weapon system become obsolete before the weapon is
manufactured. For example, the F-22 will have almost 600 obsolete
components by fiscal year 2000 while the aircraft is still in
development. The longer a weapon system's development cycle, the
more prone the program is to management and funding changes.
According to DOD, an 11-year development program historically
encounters a 30-percent cost growth over time. Based on historical
averages, DOD calculates that the typical program will have four
different program managers, eight defense acquisition executives,
and seven Secretaries of Defense-all of who are major influences
and decisionmakers on the program. In addition, the program will
have gone through 11 annual budget cycles in which funding changes
could have occurred and affected the program's content. The Under
Secretary of Defense for Acquisition and Technology has stated
that cycle time reduction is necessary to meet DOD's goals of
delivering emerging technologies to warfighters in less time and
at lower costs. The Under Secretary has set a goal to reduce the
average acquisition cycle time for all program starts in fiscal
year 1999 and beyond by 50 percent over historical averages.
Reductions in cycle times will (1) allow for earlier fielding of
increased capabilities, (2) reduce costs, (3) free up funds for
more programs, (4) reduce the potential for components becoming
obsolete, and (5) take more frequent advantage of technology
advances found in the commercial world. An emphasis on shorter
cycle times may also reduce the tendency to add technological
advances that are unproven and immature into weapon acquisition
programs. To help achieve this goal, DOD is working on several
efforts such as Defense Acquisition Pilot Programs, the Defense
Reform Initiatives, and many acquisition reform projects. The
Under Secretary has also advocated adopting the practices of
leading commercial firms and taking a more evolutionary approach
to developing weapon systems, which would lessen the amount of
technology development initially attempted within a weapon system
program. Objectives, Scope, and The Chairman and the Ranking
Minority Member, Subcommittee on Methodology
Readiness and Management Support, Senate Committee on Armed
Services, requested that we examine various aspects of the
acquisition process to determine whether the application of best
practices can improve program outcomes. To date, we have issued
reports on advanced quality Page 18
GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction concepts,
earned value management, management of a product from development
to production, and management of key suppliers (see related GAO
products). This report covers the inclusion of technology into
weapon system programs, and is, in a sense, a prequel to our
report on product development. Our overall objective was to
determine whether best practices offer methods to improve the way
DOD matures new technology so that it can be assimilated into
weapon system programs with less disruption. Specifically, we
assessed (1) the impact of technology maturity on product
outcomes, (2) best practices for managing new technologies and
incorporating them into products, and (3) ways DOD can adapt best
practices to achieve better outcomes on weapon system programs.
Our methodology consisted of analyzing 23 commercial and DOD
technologies that had transitioned or attempted to transition into
product development programs. The technologies were drawn from six
commercial firms recognized for their success in developing
technically advanced products more quickly than their predecessors
and five weapon system programs that incorporated advanced
technologies, including some that did not encounter problems and
some that did. We asked the managers of these technologies to
apply a tool, referred to as technology readiness levels (TRLs),
for our analysis. The managers used TRLs to judge the maturity of
the technologies at the time they had entered product development
or were included in programs. The National Aeronautics and Space
Administration (NASA) originally developed TRLs, and the Air Force
Research Laboratory uses them to determine when technologies are
ready to be handed off from S&T managers to product development
managers. We held discussions with the DOD and NASA users of TRLs
to better understand their applicability to our review. They
stated that TRLs can be used as general indicators of a
technology's readiness level and associated risk of including the
technology into a product development program, given its TRL at
that time. TRLs are more fully explained in chapter 2. To
understand the best practices the commercial sector used to
include technologies in product development programs, we conducted
literature searches and focused those searches as the review
progressed. On the basis of the searches, we identified a number
of commercial firms with innovative technology development
processes for including new or advanced technologies into new
products. We used structured interview questions sent in advance
of our visits to gather uniform and consistent information about
each firm's process and practice and the results achieved. In
addition, we examined four specific technology cases-Ford's night
vision, adaptive cruise control, and voice activated controls and
Page 19 GAO/NSIAD-99-162
Best Practices Chapter 1 Introduction Hughes' solar cell array-to
better understand their processes and practices. The commercial
firms we visited were * Ethicon-Endo Surgery (medical device
manufacturer), Division of Johnson and Johnson, Cincinnati, Ohio;
* Ford Motor Company (automobile manufacturer), Dearborn,
Michigan; * Harris Semiconductor (semiconductor manufacturer),
Melbourne, Florida; * Hughes Space and Communications (satellite
and spacecraft manufacturer), Los Angeles, California; * 3M
(commercial products manufacturer), St. Paul, Minnesota; and *
Motorola Corporate Research Headquarters (communications
technology manufacturer), Schaumburg, Illinois, and Motorola Land
Mobile Products Sector, Plantation, Florida. We also attended and
participated in conferences and workshops with recognized leaders
in the acquisition field to obtain information on how
organizations are improving their acquisition processes. Finally,
we interviewed officials from trade organizations concerning the
application of commercial practices to DOD operations. To better
understand DOD's technology inclusion process, we selected 19
advanced technologies that had been included in 5 DOD weapon
system programs that were in various stages of the acquisition
process. We collected technical reports, acquisition management,
and risk management documentation about the technologies. In
addition, we interviewed S&T and acquisition program management
officials about each technology's development history, costs, and
current status. The technologies and programs reviewed were *
acoustic sensor, infrared seeker, inertial measurement unit,
tandem shaped charge warhead, and processor technologies from the
Army's Brilliant Anti-Armor Submunition (referred to as BAT)
Program; * rotor, engine, integrated avionics, forward looking
infrared, and helmet mounted display technologies from the Army's
Comanche Helicopter Program; * nonpenetrating periscope and weapon
ejection system technologies from the Navy's Virginia class attack
submarine program; * high speed planing craft, power dense diesel
engine, lightweight composite armor, high power water jet, moving
map and advanced navigation technologies from the Marines'
Advanced Amphibious Assault Vehicle (AAAV) Program; and Page 20
GAO/NSIAD-99-162 Best Practices Chapter 1 Introduction * laser and
beam control technologies from the Air Force's Airborne Laser
(ABL) Program. To determine relevant DOD policy and initiatives,
we obtained documents and interviewed officials of the Office of
the Secretary of Defense; the Defense Advanced Research Projects
Agency (DARPA); and Army, Navy and Air Force Science and
Technology organizations. We also had discussions with former DOD
officials and industry experts about DOD acquisition policies and
practices. Even though we selected firms with product lines of
varying complexity, we did not concentrate only on firms whose
products had the most in common with weapon systems. Such an
approach would have limited our ability to include firms
recognized as the best at including new, advanced technologies
into programs. In our analysis, we concentrated on the criteria
and knowledge used to support technology readiness decisions.
Although the approach from product to product may vary, the basic
processes and standards leading commercial firms applied to
technology inclusion decisions were consistent. We were limited,
however, in our ability to obtain and present some relevant data
that commercial companies considered proprietary in nature. This
information included funding amounts for investing in technology
development, details on technological innovations, and some
specific data from recent technology inclusion successes. Our
report highlights the best commercial practices for including
technology into product development programs. As such, they are
not intended to describe all commercial industry practices or to
suggest that commercial firms do not have any flaws. We conducted
our review between March 1998 and June 1999 in accordance with
generally accepted government auditing standards. Page 21
GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
at Program Start Is an Important Determinant of Success
Chapter 2 The experiences of the DOD and commercial technology
development cases we reviewed indicate that demonstrating a high
level of maturity before allowing new technologies into product
development programs puts those programs in a better position to
succeed. Simply put, the more mature technology is at the start of
the program, the more likely the program will succeed in meeting
its objectives. Technologies that were included in a product
development before they were mature later contributed to cost
increases and schedule delays in those products. We found an
analytical tool-TRLs-that can assess the maturity level of
technology as well as the risk that maturity poses if the
technology is included in a product development. The tool
associates different TRLs with different levels of demonstrated
performance, ranging from paper studies to proven performance on
the intended product. The value of using the tool is that it can
presage the likely consequences of incorporating a technology at a
given level of maturity into a product development, enabling
decisionmakers to make informed choices. TRLs proved to be
reliable indicators of the relative maturity of the 23
technologies reviewed, both commercial and military, and their
eventual success after they were included in product development
programs. Technology Maturity Successful technologies
progress from initial concept to proven Can Be Measured and
performance, whether they are developed in the laboratory or in
the factory, by commercial industry or DOD. The Air Force Research
Its Consequences for Laboratory has adapted and uses
TRLs to measure the key steps in this Products Can Be
progression toward inclusion into weapon systems. TRLs are
measured Forecast along a scale of one to
nine, starting with paper studies of the basic concept and ending
with a technology that has proven itself in actual usage on the
intended product. A detailed description of TRLs is provided in
appendix II, but the following hypothetical example about an
airborne communications radio can illustrate the readiness levels.
First, the idea for a new radio is conceived. The idea reaches TRL
3 when analytical studies and some tests of the technology's
elements, such as a circuit, back it up. When initial hand-built
versions of all of the radio's basic elements are connected and
tested together, the radio reaches TRL 5. This is sometimes
referred to as a "breadboard" article; although it may function
like a radio, it does not look like one because the individual
parts are attached to plywood and hand-wired together. When the
technology is built into a generic model, which is well beyond the
breadboard tested in TRL 5, and demonstrated in a laboratory
environment, the radio reaches TRL 6. Page 22
GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
at Program Start Is an Important Determinant of Success This model
represents the last level of demonstration before the radio
becomes tailored for application to a specific aircraft. When the
components are assembled inside a case that resembles the final
radio design and are demonstrated aboard a surrogate for the
intended aircraft, the radio reaches TRL 7. TRL 8 is reached when
the radio is put in its final form, installed in the intended
aircraft's cockpit, and tested in conjunction with the other
aircraft equipment with which it must interface. TRL 9 is achieved
when the radio is successfully operated on the aircraft through
several test missions. Unexpected problems can arise at every
level, and effort must be expended to overcome them. This effort
takes time and can delay the progress to the next readiness level.
Once a technology's readiness level has been established, the
risks of including that technology in a product development can be
assessed. Unlike S&T projects, for which the main objective is to
develop knowledge, a product development's objective is to deliver
products that meet strict cost, schedule, and performance targets.
We found that most leading commercial firms, after they had
translated their own methods of assessing risk into TRLs,
determined that a TRL 8 was required before they allowed a new
technology into a product development.1 DOD launches a program in
the program definition and risk reduction phase-much earlier than
the leading commercial firms do. According to the Air Force
Research Laboratory, a TRL 6 is required for a technology to be an
acceptable risk for a program in that phase. When weapon system
development reaches the engineering and manufacturing development
phase, it more nearly approximates the point at which a commercial
product development program would start. The Air Force Research
Laboratory depicts a technology at TRL 7 as an acceptable risk for
this phase-technologies at lower levels would be considered high
risks. The lower the level of technology readiness, the more
ground must be covered to bring the technology to the point at
which it can meet the intended product's cost, schedule, and
performance requirements with little risk (see fig. 2.1). 1An
exception to this is space systems technology. Space-based
technologies are generally included on a development program once
they have been prototyped and ground tested-a TRL 6, the highest
level attainable short of space operation. Page 23
GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
at Program Start Is an Important Determinant of Success Figure
2.1: Using TRLs to Match Technology With Product Launch
Requirements High risk for Low
risk for Product product launch
product launch Requirements
8 9 Risks or 6 7
unknowns 4 5 TR L 1 2 3 The gap between the
maturity of the technology and the product's requirements
represents the risks or unknowns about the technology. As each
succeeding level of readiness is demonstrated, unknowns are
replaced by knowledge and the gap becomes smaller. Ideally, the
gap is closed before a new technology is included in a new
product's design, although the Air Force Research Laboratory
accepts the amount of risk at TRL 7 for a program entering
engineering and manufacturing development. Technologies that reach
TRL 7 or higher at the start of product development allow product
managers to focus their attention on integrating the technologies
and proving out the product design. Technologies that are included
at lower maturity levels require more of the product managers'
attention and resources, as basic knowledge about those
technologies must still be gained. Thus, a major purpose served by
TRLs is to reveal the gap between a technology's maturity and the
maturity demanded for successful inclusion in the intended
product. With TRLs as guides, the options available to
decisionmakers can be framed. Given that a key determinant of
achieving cost and schedule outcomes for a product development is
the technology's maturity at product launch, decisionmakers can
either (1) delay product Page 24
GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
at Program Start Is an Important Determinant of Success
development until the technology is matured to a high enough
readiness level or (2) reduce the product's requirements so that a
less advanced, but more mature, technology can suffice. If it is
perceived that the requirements of the product cannot be lowered
and the product launch cannot be delayed until the requisite
technology is of a sufficient readiness level, then the remaining
option is to launch the product development with the immature
technology. If this option is chosen, then the success of the
product development will depend heavily on the product manager's
ability to simultaneously close the technology maturity gap and
develop the product for manufacture, which is a very challenging
task. TRLs do not represent strictures that must be adhered to
without exception. According to the people in DOD who have used
TRLs, there are occasions when a lower than expected TRL can be
accepted, such as when the product development's schedule and
resources are generous enough that the technology will have enough
time to mature. In other instances, a higher than expected TRL may
be required, such as if the technology in question is the linchpin
for the entire product. Nonetheless, we found that TRLs ably
reconciled the different maturity levels and product experiences
of the 23 technologies reviewed. Technologies With
The 23 technologies reviewed spanned a wide range of readiness
levels at High Readiness Levels the time they were included in
product development programs. The least mature reached TRL 2 at
the time it was included in a product at Launch Were Better
development, while the most mature had reached TRL 9 at the point
of Able to Meet Product inclusion. We observed a
general relationship between TRLs and the Objectives
technologies' inclusion on the intended product developments.
Those products whose technologies reached high TRLs at the time
they were included were better able to meet cost, schedule, and
performance requirements. In fact, commercial firms informed us
that maturing the technology separately from and ahead of the
product was a main reason they were able to reduce cycle times on
their products. An official from one of the firms termed the
approach as "moving discovery to the left." Those technologies
with low TRLs at inclusion encountered maturation difficulties and
contributed to problems the products experienced. Other problems,
such as funding and schedule changes unrelated to the
technologies, also contributed to problems in the product
developments. Figure 2.2 shows the TRLs when each of the 23
technologies was included in a product design, whether at product
development launch (for commercial technologies) or at program
launch (for DOD technologies). Page 25
GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
at Program Start Is an Important Determinant of Success Figure
2.2: Readiness Levels of Technologies at the Time They Were
Included in Product Designs Non-penetrating periscope Adaptive
cruise control Night vision Voice activated control Solar cell
array High-speed planing craft High power water jet Weapon
ejection system Diesel engine Helicopter rotor Lightweight
composite armor Helicopter engine Moving map and navigation
Hemical oxygen iodin laser Beam control system Helmet mounted
display Helicopter forward linking infrared Integrated avionics
Data processor Inertial measurement unit Warhead Infrared seeker
Acousting targeting sensor 1 2 3
4 5 6 7 8 9 Technology
Readiness Levels Commercial technologies DoD technologies Page 26
GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
at Program Start Is an Important Determinant of Success The cost
and schedule experiences of some of the products or programs that
inherited the technologies are shown in table 2.1. Table 2.1:
Cost and Schedule Experiences on Product Developments Product
development TRL at Product development and
program associated technologies launch
Cost growth Schedule slippage Comanche helicopter
101 percenta 120 percenta Engine
5 Rotor 5 Forward
looking infrared 3 Helmet mounted display
3 Integrated avionics 3 BAT
88 percent 62 percent Acoustic sensor
2 Infrared seeker 3 Warhead
3 Inertial measurement unit 3 Data
processors 3 Hughes HS-702
satellite None
None Solar cell array 6 Ford Jaguar
None None Adaptive cruise control
8 Voice activated controls 8 aThe Comanche,
in particular, has experienced a great deal of cost growth and
schedule slippage for many reasons, of which technology immaturity
is only one. Other factors, such as changing the scope, funding,
and pace of the program for affordability reasons, have also
contributed. Data for three weapon system development programs,
the Virginia class attack submarine, AAAV, and the ABL, were not
included in the table because they had not been in the product
development phase long enough to report actual consequences. To
date, AAAV and the submarine have stayed within 15 percent of
their cost and schedule estimates for development. The ABL, for
which key technologies were much less mature at program launch,
still faces challenges with these technologies. Ford's night
vision technology was excluded because the firm decided not to
include the technology on a product. Details on the Comanche, BAT,
the Virginia class attack submarine, and Ford technology and
product experiences follow. Page 27
GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
at Program Start Is an Important Determinant of Success Technology
and Product The key technologies for the Ford
Jaguar and the Virginia class attack Experiences on Ford and
submarine followed the pattern of increasing TRLs until they
demonstrated Virginia Class Attack a low risk
for transition to the product. Two examples are Ford's voice
Submarine activated controls
development and DARPA's nonpenetrating periscope development for
the submarine. In both cases, the technologies were validated,
operational prototypes demonstrated, and the technologies had
demonstrated the form, fit, and function of the final article by
the beginning of product development. Ford's voice activated
controls technology, which allows a driver to control certain
functions such as windows and the radio through verbal commands,
was under development in the technology base for over 10 years,
being pushed by the firm's technology leaders. It was not until
1993 that Ford found that (1) other complementary technologies,
such as processor speeds and low cost memory, had become available
and (2) customers wanted more features and functions but less
distractions from driving. Given this market information, Ford
decided to pursue voice technology as a strategic technology in
terms of product differentiation, recognizing the importance of
being first to market with this enabling technology. Figure 2.3
shows the time line for developing this technology. Figure 2.3:
Time Line for Ford's Development of Voice Activated Controls
Technology 1983 1993
1995 1999 Ford decides to pursue
Technology is linked Technology is ready to
Technology featured voice activated controls to a
specific vehicle. transition into a product on model
year 1999 technology. Technology Cost and
performance development program. Jaguar
designs. under early development requirements are
Technology meets all in technology base.
defined. cost and schedule targets for the
product. TRL 3 - 5 TRL 6 - 7
TRL 8 TRL 9 Between 1993 and 1994, based
on discussions with customers, Ford developed cost and performance
requirements for the technology. Ford has Page 28
GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
at Program Start Is an Important Determinant of Success never
relaxed them. By September 1995, when Ford allowed the technology
into the development program for a new Jaguar design, voice
activated controls had been demonstrated as an integrated system
in the appropriate form and fit for the Jaguar. Ford officials
stated that the product has met all cost and cycle time targets
established at the outset of its development. Figure 2.4 shows the
Jaguar. Figure 2.4: Jaguar Ford demonstrated voice activated
control technology in the appropriate form and fit before
incorporating it into the Jaguar. Source: Ford Motor Company.
DARPA began developing the nonpenetrating periscope technology as
part of its submarine technology development efforts after
recognizing, in 1988, along with the Navy, that the nonpenetrating
technology would enhance operator visibility, provide greater
submarine design flexibility, and be stealthier than conventional
masts and periscopes. At the time, the Virginia class attack
submarine program had not been initiated. Once the decision was
made to include the nonpenetrating periscope, it became a key
feature of the submarine and was a major design driver for the
submarine's overall configuration. Nonpenetrating refers to the
fact that the periscope is essentially a group of sensors that are
linked to the submarine via fiber optic and other cables. This
technology uses infrared imaging and advanced sensors to replace
conventional periscopes and frees up physical space compared with
a conventional periscope. A conventional periscope relies on a
series of telescoping shafts and reflecting surfaces to see above
the water's surface. Page 29
GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
at Program Start Is an Important Determinant of Success When the
periscope is retracted, the shafts take up a column of space from
the top of the submarine to the bottom, through all decks. Its
location virtually dictates the design and placement of the
control and other rooms. If the nonpenetrating periscope
technology did not become available, then the submarine would have
to be drastically redesigned to accommodate the space required by
a conventional periscope. The new nonpenetrating periscope and
photonics mast technology underwent land testing in 1991-a TRL 5.
The Navy actually tested the new technology at sea on the U.S.S.
Memphis in 1992 and 1993. According to program officials, these
sea trials demonstrated the highest level of technology readiness:
proving the actual system through successful mission operations.
This readiness equated to a TRL 9. Yet, this technology was not
included in the Virginia class attack submarine requirements until
1995. Figure 2.5 shows an artist's concept of the Virginia class
attack submarine. Figure 2.5: Virginia Class Attack Submarine The
Navy demonstrated a key technology at the highest readiness level
before including it as a requirement for the Virginia class attack
submarine. Source: DOD. The high readiness level of the
nonpenetrating periscope afforded the Navy the opportunity to
develop an improved version of the periscope to a TRL 9. This was
a relatively low-risk endeavor as the baseline periscope was
sufficient to meet the submarine's requirements. Program officials
believe that having knowledge about key technologies, such as the
nonpenetrating periscope, for the Virginia class attack submarine
at program launch made a short program definition and risk
reduction phase possible. This phase for the Virginia class attack
submarine was about 75 percent shorter than those of previous
acquisition programs. Based on its demonstrated maturity, we
anticipate that the nonpenetrating periscope to be less likely to
impact the cost and schedule of the submarine's development
program. There are, however, several other technologies that Page
30
GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
at Program Start Is an Important Determinant of Success are
critical to the submarine program. We did not examine these
technologies and cannot predict their likely outcomes. BAT and
Comanche Cases Key technologies for the BAT and Comanche
programs had much lower readiness levels at the time the product
developments were launched. Consequently, they did not reduce the
gap between their demonstrated maturity and the maturity needed to
meet product requirements until after program launch. For some
technologies, the gap was not closed until well into the product
development program. For others, the gap has still not been
closed. Five key technologies included in the Army's BAT program
had low TRLs when they were included on the program. The level of
readiness for most of these technologies at program launch was
characterized by the program office as experimental in nature but
with major uncertainty remaining-a TRL 3. The acoustic targeting
technology was the most important enabling technology needed to
meet the weapon's performance requirements. This technology
provides BAT the capability to locate targets from great distances
based on the sounds generated by the target, such as moving tanks
and vehicles. At the time the program was launched, the Army
knew little about the feasibility of using this technology on this
program. In fact, the technology was still being defined in paper
studies-a TRL 2. The Army did not prototype this technology
until after the program had entered the engineering and
manufacturing development phase, more than 6 years after program
launch. As of December 1998, the BAT had experienced significant
development cost and schedule increases, which program officials
attribute at least, in part, to unknowns about the new
technologies. Figure 2.6 shows the BAT. Page 31
GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
at Program Start Is an Important Determinant of Success Figure
2.6: Brilliant Anti-Armor Submunition At the time the program was
launched, the Army knew relatively little about the performance of
several key technologies for the BAT Source: DOD. Two technologies
key to meeting the Comanche helicopter's requirements integrated
avionics and forward-looking infrared (FLIR) technologies- were
included on the program when they were still conceptual in nature.
The integrated avionics technology replaces individual radios,
navigation, and other communication equipment with a modular
system that shares a common processor. The FLIR is a second-
generation version that uses infrared sensors to improve the
pilot's ability to see at night and in bad Page 32
GAO/NSIAD-99-162 Best Practices Chapter 2 Maturity of Technology
at Program Start Is an Important Determinant of Success weather.
Program officials stated that both had TRLs of 3 when the
helicopter program was started. Despite the low readiness levels
of the technologies, the Army included the technologies on the
program to meet weight, cost, and performance requirements. The
development of these technologies has taken longer than the Army
expected it would. The contractor for the integrated avionics has
had difficulties in getting the multiple avionics modules to work
simultaneously within required size and weight parameters, and the
FLIR technology has undergone several design and performance
requirement changes. As of September 1998-approximately 10 years
after program launch-neither the integrated avionics nor the FLIR
technology had advanced past a TRL 5. Problems with the maturation
of these technologies have contributed to the program's cost and
schedule increases. In contrast, the advanced rotor and engine
technologies, which were the most mature of the Comanche
technologies we reviewed, have experienced fewer problems in
maturation and have not contributed significantly to the program's
cost and schedule increases. Figure 2.7 shows the Comanche. Figure
2.7: Comanche Helicopter The Army included two key technologies
in the Comanche when they were still considered conceptual to meet
weight, cost, and performance requirements Source: DOD. Page 33
GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
Affect How Well a Technology's Inclusion on a Product Can Be
Managed
Chapter 3 Closing the gap between technology maturity and product
requirements before a product is launched-and baselines are set-
distinguished the more successful cases. Notably, closing the gap
before product launch was a managed result; it put product
managers in a better position to succeed. Two conditions were
critical to achieving this kind of result. First was an
environment that put the primary responsibility for maturing
technology in the hands of S&T managers and provided them
considerable flexibility to make decisions. Second was having the
quality information and standards needed to make good technology
handoff decisions, coupled with giving the product manager the
authority to refuse new technology that did not meet product
requirements. When these conditions were not present, the handoff
to the product manager was compromised, with negative consequences
for both technology and product. In each of the successful cases,
S&T organizations played major roles in bridging the gap between
technology maturity and product requirements. Flexibility provided
by requirements communities and resource providers enabled S&T and
product managers to delay the inclusion of technology if it was
not ready or to reduce product requirements to match what mature
technology could deliver. This environment was better suited to
the unexpected results and delays that accompany technology
development. Moreover, technology maturation was managed within a
disciplined process that provided good information to be judged
against clear and high standards, like TRLs. Armed with the tools
and the authority to make technology inclusion decisions, both S&T
and product managers functioned as gatekeepers to safeguard the
product development. In the more problematic cases, S&T
organizations disengaged much earlier, and product managers had
little choice but to accept immature technologies. Accordingly,
less information about the technologies was available at the point
of inclusion. Often, the tools used to assess the technologies'
status failed to identify high risks. In retrospect, TRLs
indicated that risks were in fact high and perhaps unacceptable
from a product standpoint. Also, pressures to meet cost and
schedule estimates in product development provided a less
forgiving environment for technologies in the discovery process.
Page 34 GAO/NSIAD-99-162
Best Practices Chapter 3 Controllable Conditions Affect How Well a
Technology's Inclusion on a Product Can Be Managed Providing the
Right While most new technologies-commercial and
military-are initially Environment Is Critical managed by the S&T
community, the more successful cases we reviewed continued to be
managed by S&T organizations until they reached at least to the
Successful TRL 6 and more often TRL 8 or higher.
These technologies were provided Maturation of
the environmental advantages an S&T project has over a product
Technology development. This environment
availed S&T managers and product managers of the less risky
options of waiting or trading to get the match between technology
maturity and product requirements-rather than forcing the product
launch and gambling on the completion of technology maturity. In
contrast, the more problematic technologies did not have as benign
an environment. Often, the technologies were handed off early by
S&T organizations because inflexible performance requirements for
the product demanded their inclusion. Product development managers
launched the product development and hoped that the technology
development would succeed. Once in a product development
environment, external pressures to keep the program moving become
dominant, such as preserving cost and schedule estimates to secure
budget approval. For example, DOD policies require that a program
be funded in the current year and that funds be made available
over the next 6 years in the DOD planning cycle. If, during the
program definition phase, a program manager were to decide that an
additional year was needed to overcome unexpected technology
problems to reach the desired level of maturity, the delay could
push the start of engineering and manufacturing development back.
This delay could jeopardize the funding for that phase, thus
risking the funding support for the entire program. Consequently,
the program manager may be more likely to accept the risk of not
getting the technology to the desired level of maturity and
starting the engineering and manufacturing development phase as
planned, rather than risk the rest of the program. These
conditions compete with and detract from the needs of technology
development. One acquisition official stated that these conditions
cause the weapon system program "to pull double duty," inventing
new technology while integrating it into a product. In general, he
believed there is an equal amount of difficulty in both tasks.
Technologies Matured by In the most successful cases that
we reviewed, S&T organizations bridged S&T Organizations Made
the gap between immature technology and the maturity needed for
either Smooth Transitions into program start in DOD (TRL
6) or product development (TRL 7 or higher). Product Developments
These cases and the responsible organizations are shown in table
3.1. Page 35 GAO/NSIAD-
99-162 Best Practices Chapter 3 Controllable Conditions Affect How
Well a Technology's Inclusion on a Product Can Be Managed Table
3.1: TRLs of Technologies Managed by S&T Organizations TRL at
Responsible Receiving product Technology
handoff S&T organization development program
Nonpenetrating 9 DARPA
Virginia class attack periscope
submarine program Adaptive cruise control 8 Ford
Advanced Vehicle Jaguar vehicle team Technology Office Voice
activated controls 8 Ford Advanced Vehicle Jaguar
vehicle team Technology Office Solar cell array 6
Hughes Laboratories HS-702 satellite program Weapon
ejection 6 Office of Naval Virginia
class attack system Research
submarine program Diesel powered engine 6 Office of
Naval AAAV program Research High-speed planing
6 Office of Naval AAAV program craft
Research High-power water jet 6 Office of Naval
AAAV program Research Despite the different circumstances between
the commercial and DOD sectors and among the DOD cases themselves,
the results were similar: having S&T organizations bridge the
maturity gap reduced technology-related problems in the products.
For the leading commercial firms we visited, it is standard
practice to have S&T organizations responsible for the bridge. In
the DOD cases shown in table 3.1, the S&T organizations played
atypical roles in managing the bridge between technology and
product by delivering the technology to a TRL 6 or higher.
Different pressures and incentives that are brought to bear on the
commercial and DOD product developments explain why DOD product
managers become responsible for more technology development than
their commercial counterparts. These influences are discussed in
chapter 4. Having an S&T organization manage a technology to
maturation means more than just having a different group of people
involved than a product development. S&T projects operate in a
different environment than product developments. The process of
developing technology culminates in discovery and must, by its
nature, allow for unexpected results. S&T provides a more
forgiving environment in which events-such as test "failures," new
discoveries, and delays in the attainment of knowledge-are
considered normal. It is also a less costly environment, making
external pressures to develop knowledge on a schedule less keenly
felt. On the Page 36
GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
Affect How Well a Technology's Inclusion on a Product Can Be
Managed other hand, the process of developing a product culminates
in delivery, and thus gives great weight to design and production.
The same events and unexpected results that are considered normal
for technology development represent problems in the product
environment; they can jeopardize achievement of cost and schedule
objectives and draw criticism to the product. The ups and downs
and the resource changes associated with the technology discovery
process do not mesh well with a program's need to meet cost,
schedule, and performance goals. This situation has been described
as attempting to "schedule inventions." Successful Cases Afforded
In the early 1980s, Hughes Space and Communications began
developing Flexibility to dual junction solar cell
technology that had the potential of greatly Decisionmakers
increasing the electrical power on satellites. By 1985, a Hughes
laboratory had demonstrated the technology by ground testing
prototypes, a TRL 6, which is considered an acceptable level of
demonstration for space-based technology. Nonetheless, Hughes
was not satisfied that the supporting infrastructure (materials,
reactors, and test equipment) was mature enough to sustain
development and production of the new technology on a satellite.
The infrastructure was seen as critical to meeting the cost and
schedule requirements of a product. As a result, Hughes did not
hand off the technology to a product. Instead, the firm kept it in
a research environment, away from cost and schedule pressures. In
the early 1990s, Hughes established requirements for a new
satellite- the HS-702-that would use the solar cell technology to
leapfrog the competition. After a laboratory demonstration in
1993, Hughes successfully used the new technology on a high-
powered version of its existing HS-601 satellite before it began
product development on the HS-702 satellite. By 1994, it had
determined that the business base was available to sustain
development and production of the HS-702 satellite. In all, the
firm waited 10 years for the demonstrated technology to meet the
requirements. This experience closely resembled that of Ford's
voice activated control technology because, in both cases, the new
technology took 10 years to mature enough for product readiness.
Thus, the firms' approach was not to accelerate technology
development but to shorten product development by maturing the
technology first. Figure 3.1 shows the solar cell arrays installed
on the HS-702. Page 37
GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
Affect How Well a Technology's Inclusion on a Product Can Be
Managed Figure 3.1: Hughes Solar Cell Arrays Hughes successfully
proved solar cell array technology on a predecessor satellite
before beginning product development of the HS-702 Source: Hughes
Space and Communications. Page 38
GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
Affect How Well a Technology's Inclusion on a Product Can Be
Managed The Navy made trade-offs in choosing a technology for the
weapon ejection system, which is used to deploy weapons like
torpedoes, of the Virginia class attack submarine. Because of
quietness, weight, and cost requirements, the Navy preferred a new
elastomeric (rubber-based) technology. However, this technology
failed endurance testing, and product managers determined that the
technology was too risky to be included in the first product.
Product managers could have declined this technology and its
attendant risk without delaying the submarine's schedule because
the Navy accepted marginal increases to the cost and weight
requirements for the system so that the proven Seawolf ejection
technology could be used as a substitute. Using proven technology
on the first submarine has allowed the Navy S&T community to
continue developing the elastomeric technology, which is to be
incorporated into the new system on the fourth production
submarine. As discussed earlier, decisionmakers also had the
flexibility to wait for the nonpenetrating periscope technology to
reach TRL 9 before including it in the submarine's requirements.
Problematic Cases Provided According to Army officials, the FLIR
and integrated avionics technologies Little Flexibility to
required for the Comanche helicopter were critical for providing
an Managers increased operational
capability over existing Army helicopters. The advanced FLIR
technology was needed to meet the user's requirements for
increased targeting range and for improved piloting capabilities
in bad weather and at night. It represented a quantum leap from
existing capabilities. Integrated avionics technology was expected
to replace separate radios, navigation systems, and other
communication equipment on the helicopter with a modular system
that uses central processors. These technologies were needed to
meet weight and size requirements for the aircraft as well as
improve communications. Both were critical elements of a mission
equipment package that was supposed to reduce the pilot's workload
while improving capabilities. Requirements were inflexible. Thus,
requirements managers informed us they were unwilling to accept
the product manager's request to trade requirements that was
prompted by his concerns that the technologies could not advance
in time to meet the program's schedule. They believed the product
manager to be too risk averse and said they would not take no for
an answer. Not only did the user consider the technologies
nontradable, they became even more confined by weight and cost
restrictions that were placed on the program. For example, a more
mature FLIR technology that could possibly meet performance
requirements but also weighed more was rejected. Page 39
GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
Affect How Well a Technology's Inclusion on a Product Can Be
Managed The technological solutions that could meet the strict
requirements were limited. According to Army officials, the only
viable option was to develop the new technologies, which were in a
very immature state, to the required performance levels because no
suitable back-up technologies existed. When the Comanche
acquisition program was launched, the FLIR and integrated avionics
technologies had a TRL 3, barely demonstrated in a laboratory.
This level placed the burden on the Comanche program manager to
complete their development during the acquisition program. The
only ways for the program manager was to slip the schedule or
increase development costs. Figure 3.2 shows an early model of the
integrated avionics component for the Comanche. Figure 3.2:
Integrated Avionics for Comanche Helicopter The Army launched the
Comanche program with immature technologies, placing the burden on
the program manager to complete technology development Source:
DOD. Page 40
GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
Affect How Well a Technology's Inclusion on a Product Can Be
Managed Similarly, the acoustic sensor technology on the BAT was
critical to the submunition's performance because it provided
breakthrough improvements in the capability for precision attack
of targets at ranges of up to 500 kilometers and in most weather.
There was no flexibility for the program manager to ease
requirements to substitute a more mature technology because the
Army had no existing capability to perform this mission. Thus, the
technology, which had a TRL 2 at program launch, was the only
solution for locating and acquiring targets. Its feasibility was
based on an engineering analysis in the form of studies. Key
challenges for the acoustic sensor were to reduce noise to an
acceptable level, develop microphones with sufficient range, and
reduce the size of the sensor so it would fit into the BAT
delivery system. The technology development that was necessary to
have the sensor meet requirements had to be accomplished during
the schedule-driven, delivery-oriented product development
program. The development program encountered technical problems
that left the program manager with no choice but to slip the
schedule and increase the cost. By the start of the engineering
and manufacturing development phase, program officials stated that
the acoustic sensor had a TRL 5-still a high risk using the Air
Force Research Laboratory's criteria. Good Technology
With the right environment as a precondition, managers on the
successful Handoff Decisions cases benefited from
disciplined technology development processes that linked the
technologies to products and provided credible information on
Depend on the Tools the status of technologies. They
also had standards that were both clear and Authority Given to
and high for assessing readiness. Once a technology's feasibility
and Managers usefulness were demonstrated,
it was linked to a product through an early agreement with the
product developer to use it if it could be fully developed.
Ideally, as technologies approached the higher readiness levels
associated with the bridge, S&T managers and receiving product
managers agreed to more specific terms for accepting or rejecting
a technology. These agreements were early links to the product
that were needed for the technology to succeed. If a product
manager was not willing to make such an agreement, then the
investment to bring the technology to higher readiness levels
might not be made. S&T managers were responsible for ensuring that
information at key junctures was sufficient and that the
technology was ready for inclusion on a product. They saw their
role as to screen and develop technologies to standards acceptable
to product managers. Product managers were responsible for
ensuring that the product could be developed and brought to market
within cost and performance targets. They saw as their role to
encourage the successful Page 41
GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
Affect How Well a Technology's Inclusion on a Product Can Be
Managed development of new technology but to decline the handoff
if it did not meet product performance, cost, and schedule
requirements. When an S&T organization disengages from a
technology at a low TRL, the S&T manager gives up much of the
ability to be a gatekeeper. In the event that unyielding
requirements or other pressures force product managers to accept
technologies before they have matured, they are weakened in their
ability to safeguard the product development from technology
risks. For the cases in which technologies had problems
transitioning to products, decisionmakers were disadvantaged by
the incomplete information available to them, yet were not
empowered to say no to the handoff. Their situation was further
degraded by risk assessments that embodied lower standards for
accepting undemonstrated technology readiness. In the case of the
BAT, the S&T community was bypassed altogether, as the weapon
system and its enabling technologies were proposed by a contractor
and assigned directly to a program manager. Successful Cases
Benefited All new technologies at Ford, regardless of whether
they are proposed by From Strong Gatekeepers, inside or
outside sources, take essentially the same path and gates into
Disciplined Processes, and products. Initially, technology
proposals pass through a process that High Maturity Standards
prioritizes them according to customer needs. The proposals are
then passed on to the Advanced Vehicle Technology Office, an S&T
organization that determines the readiness of the proposed
technology and fits it into Ford's path of technology
demonstration. Once approved, the technology follows a structured
process that includes two development phases: concept ready and
implementation ready. This process results in a smooth transition
from the technology development environment into a product, once
the technology is mature. Ford's adaptive cruise control
technology went through this process, as shown in figure 3.3. Page
42 GAO/NSIAD-99-162 Best
Practices Chapter 3 Controllable Conditions Affect How Well a
Technology's Inclusion on a Product Can Be Managed Figure 3.3:
Process for Closing the Gap Between the Readiness of Adaptive
Cruise Control Technology and Jaguar Requirements Technology
Bridge between technology Product feasibility
and product development Advanced vehicle
technology office Jaguar
vehicle team Enabling technologies Interest expressed
Agreement signed Product launch Model
introduced pursued in the technology by vehicle team
between S&T and base
vehicle team TRL 5 TRL 7
TRL 8 1993 1995 1996
1997 1999 According to Ford officials,
technologies for adaptive cruise control existed as separate
projects in the technology base from about 1993 to 1995, when the
Jaguar vehicle team identified a strong demand for the capability.
Ford's S&T community inventoried the ongoing projects and
demonstrated the technology as a laboratory breadboard-a TRL 5. By
August 1996, the technologists had built a prototype that could
demonstrate the technology in a relevant environment-a TRL 7. This
work comprised the concept ready phase, in which the technology
was taken from concept to where its feasibility was demonstrated
to potential users. At the end of this phase, S&T representatives
proved that it could work, and cost, schedule, and performance
targets were established. Also, a target product and sponsor were
identified, linking the technology to a product. The sponsor
agreed that it would accept this technology if specific cost,
quality, schedule, and performance targets were met. The medium
for this acceptance was the Deliverables Agreement Log (DEAL),
which was signed in September 1996 by Jaguar's chief engineer
based on the prototype demonstration. Ford uses the DEAL as a tool
to maintain visibility over a new technology as it progresses
through the development process and to assess its readiness and
acceptability for inclusion in a vehicle program before handing it
to a sponsor, the vehicle center, or team. The DEAL formalizes
Page 43
GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
Affect How Well a Technology's Inclusion on a Product Can Be
Managed the content of the two development phases and establishes
agreements between the technologists managing the project and
those with authority to accept the technology into a product.
According to Ford officials, the DEAL is important to this process
because it is a contract between the parties that addresses the
technology's performance, cost, quality, weight, producibility,
and maintainability targets that must be met before the end of
each phase. It has been invaluable in getting parties to agree on
what is expected by the giver and receiver of a technology during
the process. Once these targets are established, the technology
moves to the implementation ready phase. For the Jaguar, the
Advanced Vehicle Technology Office matured the technology to a
high level of readiness by prototyping it in demonstrator
vehicles-a TRL 8. The technology passed the implementation ready
milestone in February 1997. At that point, the vehicle team
accepted the technology for inclusion on a Jaguar product
development. Ford used this decision-making process to develop the
night vision technology, but with a different result. Since 1991,
Ford has been working on this technology to provide a wide field
of view and depth perception for the driver at night, similar to
that provided by a FLIR. By 1998, the Advanced Vehicle Technology
Office brought the technology to a TRL 8. However, the vehicle
center did not agree to include the technology on a product
because the technology did not meet the cost targets established
in the DEAL. Other companies we visited had similar practices for
supporting technology inclusion decisions. For example, 3M takes
technology from its technology base when it believes it has a
customer need. The gatekeeper responsible for moving technology
into a concept phase-analogous to TRL 3 or 4-is the S&T
organization of a business unit. That business unit monitors the
technology's progress until a new product requirement is
identified and decides whether there is interest from a product
center to "pull" it. If an interest exists, it begins a
feasibility phase that refines requirements through quality
functional deployment and builds working prototypes of the new
product-a stage that would be analogous to TRL 7 or 8. This phase
culminates with an agreement between the technologists and the
product developers-the receivers-as to the specific cost and
schedule targets that must be met for the technology to be
included into a product. To help facilitate the transition, 3M
establishes a product development team that includes people from
research and development, Page 44
GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
Affect How Well a Technology's Inclusion on a Product Can Be
Managed marketing, manufacturing, and other functions that
transfer with the new technology and ensure it is integrated into
the new product. 3M also has high standards for measuring the
readiness of a technology before the product developer accepts it.
For example, 3M officials told us that they are developing a fuel
cell technology for which they have built 15 prototypes for
testing purposes-a TRL 7 or higher. However, because the
technology has not yet met all of the cost, schedule, and
performance targets for product development, they have not allowed
it to be included on a new product, despite demand from the
marketplace. Among the DOD cases, the process followed and the
roles played on the AAAV program had several features that enabled
good technology inclusion decisions. For almost 3 decades, the
Marine Corps has stated a need for an amphibious vehicle with far
greater capabilities than the current vehicle. Specifically, the
requirement to achieve a speed of 20 to 25 knots in the open ocean
made advances in propulsion technology key enablers for the AAAV
program. For a vehicle of the planned size and weight of the AAAV,
this requirement meant achieving 2,700 horsepower with a
relatively compact engine that must operate on land and in water.
The Corps had been exploring propulsion technologies for such a
vehicle in its technology base for many years. Despite this, the
Office of Naval Research, an S&T organization, assessed the
propulsion technology and advised that it was not mature enough to
warrant inclusion on a program. Based on this assessment, Marine
Corps and Navy decisionmakers delayed program launch from 1991 to
1995, until the technology could be brought to higher readiness
levels. Figure 3.4 illustrates the process used to transition this
technology. Page 45
GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
Affect How Well a Technology's Inclusion on a Product Can Be
Managed Figure 3.4: Process for Closing the Gap Between the
Readiness of Propulsion Technologies and AAAV Requirements
Technology Bridge between
Product feasibility technology and program
development Office of Naval Research
AAAV program office Engine proof Agreements reached
between Program launched in
Prototype of concept S&T and program office
program definition engine demonstrated
phase demonstrated TRL 3
TRL 6 TRL 7 1988
1991 1995
1999 The S&T community and the product managers agreed on what had
to be done before the program could be launched. The S&T community
then took the lead in maturing the engine to a TRL 6-a level the
Air Force Research Laboratory considers acceptable for starting
the program definition and risk reduction phase. Thus, the
assessment by the Office of Naval Research provided both the
information and the criteria that enabled decisionmakers to say no
to launching the program given the low readiness of the propulsion
technology. This was coupled with the flexibility to wait for the
technology to mature and the decision to give an S&T organization
responsibility for managing the bridge to product readiness.
Figure 3.5 shows the AAAV. Page 46
GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
Affect How Well a Technology's Inclusion on a Product Can Be
Managed Figure 3.5: AAAV The Marine Corps and Navy delayed
program launch by 4 years to develop key technologies to a higher
readiness level Source: DOD. Even with an urgent need for the
AAAV, the Marine Corps remained disciplined in its development
approach, allowing the technology to mature to the level of the
requirement. Two years before program launch, a Navy S&T
organization demonstrated the technology in a full-scale prototype
engine. By program launch in 1995, the required 2,700 horsepower
was demonstrated by a near prototype engine-a TRL 6. The remaining
risk was limited to marginal weight and size reductions, although
the demonstrator engine could be used as a backup if the size and
weight reductions could not be obtained. In early 1999, the AAAV
program office demonstrated a Page 47
GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
Affect How Well a Technology's Inclusion on a Product Can Be
Managed prototype engine at 2,700 horsepower that met size and
weight requirements-a TRL 7. Technology Handoffs Were
In the BAT program, neither the S&T community nor the product
manager Compromised When had the
opportunity to act as a gatekeeper between product requirements
Managers Had Limited and the maturity of
enabling technologies. All of the technologies for the Information
and Authority BAT came to the program after the
contractor, in 1985, had proposed a weapon concept for carrying
out unmanned, deep strike missions to attack enemy armored
vehicles. Army leadership accepted the concept and drafted
requirements for the BAT, and the acquisition program was launched
after the proposal was accepted. Thus, the technology for the
weapon came directly from the contractor's technology base into
the acquisition program, with little or no review by the Army's
S&T organization. The process, information, and standards that
were critical to successful technology inclusion decisions in
other cases were not employed on the BAT. The process followed is
shown in figure 3.6. Figure 3.6: Assimilation of New Technology
Into the BAT Program Technology maturation Technology concept
and product development Contractor technology base
BAT program office Program launched in
Start engineering, program definition phase
manufacturing and development phase TRL 2-3
TRL 5 1985 1991 The
program office accepted the acoustic sensor, infrared seeker, and
navigation technologies included on the BAT program. In
retrospect, the levels of demonstration at the time posed high
risks to the product development because the acoustic sensor
technology had a TRL of 2 and Page 48
GAO/NSIAD-99-162 Best Practices Chapter 3 Controllable Conditions
Affect How Well a Technology's Inclusion on a Product Can Be
Managed the infrared seeker and navigation technologies had TRLs
of 3. Program officials stated that a significant amount of
technology development was required during product development due
to the lack of visibility over technology readiness before program
launch. As a result, the development program's cost and schedule
significantly increased over original estimates. An interesting
sidelight to the BAT experience concerns the inertial measurement
unit, a navigation component of the submunition. When the
contractor first proposed the BAT concept, the design included a
mature inertial measurement unit in production on other systems.
However, after the program was launched, the contractor
substituted a new quartz rate technology. At the request of the
BAT program manager, the Army's Missile Research and Development
Engineering Center, an S&T organization, assessed the maturity of
the quartz rate technology. The Center concluded that the new
technology had not demonstrated a high enough level of readiness
and recommended that a more proven existing technology be used in
the program. Eventually, the new technology was dropped, and an
existing technology that was at a higher readiness level was used.
We observed additional cases in which decisionmakers relied on
comparatively low standards for including technologies. The Army
assessed the FLIR, integrated avionics, and helmet mounted display
technologies as having moderate risk when they were included in
the Comanche program. Army officials stated that they required
only the existence of an ongoing S&T technology project as
acceptable, as long as the technology was projected to be ready by
the engineering and manufacturing development phase. According to
program officials, demonstrated maturity was considered but not
required; proof that the projects were progressing as scheduled
was enough. These technologies, however, had TRLs of 3 at the time
of launch-a high risk for the program definition and risk
reduction phase. This risk assessment is more consistent with the
actual experience of the technologies' maturation in the program.
The standards used for accepting the laser technology into the ABL
program also appeared low when compared with the standards used on
the more successful cases. While the Air Force had established
demonstration standards for the laser to meet prior to program
launch, these standards were met if scale models of the laser
technology in a laboratory demonstrated they had the potential to
produce the energy needed for an operational system. This level of
technical demonstration equated to a TRL of 4, representing a high
risk for inclusion into an acquisition program. Page 49
GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
Best Practices for Technology Inclusion in DOD Are Surmountable
Chapter 4 Although product developments-commercial or defense-fare
better when key technologies are matured before they are included
in the product design, the more traditional approach within DOD is
to mature technology during a product's development. Rational
explanations are behind this tradition. S&T organizations,
operating within fixed budget levels, are not necessarily
accustomed or equipped to manage the bridge between technology
feasibility and product readiness. Programs are more able to
command the large budgets necessary for reaching higher levels of
technology readiness than S&T projects. Also, pressures are
exerted on new programs to offer unique performance and acceptable
cost and schedule projections, which encourage premature
acceptance of unproven technologies. The Under Secretary of
Defense for Acquisition and Technology not only supports shorter
cycle times and a more aggressive pursuit of technology outside of
programs, but also use of commercial best practices to get these
results. DOD has several initiatives underway that could make
conditions more favorable for S&T organizations to mature a
technology further before it is included in a product development.
One Army project calls for an S&T organization to manage all
technology maturation and integration tasks for a new combat
vehicle up to the engineering and manufacturing development phase.
Other initiatives may make the S&T community a more integral
participant in matching user requirements with technology and
tying S&T projects more closely to product development paths.
Whether these efforts are effective and can be applied on a
broader scale remains to be seen. Several Factors Make Budgetary,
organizational, and other factors within DOD make it difficult to
It Difficult to Mature bring technologies to high
readiness levels before being included in weapon systems. These
factors encourage S&T organizations to disengage Technologies
Before from technology development too soon and
weapon system program They Are Included on managers to accept
immature technology. Factors other than these Weapon Systems
encourage leading commercial firms to keep technology development
out of the product developers' hands and in those of S&T
organizations. The differences in these factors and in the
management of technology development stem from differences in what
helps commercial and DOD programs to succeed. They do not stem
from capabilities commercial firms possess that DOD does not. Page
50 GAO/NSIAD-99-162 Best
Practices Chapter 4 Impediments to Adopting Best Practices for
Technology Inclusion in DOD Are Surmountable Budget and
Organizational Budget realities within DOD-the fact that
weapon system programs Factors attract
higher levels of funding than S&T projects-make these programs a
more advantageous setting for funding technology development to
the higher readiness levels. As a practical matter, it is often
necessary to move immature technology to a weapon system program
to get needed funds and management support for maturation.
Normally, DOD S&T organizations do not see their role as going
beyond demonstrating the feasibility of a technology for generic-
versus product specific-application (a TRL 5). However, as seen in
several of the cases we reviewed, even this level often is not
reached before a product development organization takes over. The
S&T organizations that helped to bridge the gap from technology
feasibility to product readiness on the more successful cases had
gone beyond their typical role. One of the reasons that S&T
organizations disengage relatively early is that S&T work is
traditionally funded as a percentage of the overall DOD research
and development budget. S&T organizations receive about $8 billion
annually, or about 20 percent, of DOD's research and development
budget. This money funds several thousand projects, providing less
than $1 million per project on average. As a result, a project
needing $100 million or more to mature technology to higher
readiness levels than normal-not unreasonable sums-would command a
fairly large share of an S&T organization's budget, thereby
reducing funds available for other projects. Under the current
scenario, the remaining 80 percent of DOD's research and
development funds, approximately $30 billion, is spread out over a
much smaller number of specific weapon programs. A typical weapon
system program can receive several hundred million dollars
annually and occasionally over $1 billion to fund development. A
major program, such as the F-22, can command $15 billion or more
in total for product development, receiving sometimes more than $2
billion in a year. Events on the Air Force's ABL program
illustrate these realities. Originally, the Air Force had planned
the ABL as a technology development project to be managed to high
readiness levels by an S&T organization. The project was started
in 1992 as an advanced technology transition demonstration to
design, fabricate, and test a single demonstrator weapon system
and was to take 8 years to complete. The pacing technologies, the
laser and the beam control, were to be matured to a high level-
equivalent to TRL 6 or 7- before being included in a product
development program. Requirements had not been fixed. In other
words, the planned approach resembled what Page 51
GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
Best Practices for Technology Inclusion in DOD Are Surmountable we
have described as the more successful cases in our review. Figure
4.1 shows the ABL. Figure 4.1: Airborne Laser The Air Force used
relatively low readiness level standards to include a key
technology into ABL Source: DOD. In 1996, the Air Force abandoned
this approach and decided to launch ABL as a weapon system
development program, not because technologies were sufficiently
mature but because of funding and sponsorship concerns. At this
time, the two key technologies were at TRLs 3 and 4. According to
the retired manager of the S&T project, a product development
program was deemed necessary to make the technology development
effort appear real to the users and not a scientific curiosity.
Within the Air Force, the perceived lack of support by the users
placed the project in a constant state of funding jeopardy. This
perception was important because the S&T project was costly, with
a total estimated cost of $800 million, with some annual funding
requirements approaching $200 million. The annual funding
requirements would encompass a large percentage of the Air Force's
S&T budget unless additional funds were made available from weapon
system budgets or elsewhere. By transitioning to a weapon system
program linked to user requirements, the ABL was more likely to
get these funding levels. This approach was successful-the program
won user support and the desired funding. However, sacrifices were
made in technology development. According to the former project
manager, the new program focused less on the elemental technology
hurdles and more on meeting all Page 52
GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
Best Practices for Technology Inclusion in DOD Are Surmountable
user requirements. More expensive demonstrations were necessary to
meet these broader requirements without necessarily doing more to
demonstrate basic technology readiness. It became a more
traditional program with technology and product development
proceeding at the same time, with attendant higher risks. In March
1999, we reported that, while the ABL has made progress in
developing these technologies, it still faced technical
challenges.1 Other Incentives Pressures exerted on weapon
system programs can make it advantageous to include in their
design immature technologies that offer significant performance
gains. One traditional source has been the perceived threat. Users
can demand performance improvements that necessitate the
application of unproven technologies, particularly when a fielding
date is mandated, to stay ahead of the threat. Another source is
technologists, whether from S&T organizations or contractors, who
see a new weapon system as an opportunity to apply a new
technology. Also, the competition for funds can encourage
performance features-and requisite technologies-that distinguish
the new weapon system from competitors. The F-22 was justified as
being faster, stealthier, and more lethal than other fighters,
such as the F-15 and F-117, were. As a result, the F-22 is being
designed with several advanced technologies, including a very
sophisticated suite of avionics that is critical to its
performance features that distinguish it from the other fighters.
However, at the time the F-22 program was launched in 1986, the
avionics technologies were immature; they have since been a source
of problems on the program. We recently reported that the
development of the F-22's integrated avionics systems continues to
experience cost growth and schedule delays, more than 12 years
into the program.2 A different set of incentives causes leading
commercial firms to make their S&T organizations responsible for
maturing technologies to higher readiness levels. Commercial firms
are aware of the risks associated with the high investment that
product development requires. They have a strong incentive in the
realization that if a product is late, costs more, or performs
1Defense Acquisitions: DOD Efforts to Develop Laser Weapons for
Theater Defense (GAO/NSIAD-99-50, Mar. 31, 1999). 2F-22 Aircraft:
Issues in Achieving Engineering and Manufacturing Development
Costs (GAO/NSIAD-99-55, Mar. 15, 1999). Page 53
GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
Best Practices for Technology Inclusion in DOD Are Surmountable
less than expected, the customer could walk away from the product
and the investment would be lost. Minimizing the possibility of
technology being the cause of such problems is thus a top
priority. Having their S&T organizations reduce those risks is
essential to putting product developments in the best position to
succeed. DOD does not have the same incentives. DOD programs are
not penalized if a product is late, costs more, or performs less
than expected, because the customer does not walk away. Services
Encouraged Over the past several years, DOD has encouraged
the services to use best to Use Best Practices practices to
streamline the current process for acquiring new weapon systems in
order to make them faster, cheaper, and better. Shorter
acquisition cycle times are seen as critical to making the best
use of advances in technology. To encourage change, DOD has set a
goal to reduce the average acquisition cycle time for all program
starts in fiscal year 1999 and beyond by 50 percent over
historical averages. DOD has several initiatives to improve its
technology development process and to move technologies to the
warfighter faster and less expensively than the traditional means.
The initiatives also attempt to put the organizations and funding
in place to bring technologies to higher readiness levels before
they are included in programs. These initiatives-defense
technology objectives, advanced technology demonstrations, and
advanced concept technology demonstrations-call for S&T
organizations to play a bigger role in managing technologies
closer to the point of product readiness, matching requirements to
technology projects, and making better use of demonstration
standards. Defense Technology Defense technology objectives
(DTO) are used to bring more discipline to Objectives
S&T projects and to link them more closely with weapon system
development programs. A DTO typically involves a particular
technology advance, such as high temperature materials for turbine
engines and high fidelity infrared sensors. It can also group
several technologies into a larger demonstration. Each DTO
identifies a specific technology advancement that will be
developed or demonstrated, the anticipated date of the technology
availability, the ultimate customer, and the specific benefits
resulting from the technology. It places a corporate attention and
commitment on the technology project by having the technologists,
product developer, and customer involved in the project. According
to DOD, the focus of its S&T investment is enhanced and guided
through DTOs. Each DTO must go through a formal review and
approval Page 54
GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
Best Practices for Technology Inclusion in DOD Are Surmountable
process within DOD and must be directly related to advancing the
operational concepts depicted in DOD's "Joint Vision 2010"
planning document. According to DOD officials, those requirements
have helped to eliminate instances in which technologists work on
projects of particular interest to them, but with no military
application, because the projects should be linked to a specific
warfighter need. For fiscal year 1999, DOD established
approximately 350 DTOs, which accounted for $3 billion, or less
than 50 percent, of the funds DOD had allocated to S&T projects.
The remaining funds were allocated to projects under the
jurisdiction of each military service or other defense agencies
and did not go through the same review and approval process.
Advanced Technology Advanced technology
demonstrations (ATD) are intended to more rapidly Demonstrations
evolve and demonstrate new technologies so they can be
incorporated into a product, if warranted. An ATD has four
characteristics that distinguish it from a conventional S&T
project. They (1) require large-scale resources; (2) involve the
user; (3) use specific cost, schedule, and performance metrics;
and (4) identify a target product for inclusion. An ATD is managed
by an S&T organization and should conclude with an operational
demonstration of the potential capabilities of the technology,
equating to a TRL 5 or 6. The original approach to the ABL was
essentially an ATD approach. Most ATDs use laboratory hardware to
demonstrate the potential capability of nonproduct specific
technologies and not prototype hardware. If the technology is
determined to be feasible and provides some military use, then it
may proceed to the program definition and risk reduction phase of
an acquisition program. From that point, the product developer
completes the technology development for a specific product.
Advanced Concept In 1994, DOD initiated Advanced
Concept Technology Demonstrations Technology Demonstrations
(ACTD) to help expedite the transition of mature technologies from
the developers to the warfighters. ACTDs are intended to help the
DOD acquisition process adapt to budget constraints while
developing technology more rapidly. The purpose of an ACTD is to
assess the military use of a capability, such as a weapon,
comprised of mature technologies. Typically, ACTDs last 2 to 4
years and consist of building and demonstrating a prototype to
provide a warfighter the opportunity to assess a prototype's
capability in realistic operational scenarios. From this
demonstration, the warfighter can refine operational requirements,
develop an initial concept of operation, and determine the
military use of the technology before it proceeds to the product
development process. Page 55
GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
Best Practices for Technology Inclusion in DOD Are Surmountable
According to DOD, ACTDs, which are managed by S&T organizations,
will be a key mechanism to ensure technology development is
separated from product development. In related work on unmanned
air vehicles, we found that ACTDs provided decisionmakers credible
data that they used to terminate efforts or transition the
demonstrator to an acquisition program. In these cases, ACTDs put
decisionmakers in a better position to be gatekeepers. However, we
have reported that the ACTD program needs to be improved.3 We
found that DOD's process for selecting program candidates does not
include adequate criteria for assessing the maturity of proposed
technology and has resulted in the approval of projects that
included immature technologies. We found that the use of specific
criteria for determining maturity was a best practice in the most
successful technology development cases we examined. Two Unique
DOD Two DOD projects are using S&T organizations to
manage technology Projects May Provide development to
higher readiness levels. One, the Army's Future Scout and Cavalry
System, is using a modified ATD to mature technologies and make
Lessons on How to performance trade-offs in the more
flexible environment provided by Enable S&T S&T.
The other is a joint government and industry program, which the
Air Organizations to Force Research Laboratory is
managing to reduce the risks associated with new jet engine
technologies. These projects may provide insights on how Manage
Technology S&T organizations could routinely play a
bigger role in maturing Further technologies
enough for safe inclusion on weapon system programs. They may also
clarify the concern that playing a bigger role in technology
maturation could cause S&T organizations to do less basic research
and technology development. Future Scout and Cavalry In fiscal
year 1997, the Army began piloting a variation of an ATD that is
System designed to help bridge the gap
between technology development and product development by
expanding the S&T community's role in managing technologies
further into the development cycle. The Army's initiative, called
Fast Track, is intended to reduce cost and cycle time by bypassing
the program definition and risk reduction phase of the DOD
acquisition process. The Army is testing this concept with its
Future Scout and Cavalry System project. In this project, the Army
will design, develop, and build a demonstrator vehicle to show the
technical feasibility of the weapon. All of 3Defense Acquisition:
Advanced Concept Technology Demonstration Program Can Be Improved
(GAO/NSIAD-99-4, Oct. 15, 1998). Page 56
GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
Best Practices for Technology Inclusion in DOD Are Surmountable
these tasks will be done under the management of the Army's S&T
organization. The Army believes that a more extensive S&T project
will make the program definition phase unnecessary and estimates
that this concept will reduce the development process by as much
as 4 years and save about $400 million. We did not review the
Future Scout project in terms of its affordability, feasibility,
or any impacts it may have on the Army's S&T budget. Figure 4.2
compares the Fast Track development process with the traditional
approach. Figure 4.2: Comparison of Traditional Technology
Development Process With the Army's Fast Track Approach Technology
maturation Technology feasibility and
product development Traditional S&T organization
Weapon system program office approach
9 years Program launched in
Production program definition phase TRL 3-5 Bridge built to
Technology feasibility
Technology maturation increase technology and
product development maturity Future Scout Army S&T
organization Weapon system
program office and Cavalry System
4.5 years Program office Program launch
Production staff joins S&T in engineering, project
office manufacturing and development phase. TRL 6 While
we do not necessarily agree that the first phase of the
acquisition cycle can be omitted, so far the Future Scout project
is emulating technology development practices like those we
observed in the successful cases. First, it has established
demonstration criteria that must be met before the technology
enters product development. Second, it has also Page 57
GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
Best Practices for Technology Inclusion in DOD Are Surmountable
established forums that involve key players on the technology path
to keep them informed of the technology's development progress.
For example, the acquisition program manager will be integrated
into the development project during the final 1.5 years of the S&T
program. This should provide a good link between the technology
development and product development, allowing the program manager
to fully understand the technology before product development
begins. Finally, by allowing an S&T organization the flexibility
to manage technologies further into the development cycle, Army
officials believe they will be able to make trade-offs among cost,
schedule, and performance requirements before program launch,
without raising concerns about the state of the project or
breaching baselines that had been set without enough knowledge.
While this concept comes closer to the most successful technology
development cases we reviewed, it still embodies greater technical
risk. The Army expects to demonstrate some performance
capabilities of the vehicle before the product development phase
begins. However, the demonstrator vehicle will only be about 75
percent of a complete prototype, which means some key technologies
will not be demonstrated to high readiness levels before that
phase begins. Nonetheless, the project manager equated the
expected overall technical maturity of the vehicle at transition
to a TRL 6. The Army considers this a medium or acceptable level
of risk, and it is willing to enter product development with some
immature technologies. If, however, product development begins at
engineering and manufacturing development, this risk could be
assessed as high, based on TRLs. Figure 4.3 shows an artist's
concept of the Future Scout and Cavalry System. Page 58
GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
Best Practices for Technology Inclusion in DOD Are Surmountable
Figure 4.3: Future Scout and Cavalry System An Army S&T
organization is maturing technologies and proposing performance
trade-offs for the Future Scout and Cavalry System before program
launch Source: DOD. Integrated High The
Integrated High Performance Turbine Engine Technology program-a
Performance Turbine joint government and industry
effort-is focused on developing Engine Technology Program
technologies for more affordable and higher performance turbine
engines for both missiles and aircraft. It is a technology
validation program and is managed by an S&T organization to
perform demonstrations of various engine technologies to higher
readiness levels than most S&T projects. After the demonstrations,
the technologies enter a product development program. The program
takes the technology through a series of tests that range from
individual component tests to full-scale engine demonstrations.
The program has established strong links with the acquisition
programs for which the technologies are intended. For example, Air
Force Research Laboratory officials informed us that they
established formal technology transition plans with the F-22 and
Joint Strike Fighter programs that document agreements on what
technology development activities will be performed to support the
programs. Representatives from each program office are invited to
all technology demonstrations and are kept informed about
demonstrated progress. Page 59
GAO/NSIAD-99-162 Best Practices Chapter 4 Impediments to Adopting
Best Practices for Technology Inclusion in DOD Are Surmountable As
part of the program, the Air Force developed a set of standards to
assess the readiness levels of technologies similar to NASA's
TRLs. According to the Air Force, the S&T organization uses these
standards to determine when the project has been completed.
These standards were the first application of readiness levels by
the Air Force Research Laboratory. There are five technology
readiness levels ranging from component-level tests in a
laboratory to the highest level involving actual flight tests of
engines. The program typically does not take technologies to the
highest readiness level (flight test) because of the high cost.
The program stops when it has been determined the technology is
well defined within acceptable boundaries and a good correlation
exists between test results and engineering predictions. This
readiness level would translate to a TRL of 5 or 6, as used in
this report. The final step of the technology development is left
to the product developer who determines if the technology can be
packaged and integrated into the final product. Page 60
GAO/NSIAD-99-162 Best Practices Chapter 5 Conclusions and
Recommendations
Chapter 5 Conclusions Clearly, DOD's continued advancement of
new technologies is essential to the continued superiority of its
weaponry. The leading edge military capabilities the United States
possesses today, such as stealth aircraft, precision munitions,
and intelligence-gathering satellites, bear witness to the effects
of such technical advances. At the same time, the incorporation of
advanced technologies before they are mature has been a major
source of cost increases, schedule delays, and performance
problems on weapon systems. As DOD contemplates increasing its
annual investment in new weapons to $60 billion, the expectations
on program managers are great: they must develop and field weapons
of superior capability more quickly and less expensively than in
the past. The way advanced technologies are matured and included
in weapon systems will play a central role in meeting these
expectations. Although different ways to better assimilate new
technologies into weapons are legitimate topics for debate, that
it has to be done better is not. The leading commercial firms'
practices have produced results that resemble those sought by DOD:
more technically advanced, higher quality products, developed in
significantly less time, and less expensively than their
predecessors. Managing the development of advanced technology
differently--and separately--from the development of a product has
been key to these results. The firms insist that advanced
technology reach a high level of maturity, the point at which the
knowledge about that technology is essentially complete, before
allowing it into a product development. By separating the two, the
firms lessen the product manager's burden and place that person in
a better position to succeed in delivering the product. These
practices may not necessarily accelerate the pace at which
technology matures. In fact, several of the commercial
technologies we reviewed took 10 years or more to get to market.
The clear beneficiaries of the practices are the product
developments, for which the investments are much larger, and time
translates into significantly more resources than in a technology
project. Adapting these practices on its weapon system programs
can help DOD to reduce costs and the time from product launch to
fielding, and use technology advances as they become available
more frequently. Separating technology development from product
development calls for a new approach to managing technology
development. Two conditions are essential to such an approach.
First, the right environment for maturing technologies must exist.
A practice that is instrumental in providing this environment is
maturing technology to achieve product readiness before it Page 61
GAO/NSIAD-99-162 Best Practices Chapter 5 Conclusions and
Recommendations is constrained by the rules of an acquisition
program. In the successful DOD cases we reviewed, this environment
was provided by S&T organizations or a team of S&T and product
developers who managed technologies to high readiness levels
before they were included in an acquisition program. These
organizations provided an environment more conducive to the ups
and downs normally associated with the discovery process. A
corollary practice is agreeing on what level of knowledge is
needed about a new technology before it is considered for
inclusion in a product design. When that knowledge level does not
exist, the flexibility for S&T organizations and product managers
to either take the time to mature the technology or trade off
product requirements until they can be met with mature technology
is essential. It is a rare program that can proceed with a gap
between product requirements and maturity of key technologies and
still be delivered on time and within costs. Second, S&T and
product managers must be provided with the disciplined processes,
information, standards, and authority to make good handoffs of
technology to product. Prepared with the tools and authority to
make sound handoff decisions, both S&T and product managers can
function as gatekeepers to safeguard the product development from
undue technology risks. Leading commercial firms have adopted this
approach as a matter of necessity and have used the organizations,
tools, and other practices to foster technology development and
improve the outcomes of product developments. The high stakes
stemming from the large investment required for a new product and
the risks if the product does not meet customer needs reinforce
this approach in leading commercial firms. The DOD cases that
followed a similar approach were realizing better program
outcomes, at least in the sense that the programs avoided key
technology development problems. Yet, these cases are not the norm
for DOD programs for several reasons. * More typically, the
commitment to develop and produce a weapon system is made before a
match between technology and weapon system requirements exists. *
DOD programs operate under different conditions that make it more
difficult-and less rewarding-to separate technology development
from product development. * Budget realities make it more
difficult for S&T organizations to carry technologies to the high
readiness levels needed to meet product requirements; such
resources are more available within product developments. Page 62
GAO/NSIAD-99-162 Best Practices Chapter 5 Conclusions and
Recommendations * The pressures to show the unequalled performance
necessary to win funding encourage including promising, but
immature, technologies in weapon system designs. It will take
procedural, organizational, and cultural changes within DOD's
acquisition process to foster an environment in which (1)
technologies can be successfully matured outside the purview of
weapon system programs, (2) programs can be relieved of the
pressures to include immature technologies and the unrealistic
expectations that the technologies will conform to tight cost and
schedule projections, and (3) technology advances will not stall
due to inadequate funding or lack of identification with a product
in the later, more expensive stages of demonstration. Experience
has shown that such an approach can work within DOD on individual
cases. DARPA played a primary role in managing the transition of
the nonpenetrating photonics mast technology to the Virginia class
attack submarine. The Integrated High Performance Turbine Engine
Technology program has carried advanced jet engine technologies to
TRLs of between 5 and 6 for successful inclusion into programs. In
the Future Scout program, an Army S&T organization, augmented by
product development staff, is managing an ATD to lower the risk of
key technologies before a product development program is launched.
However, it remains to be seen whether the Army will be successful
in using large and expensive S&T projects, such as the Future
Scout program, without affecting other Army S&T projects. A
challenge for DOD will be whether the lessons learned from these
individual cases offer an approach that has DOD-wide application.
Meeting this challenge is essential to fielding technologically
superior weapons more quickly and within predicted costs.
Recommendations We have previously recommended that DOD
separate technology development from weapon system programs. That
recommendation was made without prejudice toward the necessity of
technology development but rather with the intent that programs
could be better managed if such development was conducted outside
of a program manager's purview. Similarly, the recommendations
that follow are made without prejudice toward-or the intention of
compromising-the basic research and other activities that S&T
organizations perform. We recognize that implementation of these
recommendations will have organizational, funding, and process
implications and will require the cooperation of the Congress.
Page 63 GAO/NSIAD-99-162
Best Practices Chapter 5 Conclusions and Recommendations To help
ensure that new technologies are vigorously pursued and
successfully moved into weapon system programs, we recommend that
the Secretary of Defense adopt a disciplined and knowledge-based
method for assessing technology maturity, such as TRLs, DOD-wide.
This practice should employ standards for assessing risks of
handoff to program managers that are based on a technology's level
of demonstration and its criticality to meeting the weapon
system's requirements. With these tools in hand, we recommend that
the Secretary (1) establish the place at which a match is achieved
between key technologies and weapon system requirements as the
proper time for committing to the cost, schedule, and performance
baseline for developing and producing that weapon system and (2)
require that key technologies reach a high maturity level-
analogous to TRL 7-before making that commitment. This would
approximate the launch point for product development as practiced
by leading commercial firms. We recommend that the Secretary find
ways to ensure that the managers responsible for maturing the
technologies and designing weapon systems before product
development are provided the more flexible environment that is
suitable for the discovery of knowledge, as distinct from the
delivery of a product. Providing more flexibility will require the
cooperation of requirements managers and resource managers so that
rigid requirements or the threat of jeopardizing the funding
planned to start product development will not put pressure on
program managers to accept immature technologies. Such an
environment may not be feasible if the program definition and risk
reduction phase remains the effective launch point for an entire
weapon system program. An implication of these recommendations is
that S&T organizations will have to play a greater role in
maturing technologies to higher levels and should be funded
accordingly. Therefore, we recommend that the Secretary of Defense
evaluate the different ways S&T organizations can play a greater
role in helping technologies reach high levels of maturity before
product development begins. For example, given that a technology
has sufficient potential for application to a weapon system, at a
minimum, an S&T organization should be responsible for taking a
technology to TRL 6 before it is handed off to a program office at
the program definition and risk reduction phase. During this
phase, the program manager would be responsible for maturing the
technology to TRL 7 before it is included in an engineering and
manufacturing development program. In a situation where a single,
design-pacing technology is to be developed for a known Page 64
GAO/NSIAD-99-162 Best Practices Chapter 5 Conclusions and
Recommendations application-like the nonpenetrating periscope-an
S&T organization should be required to mature that technology to
TRL 7 before it is turned over to a product development manager.
S&T organizations could play a similar role when a significant new
technology is being prepared for insertion into an existing weapon
system. Finally, when multiple new technologies are to be merged
to create a weapon system, S&T organizations should be required to
bring key technologies to TRL 6 and then become part of a hybrid
organization with product developers to integrate the technologies
and bring them to TRL 7 before handing full responsibility to a
product development manager. To help guard against the possibility
that the more basic research and technology development activities
would be compromised by having S&T organizations routinely take
key technologies to TRL 6 or higher, we recommend that the
Secretary extract lessons from the nonpenetrating periscope, the
AAAV, and the Army's Future Scout programs, and other ATD and ACTD
programs. Specifically, the Secretary should assess whether the
resources needed to enable S&T organizations to play a leading
role in the development of technologies and, in some cases,
preliminary system design, detracted from or displaced more basic
research and technology development programs. Finally, we
recommend that the Secretary empower managers of product
development programs to refuse to accept key technologies with low
levels of demonstrated maturity. The Secretary can encourage this
behavior through supportive decisions on individual programs, such
as by denying proposals to defer the development of key
technologies and by favoring proposals to lengthen schedules or
lessen requirements to reduce technological risk early. Agency
Comments and DOD generally concurred with a draft of this report
and its Our Evaluation recommendations, noting
that the traditional path to new weapon system development is no
longer affordable or necessary (see app. I). DOD stated that it
has embarked upon a "Revolution in Business Affairs" that will
enable new technologies to be developed more efficiently and
effectively. It believes that the first steps in this direction
have already been taken but agrees that more progress needs to be
made. DOD agreed that TRLs are necessary in assisting
decisionmakers in deciding on when and where to insert new
technologies into weapon system programs and that weapon system
managers should ensure that technology is matured to a TRL 7
before insertion occurs. DOD concurred that S&T organizations
should be Page 65
GAO/NSIAD-99-162 Best Practices Chapter 5 Conclusions and
Recommendations involved in maturing technologies to high levels,
such as TRL 6, before transitioning to the engineering and
manufacturing development phase and agreed to assess the impact of
this involvement on other S&T resources. We note that the best
practice is to mature technology to at least a TRL 7 before
starting the engineering and manufacturing development phase,
whether the technology is managed by an S&T organization, a weapon
system program manager, or a hybrid of the two organizations. DOD
noted that while TRLs are important and necessary, the increasing
projected life for new weapon systems, total ownership costs, and
urgency based upon threat assessments are also important
considerations for system development decisions. We agree and note
that our recommendations are not intended to cover all aspects of
weapon system development decisions or to suggest that technology
maturity is the only factor in such decisions. Rather, the
recommendations are in keeping with the purpose of the report, "to
determine whether best practices offer methods to improve the way
DOD matures new technology so that it can be assimilated into
weapon system programs with less disruption." We believe that a
knowledge-based approach to maturing technology, such as TRLs, can
benefit other considerations as well. For example, decisions on
what technologies to include in a weapon system and when to
include them can have a significant bearing on its total ownership
costs. DOD stated that there should be an established point for
the transition of technologies and that it plans to supplement its
milestone review process with additional guidance in the next
revisions to DOD Directive 5000.2R. It also stated that its policy
on the evolutionary approach to weapon acquisitions should be
developed in consonance with the technology transition strategy.
We cannot comment on the revisions to the directive or the
evolutionary acquisition policy because they have yet to be
published. However, under the current milestone review process,
the pressures placed on a program during the program definition
and risk reduction phase- when much technology development occurs-
can operate against the flexibility and judgments that are needed
to mature technologies. If the revisions to the directive
supplement the current milestones without relieving the pressures
brought to bear on programs as they are launched in the program
definition and risk reduction phase, it will remain difficult to
discourage the acceptance of immature technologies in the design
of new weapon systems. To relieve these pressures, we encourage
DOD, as it develops the directive and the evolutionary acquisition
policy, to separate technology development from product
development and to redefine the launch point for a program as the
point at which enough knowledge has Page 66
GAO/NSIAD-99-162 Best Practices Chapter 5 Conclusions and
Recommendations been gained to ensure that a match is reached
between the maturity of key technologies and weapon system
requirements. DOD also stated that program managers already have
the ability to reject inappropriately mature technologies, and to
the extent technology immaturity affects acquisition baselines, to
advise acquisition executives of feasible alternatives. We did not
find this to be the case in our review. Rather, we found that the
program managers' ability to reject immature technologies is
hampered by (1) untradable requirements that force acceptance of
technologies despite their immaturity and (2) reliance on tools
for judging technology maturity that fail to alert the managers of
the high risks that would prompt such a rejection. As noted in the
report, once a weapon system program begins, the environment
becomes inflexible and deviations to program baselines can attract
unwanted attention. This reality limits the program managers'
ability to reject immature technologies. Page 67
GAO/NSIAD-99-162 Best Practices Appendix I Technology Readiness
Levels and Their Definitions
Appendix I Technology readiness level
Description 1. Basic principles observed and reported.
Lowest level of technology readiness. Scientific research begins
to be translated into applied research and development. Examples
might include paper studies of a technology's basic properties. 2.
Technology concept and/or application Invention
begins. Once basic principles are observed, practical applications
can be formulated.
invented. The application is speculative and there is no proof or
detailed analysis to support the assumption. Examples are still
limited to paper studies. 3. Analytical and experimental critical
function Active research and development is initiated. This
includes analytical studies and and/or characteristic proof of
concept. laboratory studies to physically validate
analytical predictions of separate elements of the technology.
Examples include components that are not yet integrated or
representative. 4. Component and/or breadboard validation in
Basic technological components are integrated to establish that
the pieces will work laboratory environment.
together. This is relatively "low fidelity" compared to the
eventual system. Examples include integration of "ad hoc" hardware
in a laboratory. 5. Component and/or breadboard validation in
Fidelity of breadboard technology increases significantly. The
basic technological relevant environment.
components are integrated with reasonably realistic supporting
elements so that the technology can be tested in a simulated
environment. Examples include "high fidelity" laboratory
integration of components. 6. System/subsystem model or prototype
Representative model or prototype system, which is well beyond the
breadboard demonstration in a relevant environment.
tested for TRL 5, is tested in a relevant environment. Represents
a major step up in a technology's demonstrated readiness. Examples
include testing a prototype in a high fidelity laboratory
environment or in simulated operational environment. 7. System
prototype demonstration in an Prototype near or at
planned operational system. Represents a major step up from
operational environment. TRL 6,
requiring the demonstration of an actual system prototype in an
operational environment, such as in an aircraft, vehicle or space.
Examples include testing the prototype in a test bed aircraft. 8.
Actual system completed and "flight qualified" Technology has
been proven to work in its final form and under expected
conditions. through test and demonstration.
In almost all cases, this TRL represents the end of true system
development. Examples include developmental test and evaluation of
the system in its intended weapon system to determine if it meets
design specifications. 9. Actual system "flight proven" through
Actual application of the technology in its final form and under
mission conditions, successful mission operations.
such as those encountered in operational test and evaluation. In
almost all cases, this is the end of the last "bug fixing" aspects
of true system development. Examples include using the system
under operational mission conditions. Page 68
GAO/NSIAD-99-162 Best Practices Appendix II Comments From the
Department of Defense Appendix II Page 69 GAO/NSIAD-
99-162 Best Practices Appendix II Comments From the Department of
Defense Now on pp. 7 and 63-64. Now on pp. 7 and 63-64. Now on pp.
7 and 63-64. Page 70 GAO/NSIAD-
99-162 Best Practices Appendix II Comments From the Department of
Defense Now on pp. 7 and 63-64. Page 71
GAO/NSIAD-99-162 Best Practices Appendix II Comments From the
Department of Defense Now on pp. 7 and 65. Now on pp. 7 and 65.
Page 72 GAO/NSIAD-99-162 Best
Practices Appendix III GAO Contacts and Staff Acknowledgments
Appendix I II GAO Contacts Louis Rodrigues, (202) 512-4841
Paul Francis, (202) 512-2811 Acknowledgments In addition to
those named above, Michael Sullivan, Jeffrey Hunter, Matthew Lea,
Maria Santos, Rae Ann Sapp, and Katrina Taylor made key
contributions to this report. Page 73
GAO/NSIAD-99-162 Best Practices Page 74 GAO/NSIAD-99-162 Best
Practices Page 75 GAO/NSIAD-99-162 Best Practices Related GAO
Products Best Practices: Commercial Quality Assurance Practices
Offer Improvements for DOD (GAO/NSIAD-96-162, Aug. 26,1996). Major
Acquisitions: Significant Changes Underway in DOD's Earned Value
Management Process (GAO/NSIAD-97-108, May 5, 1997). Best
Practices: Successful Application to Weapon Acquisitions Requires
Changes in DOD's Environment (GAO/NSIAD-98-56, Feb. 24, 1998).
Best Practices: DOD Can Help Suppliers Contribute More to Weapon
System Programs (GAO/NSIAD-98-87, Mar. 17, 1998). Defense
Acquisition: Improved Program Outcomes Are Possible (GAO/T-NSIAD-
98-123, Mar. 18, 1998). Defense Acquisitions: Best Commercial
Practices Can Improve Program Outcomes (GAO/T-NSIAD-99-116, Mar.
17, 1999). (707336) Letter Page 76
GAO/NSIAD-99-162 Best Practices Ordering Information The first
copy of each GAO report and testimony is free. Additional copies
are $2 each. Orders should be sent to the following address,
accompanied by a check or money order made out to the
Superintendent of Documents, when necessary, VISA and MasterCard
credit cards are accepted, also. Orders for 100 or more copies to
be mailed to a single address are discounted 25 percent. Orders by
mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC
20013 or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts.
NW) U.S. General Accounting Office Washington, DC Orders may also
be placed by calling (202) 512-6000 or by using fax number (202)
512-6061, or TDD (202) 512-2537. Each day, GAO issues a list of
newly available reports and testimony. To receive facsimile
copies of the daily list or any list from the past 30 days, please
call (202) 512-6000 using a touchtone phone. A recorded menu will
provide information on how to obtain these lists. For information
on how to access GAO reports on the INTERNET, send an e-mail
message with "info" in the body to: info@www.gao.gov or visit
GAO's World Wide Web Home Page at: http://www.gao.gov United
States Bulk Rate General Accounting Office
Postage & Fees Paid Washington, D.C. 20548-0001 GAO
Permit No. GI00 Official Business Penalty for Private Use $300
Address Correction Requested
*** End of document. ***
|
NEWSLETTER
|
|
Join the GlobalSecurity.org mailing list
|
|
|
