Joint Strike Fighter Acquisition: Development Schedule Should Be Changed to Reduce Risks (Letter Report, 05/09/2000, GAO/NSIAD-00-74)
Pursuant to a congressional request, GAO reviewed the Department of
Defense's (DOD) Joint Strike Fighter (JSF) Program, focusing on: (1) the
program's acquisition strategy; and (2) whether the strategy is being
implemented in a manner that will ensure that the acquisition strategy
objectives will be achieved.
GAO noted that: (1) the key objective of the JSF acquisition strategy is
affordability--reducing the development, production, and ownership costs
of the program relative to prior fighter aircraft programs; (2) DOD
expects the JSF acquisition strategy to save nearly $18 billion (in
fiscal year 1995 dollars) in development costs; (3) to achieve its
affordability objective, the JSF program office has incorporated various
DOD and commercial acquisition initiatives into the JSF acquisition
strategy; (4) these initiatives include modifying the traditional
weapons acquisition cycle, revising the requirements determination
process, and developing critical technologies to a level where they
represent low technical risk before the engineering and manufacturing
contract is awarded; (5) the expectation is that incorporating these
initiatives into the JSF acquisition strategy will result in a better
match between the maturity of key technologies and the aircraft's
requirements; (6) matching the requirements and the maturity of
technology when a program enters engineering and manufacturing
development is a critical determinant of a program's success; (7) once
the development phase begins, a large, fixed investment in the form of
human capital, facilities, and materials is sunk into the program and
any significant changes will have a large, rippling effect on cost and
schedule; (8) beginning the engineering and manufacturing development
phase when critical technologies are at a low level of maturity serves
to significantly increase program risk and the likelihood of schedule
delays, which in turn result in increased program costs; (9) the JSF
program office's implementation of its acquisition strategy will not
ensure that the JSF program will enter the engineering and manufacturing
development phase with low technical risk; (10) the aircraft being
produced during the concept demonstration phase are not intended to
demonstrate many of the technologies considered critical for achieving
JSF program cost and performance requirements; (11) instead, many of
these technologies--such as avionics, flight systems, manufacturing and
producibility, propulsion, supportability, and weapons delivery
system--will only be demonstrated in laboratory or ground-testing
environments; and (12) therefore, these critical technologies will be at
low levels of technical maturity when the engineering and manufacturing
development contract is scheduled to be awarded.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: NSIAD-00-74
TITLE: Joint Strike Fighter Acquisition: Development Schedule
Should Be Changed to Reduce Risks
DATE: 05/09/2000
SUBJECT: Defense procurement
Defense capabilities
Fighter aircraft
Weapons systems
Weapons research and development
Private sector practices
Operational testing
Procurement planning
Defense cost control
Concurrency
IDENTIFIER: Joint Strike Fighter
DOD Joint Strike Fighter Single Acquisition Management
Plan
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. This text was extracted from a PDF file. **
** Delineations within the text indicating chapter titles, **
** headings, and bullets have not been preserved, and in some **
** cases heading text has been incorrectly merged into **
** body text in the adjacent column. Graphic images have **
** not been reproduced, but figure captions are included. **
** Tables are included, but column deliniations have not been **
** preserved. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** <info@www.gao.gov> **
** **
** with the message 'info' in the body. **
******************************************************************
GAO/NSIAD-00-74
Joint Strike Fighter Acquisition: Development Schedule Should Be Changed to Reduce Risks
A
Report to the Chairman, Subcommittee on National Security, Veterans' Affairs,
and International Relations, Committee on Government Reform, House of Representatives
May 2000 JOINT STRIKE FIGHTER ACQUISITION
Development Schedule Should Be Changed to Reduce Risks
GAO/NSIAD-00-74
Letter 3 Appendixes Appendix I: International Participation in the Joint Strike Fighter
Program 24
Appendix II: Technology Readiness Levels and Their Definitions 25
Appendix III: Comments From the Department of Defense 26
Tables
Table 1: Military Service Needs for the Joint Strike Fighter 6
Figures
Figure 1: Boeing and Lockheed Martin Joint Strike Fighter Aircraft
Design Concepts 7
Figure 2: Comparison of Traditional and Joint Strike Fighter
Acquisition Cycles 8
Figure 3: Joint Strike Fighter Critical Technology Readiness Levels
at Program Start 13
Figure 4: Joint Strike Fighter Critical Technology Readiness Levels
at Program Start and Projected for Entry Into Engineering and Manufacturing Development 14
Abbreviations
DOD Department of Defense JSF Joint Strike Fighter TRL technology readiness level
National Security and International Affairs Division
Lett er
B- 281196 May 9, 2000 The Honorable Christopher Shays Chairman, Subcommittee on National Security, Veterans' Affairs,
and International Relations Committee on Government Reform House of Representatives
Dear Mr. Chairman: The Joint Strike Fighter Program is intended to produce an affordable, next- generation aircraft to replace the Department of Defense's (DOD) aging aircraft inventory. The first aircraft deliveries are scheduled to begin in 2008. As currently planned, the program will cost about $200 billion to develop and procure over 3,000 aircraft and related support equipment for the Air Force, the Marine Corps, the Navy, and Great Britain.
DOD has designated the Joint Strike Fighter Program as a flagship program for acquisition. To date, the program has awarded contracts totaling over $2 billion to Boeing and Lockheed Martin for the current concept demonstration phase. Under these contracts, both contractors will build the aircraft they plan to fly in the demonstration phase and also design the aircraft they plan to build in the next phase of the development program engineering and manufacturing development. During engineering and manufacturing development, the Joint Strike Fighter will be fully developed, engineered, designed, fabricated, tested, and evaluated to demonstrate that the production aircraft will meet stated requirements. DOD is scheduled to award the contract for engineering and manufacturing development to either Boeing or Lockheed Martin in April 2001. 1
At your request, we reviewed the Joint Strike Fighter Program to (1) provide information on the acquisition strategy and (2) to determine whether the strategy is being implemented in a manner that will ensure that the acquisition strategy objectives will be achieved. With your permission, we discussed a draft of this report during a March 16, 2000, joint hearing by
1 At the time of this report, the Under Secretary of Defense for Acquisition, Technology, and Logistics was reviewing competition and industrial- base implications of the Joint Strike Fighter acquisition strategy.
the Subcommittees on Military Procurement and on Military Research and Development, House Committee on Armed Services. At the time of the hearing, we had not received DOD's comments on our report. This report contains DOD's comments and our evaluation of them.
Results in Brief The key objective of the Joint Strike Fighter acquisition strategy is affordability reducing the development, production, and ownership costs
of the program relative to prior fighter aircraft programs. DOD expects the Joint Strike Fighter acquisition strategy to save nearly $18 billion (in fiscal year 1995 dollars) in development costs. To achieve its affordability objective, the Joint Strike Fighter program office has incorporated various DOD and commercial acquisition initiatives into the Joint Strike Fighter acquisition strategy. These initiatives include modifying the traditional weapons acquisition cycle, revising the requirements determination process, and developing critical technologies to a level where they represent low technical risk before the engineering and manufacturing contract is awarded. The expectation is that incorporating these initiatives into the Joint Strike Fighter acquisition strategy will result in a better match between the maturity 2 of key technologies and the aircraft's requirements. Matching the requirements and the maturity of technology when a program enters engineering and manufacturing development is a critical determinant of a program's success. Once the development phase begins, a large, fixed investment in the form of human capital, facilities, and materials is sunk into the program and any significant changes will have a large, rippling effect on cost and schedule. Beginning the engineering and manufacturing development phase when critical technologies are at a low level of maturity serves to significantly increase program risk and the likelihood of schedule delays, which in turn result in increased program costs.
The Joint Strike Fighter program office's implementation of its acquisition strategy will not ensure that the Joint Strike Fighter program will enter the engineering and manufacturing development phase with low technical risk. The aircraft being produced during the concept demonstration phase are not intended to demonstrate many of the technologies considered critical for achieving Joint Strike Fighter program cost and performance requirements. Instead, many of these technologies such as avionics, flight
2 A technology is considered to be mature when it has been developed to a point that it can be readily integrated into a new product and counted on to meet product requirements.
systems, manufacturing and producibility, propulsion, supportability, and weapons delivery system will only be demonstrated in laboratory or ground- testing environments. Therefore, these critical technologies will be at low levels of technical maturity when the engineering and manufacturing development contract is scheduled to be awarded. In addition, when the competing contractors experienced design problems and cost overruns, DOD restructured the program in a manner that will provide less information than originally planned prior to selecting between the two competing contractors. Specifically, this program restructure moves away from best commercial practices that were evident in the original strategy, where technology was being developed ahead of the product. Instead, DOD's approach moves toward the traditional practice of concurrently developing technologies and products, which often raised cost- benefit issues as a result of cost increases and schedule delays as problems are encountered in technology development.
To demonstrate DOD's commitment to acquisition reform, follow best commercial practices, and reduce the risk of future cost growth, the program office should focus on risk reduction efforts by maturing critical technologies prior to entering engineering and manufacturing development, and it should be allowed to do so without the penalty of withdrawal of funding support. We make a recommendation that the Joint Strike Fighter program office adjust its currently planned engineering and manufacturing development decision date of March 2001 to allow adequate time to mature critical technologies to acceptable maturity levels before awarding the engineering and manufacturing development contract.
Background The Joint Strike Fighter is the centerpiece of DOD's tactical aircraft modernization plan, which includes the Air Force F- 22 Raptor and the Navy
F/ A- 18 E/ F Super Hornet. The program is structured to use a common production line to produce three aircraft variants that meet conventional flight requirements for the U. S. Air Force, short take- off and vertical landing characteristics for the Marine Corps, and carrier operation suitability needs for the U. S. Navy. The program will also provide aircraft to the British Royal Navy and Air Force. Table 1 shows current service plans for Joint Strike Fighter use.
Table 1: Military Service Needs for the Joint Strike Fighter Service Quantity Planned use
Air Force 1,763 Replacement for F- 16 and A- 10; complement to the F- 22 Marine Corps 609 Short take- off
and vertical landing aircraft to replace AV- 8B
and F/ A- 18 C/ D Navy 480 Carrier- based, multi- role, first day of war survivable strike
fighter to complement the F/ A- 18 E/ F Great Britain 150 Short take- off and vertical landing replacement for the Sea
Harrier and GR. 7 Source: Joint Strike Fighter program office.
DOD expects the Air Force variant of the Joint Strike Fighter to cost about $28 million per unit;
the Navy variant to be between $31 million and $38 million; and the Marine Corps variant to cost
between $30 million and $35 million. 3 Independent estimates are not so optimistic. For example,
in congressional hearings held in March 1999, the Congressional Budget Office estimated that the
unit cost of the Joint Strike Fighter could be as much as 47 percent to 51 percent higher than
expected, depending on which variant was procured. DOD and the Congressional Budget Office
estimates vary as a result of differing estimating techniques, including estimating the cost
of incorporating stealth technologies into the Joint Strike Fighter design. Figure 1 shows
planned Joint Strike Fighter aircraft designs by contractor.
3 Expected costs include the cost to produce the basic aircraft, propulsion system, and
avionics. Costs are stated in fiscal year 1994 dollars.
Figure 1: Boeing and Lockheed Martin Joint Strike Fighter Aircraft Design Concepts
Boeing Joint Strike Fighter Lockheed Martin Joint Strike Fighter Design Concept
Design Concept Source: Joint Strike Fighter program office.
Acquisition Strategy The focus of the Joint Strike Fighter Program is affordability reducing the
Designed to Achieve development, production, and ownership costs of the program relative to
prior fighter aircraft programs. To achieve this objective, the Joint Strike Affordability Goals
Fighter program office has incorporated various DOD and commercial Through Reduced
acquisition initiatives into the Joint Strike Fighter acquisition strategy. Program Risk
These initiatives include modifying the traditional weapons acquisition cycle, revising the
requirements determination process, and advancing the maturity level of critical technologies
so they represent low technical risk before the engineering and manufacturing contract is
awarded. The expectation is that incorporating these initiatives into the Joint Strike
Fighter acquisition strategy will avoid cost growth, schedule slippage, and performance
shortfalls that have been experienced in other weapons acquisition programs.
Acquisition Strategy The Joint Strike Fighter acquisition strategy modifies the cycle that
Modifies Traditional weapons acquisition programs normally follow. For example, the Joint
Acquisition Cycle Strike Fighter program office was created earlier in the acquisition cycle
than it would have been in a traditional DOD weapons system program. This enabled DOD to obtain
early input from relevant stakeholders (operators, maintenance personnel, industry representatives,
government engineers, and officials within the intelligence community) to ensure that all aspects
of cost, schedule, performance, and resource constraints are included in decision- making. In
addition, the program has encouraged
greater involvement from the international community, which provides both monetary and technical
support (see app. I).
The traditional acquisition cycle has also been modified by combining the first two traditional
acquisition phases Concept Exploration and Program Definition and Risk Reduction into one phase,
known as Concept Demonstration. Under the traditional DOD acquisition cycle, final performance
requirements are developed early in the Concept Exploration phase (see fig. 2); in the Joint
Strike Fighter program, final requirements are determined later in the acquisition cycle. Program
officials state that this modification provides the flexibility needed to conduct cost and performance
trade-offs before requirement and design decisions become final.
Figure 2: Comparison of Traditional and Joint Strike Fighter Acquisition Cycles
a Joint Strike Fighter. Source: Joint Strike Fighter program office.
Acquisition Strategy Revises In what is known as an Operational Requirements Document, DOD defines
the Requirements aircraft program requirements (such as range, speed, and acceleration)
Determination Process very early in the acquisition cycle and before considering the aircraft
design necessary to meet these requirements. For the Joint Strike Fighter, performance requirements
are instead defined in five phases; in each phase, specific aircraft design characteristics are
determined as performance requirements are set. During each phase, performance requirements are
considered in terms of the established cost targets so that trade-offs in performance capabilities
can be made as necessary. 4 The most significant trade- off to date in the Joint Strike Fighter
Program was the decision to equip the aircraft with one versus two engines. Identifying trade-offs
to balance requirements for the affordability, effectiveness, and supportability 5 of the aircraft
design concept represents an ongoing effort between the government and the two competing contractors.
Acquisition Strategy According to the Joint Strike Fighter Single Acquisition Management Plan,
Designed to Reduce a principal objective of the program is to demonstrate to a low level of
Technical Risk technical risk those critical technologies, processes, and system
characteristics necessary to produce an affordable family of strike aircraft that meets all participants' needs.
To achieve that objective, the Joint Strike Fighter acquisition strategy is designed to lower technical
risk through aircraft flight demonstrations and advanced technology development prior to awarding the
engineering and manufacturing development contract. Specifically, during the current concept demonstration
phase, DOD requires each contractor to
demonstrate specific aircraft capabilities by designing and building actual flying models, conduct
demonstrations of key technologies and processes unique to
each contractor's aircraft design, and submit their preferred Joint Strike Fighter design concept.
4 This process is referred to as using Cost as an Independent Variable. 5 The degree to which system
design characteristics and planned logistics resources, including manpower, meet system peacetime
readiness and wartime utilization requirements.
Each competing contractor is required to design and build two aircraft to demonstrate the following:
commonality/ modularity to validate the contractors' ability to produce three aircraft variants on the
same production line; short take- off and vertical landing, hover, and transition to forward
flight to demonstrate the aircraft's ability to perform specific Marine Corps and Royal Navy missions; and
satisfactory low airspeed carrier approach flying and handling qualities
to demonstrate the aircraft's ability to perform specific Navy missions. Each contractor will also be
required to submit a Preferred Weapon System Concept, which outlines their preferred design concept for
developing an affordable Joint Strike Fighter aircraft to meet the goals specified in the final requirements
document. The Preferred Weapon System Concept will include results from the flight and ground demonstrations
and will ultimately be used by DOD to select the winning aircraft design and to award the engineering and
manufacturing development contract.
Implementation of Contrary to its acquisition strategy, the Joint Strike Fighter Program will
Acquisition Strategy not enter the engineering and manufacturing development phase with low
technical risk. The aircraft to be used in the concept demonstration phase Will Not Ensure That
are not intended to demonstrate all of the Joint Strike Fighter critical Program Objectives
technologies. Therefore, these technologies will be at low levels of Are Achieved
technical maturity when the engineering and manufacturing development contract is scheduled to be awarded. In
addition, when the competing contractors experienced design problems and cost overruns, DOD restructured the
program in a manner that is moving away from the best commercial practices that were evident in the original
strategy and is instead moving toward traditional practices that have caused problems on other programs.
Critical Technologies Not The aircraft being produced during the concept demonstration phase are
Developed to Acceptable not intended to demonstrate many of the technologies considered critical
Levels for achieving Joint Strike Fighter Program cost and performance
requirements, such as those for integrated avionics. Instead, many of these technologies will be demonstrated
only in laboratory or ground- testing environments and, therefore, will be at low levels of technical maturity
when the engineering and manufacturing development contract is scheduled to be awarded.
Matching the aircraft requirements and the maturity of technology as a program begins is perhaps the most
important determinant of a program's success. Once a program begins, a large, fixed investment in the form
of human capital, facilities, and materials is sunk into the program and any significant changes will have
a large, rippling effect on schedule and cost. In the case of critical technologies, beginning an acquisition
program when the technologies are at a low level of development increases program risk and the likelihood of
schedule delays, which increases program costs.
Measuring Technology At our request, the Joint Strike Fighter program office identified eight
Readiness technology areas that are considered critical to meeting Joint Strike Fighter
cost and/ or performance objectives. These technologies address areas such as avionics, flight systems,
manufacturing and producibility, propulsion, supportability, and weapons delivery system. 6 We requested
the program office to assign maturity levels for these critical technologies using a tool referred to as
technology readiness levels (TRLs). The TRLs were pioneered by the National Aeronautics and Space
Administration and adopted by the Air Force Research Laboratory 7 to determine the readiness of technologies
to be incorporated into a weapon or other type of system. The Joint Advanced Strike Technology Program
from which the Joint Strike Fighter Program evolved used TRLs to assess early maturity levels for many
of the current Joint Strike Fighter technologies. In response to our prior work, DOD has agreed that TRLs
can be used to help guide technology maturation and transition decisions. 8 Detailed descriptions of
technology readiness levels can be found in appendix II.
In conjunction with the program office and the two competing contractors, we determined the readiness
levels of critical technologies when the Joint Strike Fighter Program was started in 1996. That assessment
showed that when the Joint Strike Fighter Program entered the concept demonstration phase, most of the
critical technologies were well below maturity levels
6 Due to the current Joint Strike Fighter competition, the critical technologies are not specified so as
not to associate them with the respective contractors. 7 The Air Force Research Laboratory is a science
and technology organization that matures advanced technologies to the point that they can be included in
weapon system programs and be expected to perform as required. The Laboratory uses the TRLs to assess the
maturity of the technologies before they are handed off to programs.
8 Best Practices: Better Management of Technology Development Can Improve Weapon System Outcomes
( GAO/ NSIAD- 99- 162, July 30, 1999).
considered acceptable for low risk by the Air Force Research Laboratory or by leading commercial firms.
Readiness levels are measured on a scale of one to nine: Studies of the basic concept have a readiness
level of one; laboratory demonstrations have a readiness level between three and six; and technologies
that have been proven through integration on the intended product have a readiness level of nine. With
this measurement tool, the program office can gauge the likely consequences of placing various technologies
at a given maturity level into a development program and make informed choices and tradeoffs if necessary
to meet program goals.
Maturity Levels of Joint Strike The Air Force Research Laboratory considers a technology readiness level
Fighter Critical Technologies of six an acceptable risk for a weapon system entering the program
definition stage, the point at which DOD typically begins its weapon programs. At a lower level of technology
readiness, the technology's ability to meet the intended product's cost, schedule, and performance requirements
is uncertain. Reaching a maturity level of six denotes a significant transition point for technology development
as the technology moves from component testing in a laboratory environment to demonstrating a model or
prototype in a relevant environment. Figure 3 shows the maturity levels for the eight critical Joint
Strike Fighter technologies in November 1996, shortly after the Joint Strike Fighter Program was started.
Figure 3: Joint Strike Fighter Critical Technology Readiness Levels at Program Start
Note: Due to the current Joint Strike Fighter competition, the critical technologies are not identified so
as not to associate them with the respective contractors.
Source: Joint Strike Fighter program office.
As shown in figure 3, none of the Joint Strike Fighter critical technologies had achieved the desired
technology readiness level of six by program start. Instead, all of the technologies were at readiness
levels of two or three, which means that, at best, analytical and laboratory studies had been completed
or very early components had been developed.
We also obtained program office and contractor data assessing the expected maturity levels for the Joint
Strike Fighter as it enters engineering and manufacturing development in March 2001. The Air Force Research
Laboratory considers TRL 7 as acceptable for low risk when entering the engineering and manufacturing
development stage. This maturity level represents an advanced prototype of each of the critical technologies
demonstrated in an operational environment, such as on a flying test bed or another aircraft similar to
the Joint Strike Fighter. A prototype at this stage would include all of the components of a critical
technology in a configuration that is very close to the size, weight, and configurations as that expected
for the Joint Strike Fighter. While the Joint Strike Fighter Program has seen improvement in many
technology areas since the program started, maturity levels have not improved enough to indicate a low-risk
transition into the next phase. Figure 4 summarizes this data.
Figure 4: Joint Strike Fighter Critical Technology Readiness Levels at Program Start and Projected for Entry
Into Engineering and Manufacturing Development
Note: Due to the current Joint Strike Fighter competition, the critical technologies are not identified so
as not to associate them with the respective contractors.
Source: Joint Strike Fighter program office.
As shown in figure 4, all of the Joint Strike Fighter critical technologies are expected to be at maturity
levels lower than that considered acceptable for low risk when entering engineering and manufacturing
development (TRL 7). Six of the technologies will still be below the level of maturity (TRL 6) that is
considered low risk for entering the demonstration phase (program start), which the Joint Strike Fighter
Program entered over 3 years ago.
Should any of these technologies be delayed or, worse still, not be available for incorporation into the
final Joint Strike Fighter design, the impact on the program would be dramatic. For example, if one of
the above critical technologies needed to be replaced with its planned backup, DOD could expect an increase
of several billion dollars in production and operation and support costs. 9 The backup technology would
also significantly increase aircraft weight, which could negatively affect aircraft performance. The
currently planned technology is expected to be TRL 5 at the beginning of the engineering and manufacturing
development phase, which indicates that substantial technology development must still occur during that phase.
DOD Response to Recent contractor reports indicate that both competing contractors have
Contractor Cost and cost growth and schedule concerns. In response to concerns over its ability
Schedule Problems Is to meet program requirements, Boeing has redesigned its preferred design
Inconsistent With configuration, including changing the wing shape; adding a horizontal tail,
which lengthens the fuselage; and switching from a forward sweep air Acquisition Strategy Goals
intake to a rearward sweep. According to a Lockheed Martin program official, the company underestimated
the cost of producing the two demonstrator aircraft. In addition, Joint Strike Fighter Program documents
suggest that, due to manufacturing delays, the flight- test schedule for both competing contractors' Marine
Corps variant could be at risk.
As a result of cost concerns, DOD restructured the Joint Strike Fighter Program to allow each contractor
leeway in correcting deficiencies. For example, the Joint Strike Fighter flight- test program has been
decreased, which will reduce the data available for final proposal evaluation. Joint Strike Fighter
Program officials stated that with these flight- test reductions, only the minimum acceptable flight quality
demonstrations are
9 Specific details cannot be provided due to the competitive nature of the Joint Strike Fighter Program.
expected. The number of preferred weapon system design updates has also been reduced, which means that the
contractors will provide DOD with less information than originally planned prior to the submission of their
proposals. Finally, DOD has eliminated risk- reduction efforts and delayed other technology demonstrations,
which will increase the program's technical risk as it transitions into engineering and manufacturing
development (see footnote 8).
Traditional Approach to DOD's traditional approach to weapon systems acquisition is to mature
Problems Is Underlying Cause technology at the same time the product is being developed. This approach
of Increased Risk differs from best commercial practices, in which technology is developed
separately and ahead of the product. Pressures exerted on weapon system programs create incentives for
programs to include immature technologies that may offer significant performance gains. This pressure
can come from users who demand performance improvements that necessitate the application of
unproven technologies to stay ahead of the perceived threat. Another source of pressure is
from technologists, who see a new weapon system as an opportunity to apply a new technology.
Also, the competition for funds can encourage performance features and requisite technologies
that can distinguish the new weapon system from competitors.
Once in a product development environment, external pressures to keep the program moving (such as
preserving cost and schedule estimates to secure budget approval) become dominant. For example,
DOD policies require that a program be funded in the current year and that funds be made available
over the next 6 years in the DOD planning cycle. If a program manager decided that an additional
year was needed to reach the desired level of technical maturity during the risk reduction/concept
demonstration phase, the planned start of the engineering and manufacturing development phase
could be delayed. This delay could jeopardize funding for that phase, thus risking the funding
support for the entire program. Consequently, the program manager may be more likely to accept
the risk of moving forward with a lower level of technology maturity rather than risk losing
the program. That decision would raise cost- benefit issues because cost increases and
performance compromises would likely occur.
Conclusions The Joint Strike Fighter Program's acquisition strategy was to develop and field an
affordable aircraft that will meet each of the military services'
requirements goals. However, a key objective of the acquisition strategy entering
into engineering and manufacturing development with low technical risk will
not be achieved due to the manner in which the Joint Strike Fighter program
office is implementing the acquisition strategy. On its current schedule,
the program will enter the engineering and manufacturing development phase
without having reduced to an acceptable level the technical risk of technologies
that the program office has identified as critical to meeting the program's cost
and requirement objectives. This approach is not consistent with best commercial
practices in which technologies are more fully developed before proceeding into
product development. It is also not consistent with DOD's planned approach to
developing the Joint Strike Fighter. Instead, the program office's revised approach
is consistent with DOD's traditional approach in weapon system programs of concurrently
developing technologies and products. This traditional approach has often raised
cost-benefit issues as a result of cost increases, schedule delays, and compromised
performance as problems arose in completing technology development.
The Joint Strike Fighter Program is at an early development stage and, therefore, DOD
still has the opportunity to both demonstrate its commitment to acquisition reform
and chart a course to avoid the problems that often befall major weapon systems. A
decision to allow the Joint Strike Fighter to proceed as planned, without mature
critical technologies, would compromise DOD's position on acquisition reform, set
aside best commercial practices, and would perpetuate conditions that have led to
cost growth and schedule delays in many prior DOD weapon system acquisition programs.
Recommendation To demonstrate DOD's commitment to acquisition reform and to reduce
the risk of future cost growth, the program office should focus on risk
reduction efforts by maturing critical technologies prior to entering engineering
and manufacturing development, and it should be allowed to do so without the penalty
of withdrawal of funding support. Therefore, we recommend that the Secretary of
Defense direct the Joint Strike Fighter program office to adjust the currently planned
March 2001 engineering and manufacturing development decision date to allow adequate
time to mature critical technologies to acceptable maturity levels, thereby closing
the gap between technology and requirements, before awarding the
engineering and manufacturing development contract. Options that the program should
consider include
delaying the selection of a single contractor for the engineering and manufacturing phase
of the program until the program's critical technologies have been developed to an
acceptable level or selecting a single contractor, but providing the time and funding for
additional risk reduction and technology maturation efforts, so that this contractor can
mature critical technologies to acceptable levels before a decision is made to begin
engineering and manufacturing development.
Matters for To ensure that the Joint Strike Fighter Program enters the engineering and
Congressional manufacturing development phase with low technical risk, as envisioned
by the original acquisition strategy, Congress may wish to consider Consideration
requiring the Secretary of Defense to identify which of the eight critical technologies
discussed in this report will be incorporated on the Joint Strike Fighter and certify
that each of the identified technologies has been demonstrated in a form that is the
right size, weight, and configuration needed for the Joint Strike Fighter aircraft.
For any of the eight technologies not initially included on the Joint Strike Fighter,
the Secretary of Defense should develop a plan showing the strategy for demonstrating
these technologies in the right size, weight, and configuration; showing the approach
for including them onto the Joint Strike Fighter; and the cost impact if these
technologies do not become available as planned. Congress may also want to consider
restricting DOD from obligating funds made available for the engineering and manufacturing
development phase of the program until it receives this information from DOD.
Agency Comments and In written comments on a draft of this report, the Director of Strategic and
Our Evaluation Tactical Systems, within the Office of the Under Secretary of Defense for
Acquisition, Technology and Logistics, disagreed with our conclusions and recommendation.
DOD stated that our conclusion was based on misinterpretation of the use of technology
readiness levels to determine the readiness of the critical technologies to enter engineering
and manufacturing development. In addition, DOD stated that (1) only the maturity of the
technology, not its integration onto the Joint Strike Fighter, should be rated to
determine its readiness to enter engineering and manufacturing development; (2) our
use of technology readiness levels does not recognize that an evolutionary acquisition
approach is being
applied to the Joint Strike Fighter Program; and (3) its own risk reduction efforts are
more meaningful than using technology readiness levels.
Contrary to DOD's comments, there was no misinterpretation by us, the contractors, or the
program office representatives about what constituted a readiness level of seven the level
considered necessary for acceptable risk for entering engineering and manufacturing development.
As pointed out in this report, the readiness level definitions clearly state that for a technology
to be rated at a level seven, it must be demonstrated using prototype hardware, such as a complete
radar subsystem that is the same size, weight, and configuration planned for the Joint Strike
Fighter in an operational environment. This definition was developed by the Air Force Research
Laboratory and was provided to the contractors prior to their scoring of the critical
technologies, and we discussed them at length during several days of meetings when the
contractors, with DOD program office personnel present, assigned the readiness levels for
the critical technologies. During those discussions, we made it clear that demonstrating
the technology in a relevant environment would include demonstrating the technology in a
flying test bed aircraft, such as an F- 16 or some other existing aircraft, and not
necessarily on an actual Joint Strike Fighter aircraft. As a result of those discussions,
there was agreement on the readiness levels assigned to each of the critical technologies
discussed. The program office then independently scored the critical technologies. The
program office scores, which are those used in this report, were consistent with the
contractors' scores.
DOD stated that the Joint Strike Fighter Program will address the integration risk of the
critical technologies during, rather than prior to, the engineering and manufacturing development
phase. As indicated in our report, we agree that the risk of integrating a subsystem such as a
radar onto the actual Joint Strike Fighter aircraft is an activity that is acceptable for the
engineering and manufacturing phase. However, we do not agree that integrating various
components of a subsystem such as an antenna, receiver, transmitter, and processor
that make up a radar subsystem into a configuration that can be inserted into the
Joint Strike Fighter is a task to be left for the engineering and manufacturing
development phase. In that regard, commercial firms have told us that a key part
of technology development is getting the technology into the right size, weight,
and configuration needed for the intended product. Once this has been demonstrated,
the technology is at an acceptable level for engineering and manufacturing development,
where the emphasis should be on building the actual Joint Strike Fighter aircraft.
In separate technical comments on this
report, DOD acknowledged that the highest program risks will come from integrating critical
technologies, which it intends to do during engineering and manufacturing development. DOD's
approach, of developing technology during engineering and manufacturing development,
represents a fundamental difference between best commercial practices and DOD practices
and has been a leading reason for DOD's weapon programs incurring cost growth, schedule
slippage, and technical problems.
In its comments, DOD described an evolutionary acquisition approach being applied
to the Joint Strike Fighter Program as a way to further mitigate technology and
program risk. This approach means that the Joint Strike Fighter Program will use
time- phased requirements and capabilities, and only those technologies and capabilities
that are mature and at low risk will be installed on the first Joint Strike Fighter
aircraft that are produced. Cost and requirement trade- offs will be considered so
that technologies will not be included on the Joint Strike Fighter until they have
demonstrated an acceptable maturity level. We are concerned that since many of the
technologies assessed at low maturity levels in our report are critical to obtaining
an affordable aircraft a primary objective for both the Joint Strike Fighter and DOD's
overall tactical aircraft modernization plan their absence from the Joint Strike Fighter
design could result in much higher development, production, and support costs. The
impact of developing and producing Joint Strike Fighter aircraft without these
technologies must be considered, otherwise, DOD could find itself committed to a
program dependent on future maturation of these technologies in order to meet
program cost objectives. This would be typical of DOD's historical approach to
developing weapon systems, and it indicates that DOD is willing to assume greater
risk during engineering and manufacturing development than was envisioned in the
Joint Strike Fighter Program's original acquisition strategy.
Finally, DOD stated that its own risk reduction methodology which include risk
mitigation plans and engineering judgment is a more meaningful measure of risk
versus the use of technology readiness levels. We do not share DOD's confidence
in this regard. The objective of technology readiness levels is to make decisions
based on actual demonstrations that technologies will work as needed for the intended
product as opposed to engineering judgment, which is subjective and open to interpretation.
Our prior work has shown that in place of risk mitigation plans and engineering judgment,
no matter how well intentioned, using technology readiness levels results in a
straightforward, objective, and quantifiable process for determining a technology's
readiness for proceeding into the engineering
and manufacturing development phase of a program. DOD has recognized the value of the
technology readiness level process as indicated by its intention to include the process
in the revised acquisition guidelines commonly referred to as the 5000 acquisition series.
In summary, the information provided by DOD in its comments on our draft report does
not change our position that, as currently structured, the Joint Strike Fighter
Program will move into the engineering and manufacturing development phase with
unacceptable risk levels for many critical technologies. Therefore, we have not
changed the recommendation that was in our draft report but we have added some
matters for Congress to consider. In essence, we suggest that Congress consider
restricting DOD from obligating funds for the engineering and manufacturing development
phase of the program until the Secretary of Defense certifies that the program's
critical technologies have been demonstrated to acceptable levels of maturity.
The full text of DOD's comments are included in appendix III. DOD also provided
separate technical comments that we have incorporated into the report as appropriate.
Scope and To determine the acquisition strategy for meeting affordability goals
Methodology through reduced risk prior to awarding the engineering and manufacturing
development contract, we reviewed program documentation on acquisition strategy and
acquisition reform initiatives and discussed these materials with DOD and program
officials. These materials include data such as the Joint Strike Fighter Single
Acquisition Management Plan, the Interim Test and Evaluation Master Plan, Joint
Interim Requirements Document, Key Performance Parameters in the draft Joint
Operational Requirements Document, and Cost as an Independent Variable documentation.
To determine whether the program office is implementing the Joint Strike
Fighter acquisition strategy in a manner that will reduce risk and meet Joint
Strike Fighter cost, schedule, and performance goals, we collected data and
interviewed officials at various DOD locations, the Joint Strike Fighter program
office (Arlington, Virginia), and cognizant Navy, Marine Corps, and Air Force
requirements organizations. We interviewed officials and collected contractor
data from Lockheed Martin Tactical Aircraft Systems, Fort Worth, Texas, and
Palmdale, California; the Boeing Company, Seattle, Washington, and Palmdale,
California; General Electric, Cincinnati, Ohio; and Pratt & Whitney, West
Palm Beach, Florida.
To assess the maturity levels of key technologies, we used information provided
by the Joint Strike Fighter program office and contractors and used the technology
readiness level tool developed by the National Aeronautics and Space Administration.
On separate visits to the contractors, with DOD program office personnel present, we
provided relevant technology managers the TRL scoring sheet found in appendix II of
this report. After significant discussion, and additional TRL information, we asked
these managers to score those technologies they considered critical to enable their
Joint Strike Fighter design to meet DOD requirements for the aircraft. Upon reviewing
these scores with the program office and in order to gain an overall Joint Strike Fighter
Program perspective on technical maturity, the Joint Strike Fighter office agreed to
provide us with TRL scores for the eight technologies they considered critical for
meeting program cost and performance requirements. Those scores are presented in this report.
We conducted our review from September 1998 through January 2000 in accordance with
generally accepted government auditing standards.
Unless you publicly announce its contents earlier, we plan no further distribution of
this report until 30 days from its issue date. At that time, we will send copies to the
congressional defense committees; the Honorable William S. Cohen, Secretary of Defense;
the Honorable F. Whitten Peters, Secretary of the Air Force; the Honorable Richard Danzig,
Secretary of the Navy; General James L. Jones, Commandant, U. S. Marine Corps; and the
Honorable Jacob J. Lew, Director, Office of Management and Budget. We will also make
copies available to other interested parties on request.
Please contact me at (202) 512- 4841 if you or your staff have any questions concerning
this report. Key contributors to this assignment were Steve Kuhta, Brian Mullins, Delores
Cohen, and Matt Lea.
Sincerely yours, Louis J. Rodrigues Director, Defense Acquisitions Issues
Appendi xes International Participation in the Joint Strike
Appendi xI
Fighter Program Full collaborative partner full access to program data and structure, including representative
personnel within the program office ability to influence requirement definition and performance
characteristics Great Britain is the only participant in this category, committing
$200 million for the concept demonstration phase Associate partner limited access to data
and limited requirements influence representative personnel resident within the Joint Strike Fighter
program office all aspects of participation are negotiated with the Joint Strike Fighter
program office Denmark, the Netherlands, and Norway formed a group and committed
a total of $30 million to participate at this level Informed customer limited access to
program information and representation within the
program office is negotiable no influence on requirements Canada and Italy have
committed $10 million each to participate at this
level Major participants recently created category; also referred to as Foreign Military Sales
participation or Fee for Service negotiate directly with the program office for specific Joint Strike
Fighter Program information (e. g., Cost and Operational Performance Trade processes
and modeling and simulation studies) no representative personnel resident within the Joint Strike Fighter
program office Singapore, Turkey, and Israel are currently participating at this level
Technology Readiness Levels and Their
Appendi xII
Definitions Technology readiness level Description
1. Basic principles observed and reported. Lowest level of technology readiness. Scientific
research begins to be translated into applied research and development. Examples might
include paper studies of a technology's basic properties
2. Technology concept and/ or application Invention begins. Once basic principles are
observed, practical applications can be formulated. invented. The application is speculative
and there is no proof or detailed analysis to support
the assumption. Examples are still limited to paper studies. 3. Analytical and experimental critical
Active research and development is initiated. This includes analytical studies and laboratory
function and/ or characteristic proof of
studies to physically validate analytical predictions of separate elements of the technology. concept.
Examples include components that are not yet integrated or representative. 4. Component and/or
breadboard validation
Basic technological components are integrated to establish that the pieces will work in
laboratory environment. together. This is relatively low fidelity compared to the eventual
system. Examples include
integration of ad hoc hardware in a laboratory. 5. Component and/ or breadboard validation
Fidelity of breadboard technology increases significantly. The basic technological in relevant
environment. components are integrated with reasonably realistic supporting elements so that the
technology can be tested in a simulated environment. Examples include high fidelity laboratory
integration of components.
6. System/ subsystem model or prototype Representative model or prototype system, which is well
beyond the breadboard tested for demonstration in a relevant environment. technology readiness
level (TRL) 5, is tested in a relevant environment. Represents a major
step up in a technology's demonstrated readiness. Examples include testing a prototype in a
high fidelity laboratory environment or in simulated operational environment.
7. System prototype demonstration in an Prototype near or at planned operational system. Represents
a major step up from TRL 6, operational environment. requiring the demonstration of an actual
system prototype in an operational environment,
such as in an aircraft, vehicle, or space. Examples include testing the prototype in a test bed
aircraft.
8. Actual system completed and flight Technology has been proven to work in its final form and
under expected conditions. In qualified through test and demonstration. almost all cases, this
TRL represents the end of true system development. Examples
include developmental test and evaluation of the system in its intended weapon systems to determine
if it meets design specifications.
9. Actual system flight proven through Actual application of the technology in its final form and
under mission conditions, such as successful mission operations. those encountered in operational
test and evaluation. In almost all cases, this is the end of
the last bug fixing aspects of true system development. Examples include using the system under
operational mission conditions.
Source: Best Practices: Better Management of Technology Development Can Improve Weapon System
Outcomes( GAO/ NSIAD- 99- 162, July 30, 1999).
Appendi xI II
Comments From the Department of Defense Note: GAO comments supplementing those in the report text
appear at the end of this appendix.
See comment 1.
See comment 2.
The following are GAO's comments on the Department of Defense's (DOD) letter dated April 17, 2000.
GAO Comments 1. DOD provided reevaluated technology readiness levels to indicate that if the risk
of integrating critical technologies was not considered, then
the technology risk is expected to be at an appropriate level. Notwithstanding our disagreement
over whether integration risks should be considered in this assessment, we believe DOD's
standards are below minimum acceptable levels. Only half of the critical technologies are
projected to be at readiness level 7 while the other half will still be at readiness level 6.
We disagree with DOD that readiness level 6 is acceptable for low- risk entry into engineering
and manufacturing development. Leading commercial firms typically insist on a readiness level
8-a higher standard than that used by the Air Force Research Laboratory-before a technology
can be included on a product.
2. DOD noted that in addition to technology readiness levels, other considerations were
necessary to decide when and where to insert new technologies into weapon system programs.
While DOD did not elaborate on what other considerations were applicable to the Joint
Strike Fighter Program, its similar response to our prior work involving technology
readiness levels referred to such considerations as the increasing projected life for
new weapon systems, total ownership costs, and urgency based upon threat assessments.
Many of the technologies presented in this current report directly address these other
considerations and are critical to the success of the Joint Strike Fighter Program.
Therefore, using TRLs in the Joint Strike Fighter Program not only allows DOD to manage
performance risk, but also to address those technologies critical to meeting these other
considerations. With regard to risk assessment, leading commercial firms who have adopted
knowledge- based risk assessment approaches, such as TRLs, have produced results such as
more technically advanced, higher quality products, in less time and at a lower cost than their predecessors.
(707386) Lett er
GAO United States General Accounting Office
Page 1 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
Contents
Page 2 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
Page 3 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition United States General Accounting Office
Washington, D. C. 20548 Page 3 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 4 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 5 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 6 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 7 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 8 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 9 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 10 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 11 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 12 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 13 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 14 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 15 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 16 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 17 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 18 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 19 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 20 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 21 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
B- 281196 Page 22 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
Page 23 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
Page 24 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
Appendix I
Page 25 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
Appendix II
Page 26 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
Appendix III
Appendix III Comments From the Department of Defense
Page 27 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
Appendix III Comments From the Department of Defense
Page 28 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
Appendix III Comments From the Department of Defense
Page 29 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
Appendix III Comments From the Department of Defense
Page 30 GAO/ NSIAD- 00- 74 Joint Strike Fighter Acquisition
Ordering Information The first copy of each GAO report is free. Additional copies of
reports are $2 each. A check or money order should be made out to
the Superintendent of Documents. VISA and MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
Orders by mail: U. S. General Accounting Office P. O. Box 37050 Washington, DC 20013
Orders by visiting: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U. S. General
Accounting Office Washington, DC
Orders by phone: (202) 512- 6000 fax: (202) 512- 6061 TDD (202) 512- 2537
Each day, GAO issues a list of newly available reports and testimony. To receive facsimile
copies of the daily list or any list from the past 30 days, please call (202) 512- 6000
using a touchtone phone. A recorded menu will provide information on how to obtain these lists.
Orders by Internet: For information on how to access GAO reports on the Internet, send an
e-mail message with info in the body to: info@ www. gao. gov or visit GAO's World Wide Web
home page at: http:// www. gao. gov
To Report Fraud, Waste, or Abuse in Federal Programs
Contact one: Web site: http:// www. gao. gov/ fraudnet/ fraudnet. htm e- mail: fraudnet@ gao. gov
1- 800- 424- 5454 (automated answering system)
United States General Accounting Office Washington, D. C. 20548- 0001
Official Business Penalty for Private Use $300
Address Correction Requested Bulk Rate
Postage & Fees Paid GAO Permit No. GI00
Joint Strike Fighter Acquisition: Development Schedule Should Be
Changed to Reduce Risks A Report to the Chairman, Subcommittee on
National Security, Veterans' Affairs, and International Relations,
Committee on Government Reform, House of Representatives May 2000
JOINT STRIKE FIGHTER ACQUISITION Development Schedule Should Be
Changed to Reduce Risks GAO/NSIAD-00-74 Letter 3 Appendixes
Appendix I: International Participation in the Joint Strike
Fighter Program 24 Appendix II: Technology Readiness Levels and
Their Definitions 25 Appendix III: Comments From the Department of
Defense 26 Tables Table 1: Military Service Needs for the Joint
Strike Fighter 6 Figures Figure 1: Boeing and Lockheed Martin
Joint Strike Fighter Aircraft Design Concepts 7 Figure 2:
Comparison of Traditional and Joint Strike Fighter Acquisition
Cycles 8 Figure 3: Joint Strike Fighter Critical Technology
Readiness Levels at Program Start 13 Figure 4: Joint Strike
Fighter Critical Technology Readiness Levels at Program Start and
Projected for Entry Into Engineering and Manufacturing Development
14 Abbreviations DOD Department of Defense JSF Joint Strike
Fighter TRL technology readiness level National Security and
International Affairs Division Lett er B-281196 May 9, 2000 The
Honorable Christopher Shays Chairman, Subcommittee on National
Security, Veterans' Affairs, and International Relations Committee
on Government Reform House of Representatives Dear Mr. Chairman:
The Joint Strike Fighter Program is intended to produce an
affordable, next- generation aircraft to replace the Department of
Defense's (DOD) aging aircraft inventory. The first aircraft
deliveries are scheduled to begin in 2008. As currently planned,
the program will cost about $200 billion to develop and procure
over 3,000 aircraft and related support equipment for the Air
Force, the Marine Corps, the Navy, and Great Britain. DOD has
designated the Joint Strike Fighter Program as a flagship program
for acquisition. To date, the program has awarded contracts
totaling over $2 billion to Boeing and Lockheed Martin for the
current concept demonstration phase. Under these contracts, both
contractors will build the aircraft they plan to fly in the
demonstration phase and also design the aircraft they plan to
build in the next phase of the development program engineering and
manufacturing development. During engineering and manufacturing
development, the Joint Strike Fighter will be fully developed,
engineered, designed, fabricated, tested, and evaluated to
demonstrate that the production aircraft will meet stated
requirements. DOD is scheduled to award the contract for
engineering and manufacturing development to either Boeing or
Lockheed Martin in April 2001. 1 At your request, we reviewed the
Joint Strike Fighter Program to (1) provide information on the
acquisition strategy and (2) to determine whether the strategy is
being implemented in a manner that will ensure that the
acquisition strategy objectives will be achieved. With your
permission, we discussed a draft of this report during a March 16,
2000, joint hearing by 1 At the time of this report, the Under
Secretary of Defense for Acquisition, Technology, and Logistics
was reviewing competition and industrial- base implications of the
Joint Strike Fighter acquisition strategy. the Subcommittees on
Military Procurement and on Military Research and Development,
House Committee on Armed Services. At the time of the hearing, we
had not received DOD's comments on our report. This report
contains DOD's comments and our evaluation of them. Results in
Brief The key objective of the Joint Strike Fighter acquisition
strategy is affordability reducing the development, production,
and ownership costs of the program relative to prior fighter
aircraft programs. DOD expects the Joint Strike Fighter
acquisition strategy to save nearly $18 billion (in fiscal year
1995 dollars) in development costs. To achieve its affordability
objective, the Joint Strike Fighter program office has
incorporated various DOD and commercial acquisition initiatives
into the Joint Strike Fighter acquisition strategy. These
initiatives include modifying the traditional weapons acquisition
cycle, revising the requirements determination process, and
developing critical technologies to a level where they represent
low technical risk before the engineering and manufacturing
contract is awarded. The expectation is that incorporating these
initiatives into the Joint Strike Fighter acquisition strategy
will result in a better match between the maturity 2 of key
technologies and the aircraft's requirements. Matching the
requirements and the maturity of technology when a program enters
engineering and manufacturing development is a critical
determinant of a program's success. Once the development phase
begins, a large, fixed investment in the form of human capital,
facilities, and materials is sunk into the program and any
significant changes will have a large, rippling effect on cost and
schedule. Beginning the engineering and manufacturing development
phase when critical technologies are at a low level of maturity
serves to significantly increase program risk and the likelihood
of schedule delays, which in turn result in increased program
costs. The Joint Strike Fighter program office's implementation of
its acquisition strategy will not ensure that the Joint Strike
Fighter program will enter the engineering and manufacturing
development phase with low technical risk. The aircraft being
produced during the concept demonstration phase are not intended
to demonstrate many of the technologies considered critical for
achieving Joint Strike Fighter program cost and performance
requirements. Instead, many of these technologies such as
avionics, flight 2 A technology is considered to be mature when it
has been developed to a point that it can be readily integrated
into a new product and counted on to meet product requirements.
systems, manufacturing and producibility, propulsion,
supportability, and weapons delivery system will only be
demonstrated in laboratory or ground- testing environments.
Therefore, these critical technologies will be at low levels of
technical maturity when the engineering and manufacturing
development contract is scheduled to be awarded. In addition, when
the competing contractors experienced design problems and cost
overruns, DOD restructured the program in a manner that will
provide less information than originally planned prior to
selecting between the two competing contractors. Specifically,
this program restructure moves away from best commercial practices
that were evident in the original strategy, where technology was
being developed ahead of the product. Instead, DOD's approach
moves toward the traditional practice of concurrently developing
technologies and products, which often raised cost- benefit issues
as a result of cost increases and schedule delays as problems are
encountered in technology development. To demonstrate DOD's
commitment to acquisition reform, follow best commercial
practices, and reduce the risk of future cost growth, the program
office should focus on risk reduction efforts by maturing critical
technologies prior to entering engineering and manufacturing
development, and it should be allowed to do so without the penalty
of withdrawal of funding support. We make a recommendation that
the Joint Strike Fighter program office adjust its currently
planned engineering and manufacturing development decision date of
March 2001 to allow adequate time to mature critical technologies
to acceptable maturity levels before awarding the engineering and
manufacturing development contract. Background The Joint Strike
Fighter is the centerpiece of DOD's tactical aircraft
modernization plan, which includes the Air Force F- 22 Raptor and
the Navy F/ A- 18 E/ F Super Hornet. The program is structured to
use a common production line to produce three aircraft variants
that meet conventional flight requirements for the U. S. Air
Force, short take- off and vertical landing characteristics for
the Marine Corps, and carrier operation suitability needs for the
U. S. Navy. The program will also provide aircraft to the British
Royal Navy and Air Force. Table 1 shows current service plans for
Joint Strike Fighter use. Table 1: Military Service Needs for the
Joint Strike Fighter Service Quantity Planned use Air Force 1,763
Replacement for F- 16 and A- 10; complement to the F- 22 Marine
Corps 609 Short take- off and vertical landing aircraft to replace
AV- 8B and F/ A- 18 C/ D Navy 480 Carrier- based, multi- role,
first day of war survivable strike fighter to complement the F/ A-
18 E/ F Great Britain 150 Short take- off and vertical landing
replacement for the Sea Harrier and GR. 7 Source: Joint Strike
Fighter program office. DOD expects the Air Force variant of the
Joint Strike Fighter to cost about $28 million per unit; the Navy
variant to be between $31 million and $38 million; and the Marine
Corps variant to cost between $30 million and $35 million. 3
Independent estimates are not so optimistic. For example, in
congressional hearings held in March 1999, the Congressional
Budget Office estimated that the unit cost of the Joint Strike
Fighter could be as much as 47 percent to 51 percent higher than
expected, depending on which variant was procured. DOD and the
Congressional Budget Office estimates vary as a result of
differing estimating techniques, including estimating the cost of
incorporating stealth technologies into the Joint Strike Fighter
design. Figure 1 shows planned Joint Strike Fighter aircraft
designs by contractor. 3 Expected costs include the cost to
produce the basic aircraft, propulsion system, and avionics. Costs
are stated in fiscal year 1994 dollars. Figure 1: Boeing and
Lockheed Martin Joint Strike Fighter Aircraft Design Concepts
Boeing Joint Strike Fighter Lockheed Martin Joint Strike Fighter
Design Concept Design Concept Source: Joint Strike Fighter program
office. Acquisition Strategy The focus of the Joint Strike Fighter
Program is affordability reducing the Designed to Achieve
development, production, and ownership costs of the program
relative to prior fighter aircraft programs. To achieve this
objective, the Joint Strike Affordability Goals Fighter program
office has incorporated various DOD and commercial Through Reduced
acquisition initiatives into the Joint Strike Fighter acquisition
strategy. Program Risk These initiatives include modifying the
traditional weapons acquisition cycle, revising the requirements
determination process, and advancing the maturity level of
critical technologies so they represent low technical risk before
the engineering and manufacturing contract is awarded. The
expectation is that incorporating these initiatives into the Joint
Strike Fighter acquisition strategy will avoid cost growth,
schedule slippage, and performance shortfalls that have been
experienced in other weapons acquisition programs. Acquisition
Strategy The Joint Strike Fighter acquisition strategy modifies
the cycle that Modifies Traditional weapons acquisition programs
normally follow. For example, the Joint Acquisition Cycle Strike
Fighter program office was created earlier in the acquisition
cycle than it would have been in a traditional DOD weapons system
program. This enabled DOD to obtain early input from relevant
stakeholders (operators, maintenance personnel, industry
representatives, government engineers, and officials within the
intelligence community) to ensure that all aspects of cost,
schedule, performance, and resource constraints are included in
decision- making. In addition, the program has encouraged greater
involvement from the international community, which provides both
monetary and technical support (see app. I). The traditional
acquisition cycle has also been modified by combining the first
two traditional acquisition phases Concept Exploration and Program
Definition and Risk Reduction into one phase, known as Concept
Demonstration. Under the traditional DOD acquisition cycle, final
performance requirements are developed early in the Concept
Exploration phase (see fig. 2); in the Joint Strike Fighter
program, final requirements are determined later in the
acquisition cycle. Program officials state that this modification
provides the flexibility needed to conduct cost and performance
trade- offs before requirement and design decisions become final.
Figure 2: Comparison of Traditional and Joint Strike Fighter
Acquisition Cycles a Joint Strike Fighter. Source: Joint Strike
Fighter program office. Acquisition Strategy Revises In what is
known as an Operational Requirements Document, DOD defines the
Requirements aircraft program requirements (such as range, speed,
and acceleration) Determination Process very early in the
acquisition cycle and before considering the aircraft design
necessary to meet these requirements. For the Joint Strike
Fighter, performance requirements are instead defined in five
phases; in each phase, specific aircraft design characteristics
are determined as performance requirements are set. During each
phase, performance requirements are considered in terms of the
established cost targets so that trade- offs in performance
capabilities can be made as necessary. 4 The most significant
trade- off to date in the Joint Strike Fighter Program was the
decision to equip the aircraft with one versus two engines.
Identifying trade- offs to balance requirements for the
affordability, effectiveness, and supportability 5 of the aircraft
design concept represents an ongoing effort between the government
and the two competing contractors. Acquisition Strategy According
to the Joint Strike Fighter Single Acquisition Management Plan,
Designed to Reduce a principal objective of the program is to
demonstrate to a low level of Technical Risk technical risk those
critical technologies, processes, and system characteristics
necessary to produce an affordable family of strike aircraft that
meets all participants' needs. To achieve that objective, the
Joint Strike Fighter acquisition strategy is designed to lower
technical risk through aircraft flight demonstrations and advanced
technology development prior to awarding the engineering and
manufacturing development contract. Specifically, during the
current concept demonstration phase, DOD requires each contractor
to demonstrate specific aircraft capabilities by designing and
building actual flying models, conduct demonstrations of key
technologies and processes unique to each contractor's aircraft
design, and submit their preferred Joint Strike Fighter design
concept. 4 This process is referred to as using Cost as an
Independent Variable. 5 The degree to which system design
characteristics and planned logistics resources, including
manpower, meet system peacetime readiness and wartime utilization
requirements. Each competing contractor is required to design and
build two aircraft to demonstrate the following: commonality/
modularity to validate the contractors' ability to produce three
aircraft variants on the same production line; short take- off
and vertical landing, hover, and transition to forward flight to
demonstrate the aircraft's ability to perform specific Marine
Corps and Royal Navy missions; and satisfactory low airspeed
carrier approach flying and handling qualities to demonstrate the
aircraft's ability to perform specific Navy missions. Each
contractor will also be required to submit a Preferred Weapon
System Concept, which outlines their preferred design concept for
developing an affordable Joint Strike Fighter aircraft to meet the
goals specified in the final requirements document. The Preferred
Weapon System Concept will include results from the flight and
ground demonstrations and will ultimately be used by DOD to select
the winning aircraft design and to award the engineering and
manufacturing development contract. Implementation of Contrary to
its acquisition strategy, the Joint Strike Fighter Program will
Acquisition Strategy not enter the engineering and manufacturing
development phase with low technical risk. The aircraft to be used
in the concept demonstration phase Will Not Ensure That are not
intended to demonstrate all of the Joint Strike Fighter critical
Program Objectives technologies. Therefore, these technologies
will be at low levels of Are Achieved technical maturity when the
engineering and manufacturing development contract is scheduled to
be awarded. In addition, when the competing contractors
experienced design problems and cost overruns, DOD restructured
the program in a manner that is moving away from the best
commercial practices that were evident in the original strategy
and is instead moving toward traditional practices that have
caused problems on other programs. Critical Technologies Not The
aircraft being produced during the concept demonstration phase are
Developed to Acceptable not intended to demonstrate many of the
technologies considered critical Levels for achieving Joint Strike
Fighter Program cost and performance requirements, such as those
for integrated avionics. Instead, many of these technologies will
be demonstrated only in laboratory or ground- testing environments
and, therefore, will be at low levels of technical maturity when
the engineering and manufacturing development contract is
scheduled to be awarded. Matching the aircraft requirements and
the maturity of technology as a program begins is perhaps the most
important determinant of a program's success. Once a program
begins, a large, fixed investment in the form of human capital,
facilities, and materials is sunk into the program and any
significant changes will have a large, rippling effect on schedule
and cost. In the case of critical technologies, beginning an
acquisition program when the technologies are at a low level of
development increases program risk and the likelihood of schedule
delays, which increases program costs. Measuring Technology At our
request, the Joint Strike Fighter program office identified eight
Readiness technology areas that are considered critical to meeting
Joint Strike Fighter cost and/ or performance objectives. These
technologies address areas such as avionics, flight systems,
manufacturing and producibility, propulsion, supportability, and
weapons delivery system. 6 We requested the program office to
assign maturity levels for these critical technologies using a
tool referred to as technology readiness levels (TRLs). The TRLs
were pioneered by the National Aeronautics and Space
Administration and adopted by the Air Force Research Laboratory 7
to determine the readiness of technologies to be incorporated into
a weapon or other type of system. The Joint Advanced Strike
Technology Program from which the Joint Strike Fighter Program
evolved used TRLs to assess early maturity levels for many of the
current Joint Strike Fighter technologies. In response to our
prior work, DOD has agreed that TRLs can be used to help guide
technology maturation and transition decisions. 8 Detailed
descriptions of technology readiness levels can be found in
appendix II. In conjunction with the program office and the two
competing contractors, we determined the readiness levels of
critical technologies when the Joint Strike Fighter Program was
started in 1996. That assessment showed that when the Joint Strike
Fighter Program entered the concept demonstration phase, most of
the critical technologies were well below maturity levels 6 Due to
the current Joint Strike Fighter competition, the critical
technologies are not specified so as not to associate them with
the respective contractors. 7 The Air Force Research Laboratory is
a science and technology organization that matures advanced
technologies to the point that they can be included in weapon
system programs and be expected to perform as required. The
Laboratory uses the TRLs to assess the maturity of the
technologies before they are handed off to programs. 8 Best
Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes( GAO/NSIAD-99-162, July 30, 1999).
considered acceptable for low risk by the Air Force Research
Laboratory or by leading commercial firms. Readiness levels are
measured on a scale of one to nine: Studies of the basic concept
have a readiness level of one; laboratory demonstrations have a
readiness level between three and six; and technologies that have
been proven through integration on the intended product have a
readiness level of nine. With this measurement tool, the program
office can gauge the likely consequences of placing various
technologies at a given maturity level into a development program
and make informed choices and tradeoffs if necessary to meet
program goals. Maturity Levels of Joint Strike The Air Force
Research Laboratory considers a technology readiness level Fighter
Critical Technologies of six an acceptable risk for a weapon
system entering the program definition stage, the point at which
DOD typically begins its weapon programs. At a lower level of
technology readiness, the technology's ability to meet the
intended product's cost, schedule, and performance requirements is
uncertain. Reaching a maturity level of six denotes a significant
transition point for technology development as the technology
moves from component testing in a laboratory environment to
demonstrating a model or prototype in a relevant environment.
Figure 3 shows the maturity levels for the eight critical Joint
Strike Fighter technologies in November 1996, shortly after the
Joint Strike Fighter Program was started. Figure 3: Joint Strike
Fighter Critical Technology Readiness Levels at Program Start
Note: Due to the current Joint Strike Fighter competition, the
critical technologies are not identified so as not to associate
them with the respective contractors. Source: Joint Strike Fighter
program office. As shown in figure 3, none of the Joint Strike
Fighter critical technologies had achieved the desired technology
readiness level of six by program start. Instead, all of the
technologies were at readiness levels of two or three, which means
that, at best, analytical and laboratory studies had been
completed or very early components had been developed. We also
obtained program office and contractor data assessing the expected
maturity levels for the Joint Strike Fighter as it enters
engineering and manufacturing development in March 2001. The Air
Force Research Laboratory considers TRL 7 as acceptable for low
risk when entering the engineering and manufacturing development
stage. This maturity level represents an advanced prototype of
each of the critical technologies demonstrated in an operational
environment, such as on a flying test bed or another aircraft
similar to the Joint Strike Fighter. A prototype at this stage
would include all of the components of a critical technology in a
configuration that is very close to the size, weight, and
configurations as that expected for the Joint Strike Fighter.
While the Joint Strike Fighter Program has seen improvement in
many technology areas since the program started, maturity levels
have not improved enough to indicate a low- risk transition into
the next phase. Figure 4 summarizes this data. Figure 4: Joint
Strike Fighter Critical Technology Readiness Levels at Program
Start and Projected for Entry Into Engineering and Manufacturing
Development Note: Due to the current Joint Strike Fighter
competition, the critical technologies are not identified so as
not to associate them with the respective contractors. Source:
Joint Strike Fighter program office. As shown in figure 4, all of
the Joint Strike Fighter critical technologies are expected to be
at maturity levels lower than that considered acceptable for low
risk when entering engineering and manufacturing development (TRL
7). Six of the technologies will still be below the level of
maturity (TRL 6) that is considered low risk for entering the
demonstration phase (program start), which the Joint Strike
Fighter Program entered over 3 years ago. Should any of these
technologies be delayed or, worse still, not be available for
incorporation into the final Joint Strike Fighter design, the
impact on the program would be dramatic. For example, if one of
the above critical technologies needed to be replaced with its
planned backup, DOD could expect an increase of several billion
dollars in production and operation and support costs. 9 The
backup technology would also significantly increase aircraft
weight, which could negatively affect aircraft performance. The
currently planned technology is expected to be TRL 5 at the
beginning of the engineering and manufacturing development phase,
which indicates that substantial technology development must still
occur during that phase. DOD Response to Recent contractor reports
indicate that both competing contractors have Contractor Cost and
cost growth and schedule concerns. In response to concerns over
its ability Schedule Problems Is to meet program requirements,
Boeing has redesigned its preferred design Inconsistent With
configuration, including changing the wing shape; adding a
horizontal tail, which lengthens the fuselage; and switching from
a forward sweep air Acquisition Strategy Goals intake to a
rearward sweep. According to a Lockheed Martin program official,
the company underestimated the cost of producing the two
demonstrator aircraft. In addition, Joint Strike Fighter Program
documents suggest that, due to manufacturing delays, the flight-
test schedule for both competing contractors' Marine Corps variant
could be at risk. As a result of cost concerns, DOD restructured
the Joint Strike Fighter Program to allow each contractor leeway
in correcting deficiencies. For example, the Joint Strike Fighter
flight- test program has been decreased, which will reduce the
data available for final proposal evaluation. Joint Strike Fighter
Program officials stated that with these flight- test reductions,
only the minimum acceptable flight quality demonstrations are 9
Specific details cannot be provided due to the competitive nature
of the Joint Strike Fighter Program. expected. The number of
preferred weapon system design updates has also been reduced,
which means that the contractors will provide DOD with less
information than originally planned prior to the submission of
their proposals. Finally, DOD has eliminated risk- reduction
efforts and delayed other technology demonstrations, which will
increase the program's technical risk as it transitions into
engineering and manufacturing development (see footnote 8).
Traditional Approach to DOD's traditional approach to weapon
systems acquisition is to mature Problems Is Underlying Cause
technology at the same time the product is being developed. This
approach of Increased Risk differs from best commercial practices,
in which technology is developed separately and ahead of the
product. Pressures exerted on weapon system programs create
incentives for programs to include immature technologies that may
offer significant performance gains. This pressure can come from
users who demand performance improvements that necessitate the
application of unproven technologies to stay ahead of the
perceived threat. Another source of pressure is from
technologists, who see a new weapon system as an opportunity to
apply a new technology. Also, the competition for funds can
encourage performance features and requisite technologies that can
distinguish the new weapon system from competitors. Once in a
product development environment, external pressures to keep the
program moving (such as preserving cost and schedule estimates to
secure budget approval) become dominant. For example, DOD policies
require that a program be funded in the current year and that
funds be made available over the next 6 years in the DOD planning
cycle. If a program manager decided that an additional year was
needed to reach the desired level of technical maturity during the
risk reduction/ concept demonstration phase, the planned start of
the engineering and manufacturing development phase could be
delayed. This delay could jeopardize funding for that phase, thus
risking the funding support for the entire program. Consequently,
the program manager may be more likely to accept the risk of
moving forward with a lower level of technology maturity rather
than risk losing the program. That decision would raise cost-
benefit issues because cost increases and performance compromises
would likely occur. Conclusions The Joint Strike Fighter Program's
acquisition strategy was to develop and field an affordable
aircraft that will meet each of the military services'
requirements goals. However, a key objective of the acquisition
strategy entering into engineering and manufacturing development
with low technical risk will not be achieved due to the manner in
which the Joint Strike Fighter program office is implementing the
acquisition strategy. On its current schedule, the program will
enter the engineering and manufacturing development phase without
having reduced to an acceptable level the technical risk of
technologies that the program office has identified as critical to
meeting the program's cost and requirement objectives. This
approach is not consistent with best commercial practices in which
technologies are more fully developed before proceeding into
product development. It is also not consistent with DOD's planned
approach to developing the Joint Strike Fighter. Instead, the
program office's revised approach is consistent with DOD's
traditional approach in weapon system programs of concurrently
developing technologies and products. This traditional approach
has often raised cost- benefit issues as a result of cost
increases, schedule delays, and compromised performance as
problems arose in completing technology development. The Joint
Strike Fighter Program is at an early development stage and,
therefore, DOD still has the opportunity to both demonstrate its
commitment to acquisition reform and chart a course to avoid the
problems that often befall major weapon systems. A decision to
allow the Joint Strike Fighter to proceed as planned, without
mature critical technologies, would compromise DOD's position on
acquisition reform, set aside best commercial practices, and would
perpetuate conditions that have led to cost growth and schedule
delays in many prior DOD weapon system acquisition programs.
Recommendation To demonstrate DOD's commitment to acquisition
reform and to reduce the risk of future cost growth, the program
office should focus on risk reduction efforts by maturing critical
technologies prior to entering engineering and manufacturing
development, and it should be allowed to do so without the penalty
of withdrawal of funding support. Therefore, we recommend that the
Secretary of Defense direct the Joint Strike Fighter program
office to adjust the currently planned March 2001 engineering and
manufacturing development decision date to allow adequate time to
mature critical technologies to acceptable maturity levels,
thereby closing the gap between technology and requirements,
before awarding the engineering and manufacturing development
contract. Options that the program should consider include
delaying the selection of a single contractor for the engineering
and manufacturing phase of the program until the program's
critical technologies have been developed to an acceptable level
or selecting a single contractor, but providing the time and
funding for additional risk reduction and technology maturation
efforts, so that this contractor can mature critical technologies
to acceptable levels before a decision is made to begin
engineering and manufacturing development. Matters for To ensure
that the Joint Strike Fighter Program enters the engineering and
Congressional manufacturing development phase with low technical
risk, as envisioned by the original acquisition strategy, Congress
may wish to consider Consideration requiring the Secretary of
Defense to identify which of the eight critical technologies
discussed in this report will be incorporated on the Joint Strike
Fighter and certify that each of the identified technologies has
been demonstrated in a form that is the right size, weight, and
configuration needed for the Joint Strike Fighter aircraft. For
any of the eight technologies not initially included on the Joint
Strike Fighter, the Secretary of Defense should develop a plan
showing the strategy for demonstrating these technologies in the
right size, weight, and configuration; showing the approach for
including them onto the Joint Strike Fighter; and the cost impact
if these technologies do not become available as planned. Congress
may also want to consider restricting DOD from obligating funds
made available for the engineering and manufacturing development
phase of the program until it receives this information from DOD.
Agency Comments and In written comments on a draft of this report,
the Director of Strategic and Our Evaluation Tactical Systems,
within the Office of the Under Secretary of Defense for
Acquisition, Technology and Logistics, disagreed with our
conclusions and recommendation. DOD stated that our conclusion was
based on misinterpretation of the use of technology readiness
levels to determine the readiness of the critical technologies to
enter engineering and manufacturing development. In addition, DOD
stated that (1) only the maturity of the technology, not its
integration onto the Joint Strike Fighter, should be rated to
determine its readiness to enter engineering and manufacturing
development; (2) our use of technology readiness levels does not
recognize that an evolutionary acquisition approach is being
applied to the Joint Strike Fighter Program; and (3) its own risk
reduction efforts are more meaningful than using technology
readiness levels. Contrary to DOD's comments, there was no
misinterpretation by us, the contractors, or the program office
representatives about what constituted a readiness level of seven
the level considered necessary for acceptable risk for entering
engineering and manufacturing development. As pointed out in this
report, the readiness level definitions clearly state that for a
technology to be rated at a level seven, it must be demonstrated
using prototype hardware, such as a complete radar subsystem that
is the same size, weight, and configuration planned for the Joint
Strike Fighter in an operational environment. This definition was
developed by the Air Force Research Laboratory and was provided to
the contractors prior to their scoring of the critical
technologies, and we discussed them at length during several days
of meetings when the contractors, with DOD program office
personnel present, assigned the readiness levels for the critical
technologies. During those discussions, we made it clear that
demonstrating the technology in a relevant environment would
include demonstrating the technology in a flying test bed
aircraft, such as an F- 16 or some other existing aircraft, and
not necessarily on an actual Joint Strike Fighter aircraft. As a
result of those discussions, there was agreement on the readiness
levels assigned to each of the critical technologies discussed.
The program office then independently scored the critical
technologies. The program office scores, which are those used in
this report, were consistent with the contractors' scores. DOD
stated that the Joint Strike Fighter Program will address the
integration risk of the critical technologies during, rather than
prior to, the engineering and manufacturing development phase. As
indicated in our report, we agree that the risk of integrating a
subsystem such as a radar onto the actual Joint Strike Fighter
aircraft is an activity that is acceptable for the engineering and
manufacturing phase. However, we do not agree that integrating
various components of a subsystem such as an antenna, receiver,
transmitter, and processor that make up a radar subsystem into a
configuration that can be inserted into the Joint Strike Fighter
is a task to be left for the engineering and manufacturing
development phase. In that regard, commercial firms have told us
that a key part of technology development is getting the
technology into the right size, weight, and configuration needed
for the intended product. Once this has been demonstrated, the
technology is at an acceptable level for engineering and
manufacturing development, where the emphasis should be on
building the actual Joint Strike Fighter aircraft. In separate
technical comments on this report, DOD acknowledged that the
highest program risks will come from integrating critical
technologies, which it intends to do during engineering and
manufacturing development. DOD's approach, of developing
technology during engineering and manufacturing development,
represents a fundamental difference between best commercial
practices and DOD practices and has been a leading reason for
DOD's weapon programs incurring cost growth, schedule slippage,
and technical problems. In its comments, DOD described an
evolutionary acquisition approach being applied to the Joint
Strike Fighter Program as a way to further mitigate technology and
program risk. This approach means that the Joint Strike Fighter
Program will use time- phased requirements and capabilities, and
only those technologies and capabilities that are mature and at
low risk will be installed on the first Joint Strike Fighter
aircraft that are produced. Cost and requirement trade- offs will
be considered so that technologies will not be included on the
Joint Strike Fighter until they have demonstrated an acceptable
maturity level. We are concerned that since many of the
technologies assessed at low maturity levels in our report are
critical to obtaining an affordable aircraft a primary objective
for both the Joint Strike Fighter and DOD's overall tactical
aircraft modernization plan their absence from the Joint Strike
Fighter design could result in much higher development,
production, and support costs. The impact of developing and
producing Joint Strike Fighter aircraft without these technologies
must be considered, otherwise, DOD could find itself committed to
a program dependent on future maturation of these technologies in
order to meet program cost objectives. This would be typical of
DOD's historical approach to developing weapon systems, and it
indicates that DOD is willing to assume greater risk during
engineering and manufacturing development than was envisioned in
the Joint Strike Fighter Program's original acquisition strategy.
Finally, DOD stated that its own risk reduction methodology which
include risk mitigation plans and engineering judgment is a more
meaningful measure of risk versus the use of technology readiness
levels. We do not share DOD's confidence in this regard. The
objective of technology readiness levels is to make decisions
based on actual demonstrations that technologies will work as
needed for the intended product as opposed to engineering
judgment, which is subjective and open to interpretation. Our
prior work has shown that in place of risk mitigation plans and
engineering judgment, no matter how well intentioned, using
technology readiness levels results in a straightforward,
objective, and quantifiable process for determining a technology's
readiness for proceeding into the engineering and manufacturing
development phase of a program. DOD has recognized the value of
the technology readiness level process as indicated by its
intention to include the process in the revised acquisition
guidelines commonly referred to as the 5000 acquisition series. In
summary, the information provided by DOD in its comments on our
draft report does not change our position that, as currently
structured, the Joint Strike Fighter Program will move into the
engineering and manufacturing development phase with unacceptable
risk levels for many critical technologies. Therefore, we have not
changed the recommendation that was in our draft report but we
have added some matters for Congress to consider. In essence, we
suggest that Congress consider restricting DOD from obligating
funds for the engineering and manufacturing development phase of
the program until the Secretary of Defense certifies that the
program's critical technologies have been demonstrated to
acceptable levels of maturity. The full text of DOD's comments are
included in appendix III. DOD also provided separate technical
comments that we have incorporated into the report as appropriate.
Scope and To determine the acquisition strategy for meeting
affordability goals Methodology through reduced risk prior to
awarding the engineering and manufacturing development contract,
we reviewed program documentation on acquisition strategy and
acquisition reform initiatives and discussed these materials with
DOD and program officials. These materials include data such as
the Joint Strike Fighter Single Acquisition Management Plan, the
Interim Test and Evaluation Master Plan, Joint Interim
Requirements Document, Key Performance Parameters in the draft
Joint Operational Requirements Document, and Cost as an
Independent Variable documentation. To determine whether the
program office is implementing the Joint Strike Fighter
acquisition strategy in a manner that will reduce risk and meet
Joint Strike Fighter cost, schedule, and performance goals, we
collected data and interviewed officials at various DOD locations,
the Joint Strike Fighter program office (Arlington, Virginia), and
cognizant Navy, Marine Corps, and Air Force requirements
organizations. We interviewed officials and collected contractor
data from Lockheed Martin Tactical Aircraft Systems, Fort Worth,
Texas, and Palmdale, California; the Boeing Company, Seattle,
Washington, and Palmdale, California; General Electric,
Cincinnati, Ohio; and Pratt & Whitney, West Palm Beach, Florida.
To assess the maturity levels of key technologies, we used
information provided by the Joint Strike Fighter program office
and contractors and used the technology readiness level tool
developed by the National Aeronautics and Space Administration. On
separate visits to the contractors, with DOD program office
personnel present, we provided relevant technology managers the
TRL scoring sheet found in appendix II of this report. After
significant discussion, and additional TRL information, we asked
these managers to score those technologies they considered
critical to enable their Joint Strike Fighter design to meet DOD
requirements for the aircraft. Upon reviewing these scores with
the program office and in order to gain an overall Joint Strike
Fighter Program perspective on technical maturity, the Joint
Strike Fighter office agreed to provide us with TRL scores for the
eight technologies they considered critical for meeting program
cost and performance requirements. Those scores are presented in
this report. We conducted our review from September 1998 through
January 2000 in accordance with generally accepted government
auditing standards. Unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30
days from its issue date. At that time, we will send copies to the
congressional defense committees; the Honorable William S. Cohen,
Secretary of Defense; the Honorable F. Whitten Peters, Secretary
of the Air Force; the Honorable Richard Danzig, Secretary of the
Navy; General James L. Jones, Commandant, U. S. Marine Corps; and
the Honorable Jacob J. Lew, Director, Office of Management and
Budget. We will also make copies available to other interested
parties on request. Please contact me at (202) 512- 4841 if you or
your staff have any questions concerning this report. Key
contributors to this assignment were Steve Kuhta, Brian Mullins,
Delores Cohen, and Matt Lea. Sincerely yours, Louis J. Rodrigues
Director, Defense Acquisitions Issues Appendi xes International
Participation in the Joint Strike Appendi xI Fighter Program Full
collaborative partner full access to program data and structure,
including representative personnel within the program office
ability to influence requirement definition and performance
characteristics Great Britain is the only participant in this
category, committing $200 million for the concept demonstration
phase Associate partner limited access to data and limited
requirements influence representative personnel resident within
the Joint Strike Fighter program office all aspects of
participation are negotiated with the Joint Strike Fighter program
office Denmark, the Netherlands, and Norway formed a group and
committed a total of $30 million to participate at this level
Informed customer limited access to program information and
representation within the program office is negotiable no
influence on requirements Canada and Italy have committed $10
million each to participate at this level Major participants
recently created category; also referred to as Foreign Military
Sales participation or Fee for Service negotiate directly with
the program office for specific Joint Strike Fighter Program
information (e. g., Cost and Operational Performance Trade
processes and modeling and simulation studies) no representative
personnel resident within the Joint Strike Fighter program office
Singapore, Turkey, and Israel are currently participating at this
level Technology Readiness Levels and Their Appendi xII
Definitions Technology readiness level Description 1. Basic
principles observed and reported. Lowest level of technology
readiness. Scientific research begins to be translated into
applied research and development. Examples might include paper
studies of a technology's basic properties 2. Technology concept
and/ or application Invention begins. Once basic principles are
observed, practical applications can be formulated. invented. The
application is speculative and there is no proof or detailed
analysis to support the assumption. Examples are still limited to
paper studies. 3. Analytical and experimental critical Active
research and development is initiated. This includes analytical
studies and laboratory function and/ or characteristic proof of
studies to physically validate analytical predictions of separate
elements of the technology. concept. Examples include components
that are not yet integrated or representative. 4. Component and/
or breadboard validation Basic technological components are
integrated to establish that the pieces will work in laboratory
environment. together. This is relatively low fidelity compared to
the eventual system. Examples include integration of ad hoc
hardware in a laboratory. 5. Component and/ or breadboard
validation Fidelity of breadboard technology increases
significantly. The basic technological in relevant environment.
components are integrated with reasonably realistic supporting
elements so that the technology can be tested in a simulated
environment. Examples include high fidelity laboratory integration
of components. 6. System/ subsystem model or prototype
Representative model or prototype system, which is well beyond the
breadboard tested for demonstration in a relevant environment.
technology readiness level (TRL) 5, is tested in a relevant
environment. Represents a major step up in a technology's
demonstrated readiness. Examples include testing a prototype in a
high fidelity laboratory environment or in simulated operational
environment. 7. System prototype demonstration in an Prototype
near or at planned operational system. Represents a major step up
from TRL 6, operational environment. requiring the demonstration
of an actual system prototype in an operational environment, such
as in an aircraft, vehicle, or space. Examples include testing the
prototype in a test bed aircraft. 8. Actual system completed and
flight Technology has been proven to work in its final form and
under expected conditions. In qualified through test and
demonstration. almost all cases, this TRL represents the end of
true system development. Examples include developmental test and
evaluation of the system in its intended weapon systems to
determine if it meets design specifications. 9. Actual system
flight proven through Actual application of the technology in its
final form and under mission conditions, such as successful
mission operations. those encountered in operational test and
evaluation. In almost all cases, this is the end of the last bug
fixing aspects of true system development. Examples include using
the system under operational mission conditions. Source: Best
Practices: Better Management of Technology Development Can Improve
Weapon System Outcomes( GAO/NSIAD-99-162, July 30, 1999). Appendi
xI II Comments From the Department of Defense Note: GAO comments
supplementing those in the report text appear at the end of this
appendix. See comment 1. See comment 2. The following are GAO's
comments on the Department of Defense's (DOD) letter dated April
17, 2000. GAO Comments 1. DOD provided reevaluated technology
readiness levels to indicate that if the risk of integrating
critical technologies was not considered, then the technology risk
is expected to be at an appropriate level. Notwithstanding our
disagreement over whether integration risks should be considered
in this assessment, we believe DOD's standards are below minimum
acceptable levels. Only half of the critical technologies are
projected to be at readiness level 7 while the other half will
still be at readiness level 6. We disagree with DOD that readiness
level 6 is acceptable for low- risk entry into engineering and
manufacturing development. Leading commercial firms typically
insist on a readiness level 8-a higher standard than that used by
the Air Force Research Laboratory-before a technology can be
included on a product. 2. DOD noted that in addition to technology
readiness levels, other considerations were necessary to decide
when and where to insert new technologies into weapon system
programs. While DOD did not elaborate on what other considerations
were applicable to the Joint Strike Fighter Program, its similar
response to our prior work involving technology readiness levels
referred to such considerations as the increasing projected life
for new weapon systems, total ownership costs, and urgency based
upon threat assessments. Many of the technologies presented in
this current report directly address these other considerations
and are critical to the success of the Joint Strike Fighter
Program. Therefore, using TRLs in the Joint Strike Fighter Program
not only allows DOD to manage performance risk, but also to
address those technologies critical to meeting these other
considerations. With regard to risk assessment, leading commercial
firms who have adopted knowledge- based risk assessment
approaches, such as TRLs, have produced results such as more
technically advanced, higher quality products, in less time and at
a lower cost than their predecessors. (707386) Lett er GAO United
States General Accounting Office Page 1 GAO/NSIAD-00-74 Joint
Strike Fighter Acquisition Contents Page 2 GAO/NSIAD-00-74 Joint
Strike Fighter Acquisition Page 3 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition United States General Accounting Office
Washington, D. C. 20548 Page 3 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 4 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 5 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 6 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 7 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 8 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 9 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 10 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 11 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 12 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 13 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 14 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 15 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 16 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 17 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 18 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 19 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 20 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 21 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition B-281196 Page 22 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition Page 23 GAO/NSIAD-00-74 Joint Strike Fighter
Acquisition Page 24 GAO/NSIAD-00-74 Joint Strike Fighter
Acquisition Appendix I Page 25 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition Appendix II Page 26 GAO/NSIAD-00-74 Joint
Strike Fighter Acquisition Appendix III Appendix III Comments From
the Department of Defense Page 27 GAO/NSIAD-00-74 Joint Strike
Fighter Acquisition Appendix III Comments From the Department of
Defense Page 28 GAO/NSIAD-00-74 Joint Strike Fighter Acquisition
Appendix III Comments From the Department of Defense Page 29
GAO/NSIAD-00-74 Joint Strike Fighter Acquisition Appendix III
Comments From the Department of Defense Page 30 GAO/NSIAD-00-74
Joint Strike Fighter Acquisition Ordering Information The first
copy of each GAO report is free. Additional copies of reports are
$2 each. A check or money order should be made out to the
Superintendent of Documents. VISA and MasterCard credit cards are
accepted, also. Orders for 100 or more copies to be mailed to a
single address are discounted 25 percent. Orders by mail: U. S.
General Accounting Office P. O. Box 37050 Washington, DC 20013
Orders by visiting: Room 1100 700 4th St. NW (corner of 4th and G
Sts. NW) U. S. General Accounting Office Washington, DC Orders by
phone: (202) 512- 6000 fax: (202) 512- 6061 TDD (202) 512- 2537
Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512- 6000 using a
touchtone phone. A recorded menu will provide information on how
to obtain these lists. Orders by Internet: For information on how
to access GAO reports on the Internet, send an e- mail message
with info in the body to: info@ www. gao. gov or visit GAO's World
Wide Web home page at: http:// www. gao. gov To Report Fraud,
Waste, or Abuse in Federal Programs Contact one: Web site:
http:// www. gao. gov/ fraudnet/ fraudnet. htm e- mail: fraudnet@
gao. gov 1- 800- 424- 5454 (automated answering system) United
States General Accounting Office Washington, D. C. 20548- 0001
Official Business Penalty for Private Use $300 Address Correction
Requested Bulk Rate Postage & Fees Paid GAO Permit No. GI00
*** End of document. ***
|
NEWSLETTER
|
| Join the GlobalSecurity.org mailing list |
|
|
|
