Test and Evaluation: Impact of DOD's Office of the Director of Operational Test and Evaluation (Letter Report, 10/24/97, GAO/NSIAD-98-22)
Pursuant to a congressional request, GAO reviewed the Department of
Defense's (DOD) Office of the Director of Operational Test and
Evaluation's (DOT&E) operations and organizational structure for
overseeing operational testing, focusing on: (1) DOT&E's efforts and
their impact on the quality of operational testing and evaluation in
DOD; and (2) the strengths and weaknesses of the current organizational
framework in DOD for operational testing.
GAO noted that: (1) GAO's review of 13 case studies indicated that DOT&E
oversight of operational testing and evaluation increased the
probability that testing would be more realistic and more thorough; (2)
specifically, DOT&E was influential in advocating increasing the
reliability of the observed performance and reducing the risk of
unknowns through more thorough testing, conducting more realistic
testing, enhancing data collection and analysis, reporting independent
findings, and recommending follow-on operational test and evaluation
when suitability or effectiveness was not fully demonstrated prior to
initiating full-rate production; (3) the independence of DOT&E--and its
resulting authority to report directly to Congress--is the foundation of
its effectiveness; (4) that independence, along with its legislative
mandate, provides sufficient freedom and authority to exercise effective
oversight of the operational testing and evaluation of new systems
before a decision is made to begin full-rate production; (5) DOT&E can
reduce the risk that systems are not adequately tested prior to the
full-rate production decision but DOT&E cannot ensure that: (a) only
systems whose operational effectiveness and suitability have been
demonstrated through operational testing will proceed to the full-rate
production decision; or (b) new fielded systems will accomplish their
missions as intended or that the fielded systems are safe, survivable,
and effective; (6) DOT&E management must balance its oversight
responsibilities for operational testing with the broader acquisition
priorities of program managers and service test agencies; (7) though
supportive of DOT&E's mission and independence, program and service
representatives frequently considered the time, expense, and resources
expended to accommodate DOT&E concerns to be ill-advised; (8) several
current trends may challenge DOT&E's ability to manage its workload and
its ability to impact operational test and evaluation: (a) service
challenges to DOT&E's authority to require and oversee follow-on
operational testing and evaluation; (b) a decline in resources available
for oversight; (c) an expansion of DOT&E involvement in activities other
than oversight of major acquisition programs; (d) participation of DOT&E
in the acquisition process as a member of working-level integrated
product teams; and (e) greater integration of developmental and
operational testing; and (9) these trends make it imperative that DOT&E
prioritize its workload to achieve a balance between the oversight of
major defense acquisition programs and other initiatives important to
the quality of operational test and evaluation.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: NSIAD-98-22
TITLE: Test and Evaluation: Impact of DOD's Office of the Director
of Operational Test and Evaluation
DATE: 10/24/97
SUBJECT: Testing
Weapons systems
Product performance evaluation
Military systems analysis
Cost effectiveness analysis
Interagency relations
Prioritizing
Military cost control
Defense procurement
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** <info@www.gao.gov> **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Honorable
William V. Roth and the Honorable
Charles E. Grassley, U.S. Senate
October 1997
TEST AND EVALUATION - IMPACT OF
DOD'S OFFICE OF THE DIRECTOR OF
OPERATIONAL TEST AND EVALUATION
GAO/NSIAD-98-22
DOT&E Impact
(973444)
Abbreviations
=============================================================== ABBREV
AWACS - airborne warning and control system
DOD - Department of Defense
DODIG - Department of Defense Inspector General
DOT&E - Office of the Director of Operational Test and Evaluation
IDA - Institute for Defense Analyses
JointSTARS - Joint Surveillance Target Attack Radar System
LCAC - landing craft air cushion
LRIP - low-rate initial production
OSD - Office of the Secretary of Defense
RSIP - radar system improvement program
USD(A&T) - Under Secretary of Defense (Acquisition and Technology)
Letter
=============================================================== LETTER
B-276799
October 24, 1997
The Honorable William V. Roth
The Honorable Charles E. Grassley
United States Senate
The Department of Defense (DOD) has proposed that the practices and
policies of the Office of the Director of Operational Test and
Evaluation (DOT&E) be modified to reduce the time and cost of
developing and fielding new weapon systems. To help focus
deliberations on DOD's proposal, you asked us to review DOT&E's
operations and organizational structure for overseeing operational
testing. Specifically, you asked us to assess (1) DOT&E's efforts
and their impact on the quality of operational testing and
evaluation\1 in DOD and (2) the strengths and weaknesses of the
current organizational framework in DOD for operational testing. As
part of our review, we conducted 13 case studies of the testing of
individual weapon systems. (Our scope and methodology are described
in app. I, and brief descriptions of the 13 weapon systems are
provided in app. II.)
--------------------
\1 The term "operational test and evaluation" means (1) the field
test, under realistic conditions, of any item or key component of a
weapon system, equipment, or munition for the purpose of determining
the effectiveness and suitability of the weapon, equipment, or
munition for use in combat by typical military users and (2) the
evaluation of the results of the test.
BACKGROUND
------------------------------------------------------------ Letter :1
In 1983, Congress established DOT&E to coordinate, monitor, and
evaluate operational testing of major weapon systems.\2 As part of
the Office of the Secretary of Defense (OSD), DOT&E is separate from
the acquisition community that conducts developmental and operational
testing and therefore is in a position to provide the Secretary and
Congress with an independent view. Congress created DOT&E in
response to reports of conflicts of interest in the acquisition
community's oversight of operational testing leading to inadequate
testing of operational suitability\3 and effectiveness\4 and the
fielding of new systems that performed poorly. (DOD's system
acquisition process is described in app. III.)
By law, DOT&E serves as the principal adviser on operational test and
evaluation in DOD and bears several key responsibilities, including
-- monitoring and reviewing all operational test and evaluation in
DOD,
-- reporting to the Secretary of Defense and congressional
committees whether the tests and evaluations of weapon systems
were adequate and whether the results confirmed that the system
is operationally suitable and effective for combat before a
decision is made to proceed to full-rate production, and
-- submitting to the Secretary of Defense and congressional
decisionmakers an annual report summarizing operational test and
evaluation activities during the preceding fiscal year.
In 1993, DOD's advisory panel on streamlining and codifying
acquisition laws\5 concluded that DOT&E was impeding the goals of
acquisition reform by (1) promoting unnecessary oversight, (2)
requiring excessive reporting detail, (3) inhibiting the services'
discretion in testing, and (4) limiting participation of system
contractors in operational tests where such involvement is deemed
necessary by the services. The following year, DOD proposed
legislative changes that would have reduced the scope and authority
of DOT&E. In testimony, we opposed these changes because they were
directed at perceived rather than documented problems and would
undermine a key management control over the acquisition
process--independent oversight of operational test and evaluation.\6
Although the legislative proposals were not adopted, in 1995 the
Secretary of Defense implemented several operational test and
evaluation initiatives in the Department to (1) involve operational
testers earlier in the acquisition process, (2) use models and
simulations effectively, (3) combine tests where possible, and (4)
combine tests and training. The goals of these initiatives included
saving time and money by identifying and addressing testing issues
earlier in the acquisition process; merging or closely coordinating
historically distinct phases, such as developmental and operational
testing to avoid duplication; and using existing technologies and
training exercises to create realistic and affordable test
conditions.
--------------------
\2 P.L. 98-94 sec. 1211(a)(1), 97 Stat. 684. DOT&E's legislation
is now codified at 10 U.S.C. 139.
\3 DOD defines "operationally suitable" as the degree to which a
system can be placed satisfactorily in field use, with consideration
given to such factors as availability, compatibility,
transportability, interoperability, reliability, wartime usage rates,
maintainability, safety, and supportability.
\4 DOD defines "operationally effective" as the overall degree of
mission accomplishment of a system when used by representative
personnel in the environment planned or expected for operational
employment of the system, considering organization, doctrine,
tactics, survivability, vulnerability, and threat.
\5 Established under section 800 of the National Defense
Authorization Act for Fiscal Year 1991
(P.L. 101-510, 1990).
\6 Acquisition Reform: Role of Test and Evaluation in System
Acquisition Should Not Be Weakened (GAO/T-NSIAD-94-124, Mar. 22,
1994).
RESULTS IN BRIEF
------------------------------------------------------------ Letter :2
Our review of 13 case studies indicated that DOT&E oversight of
operational testing and evaluation increased the probability that
testing would be more realistic and more thorough.\7 Specifically,
DOT&E was influential in advocating increasing the reliability of the
observed performance and reducing the risk of unknowns through more
thorough testing; conducting more realistic testing; enhancing data
collection and analysis; reporting independent findings; and
recommending follow-on operational test and evaluation when
suitability or effectiveness was not fully demonstrated prior to
initiating full-rate production.
The independence of DOT&E--and its resulting authority to report
directly to Congress--is the foundation of its effectiveness. That
independence, along with its legislative mandate, provides sufficient
freedom and authority to exercise effective oversight of the
operational testing and evaluation of new systems before a decision
is made to begin full-rate production. In the conduct of its
oversight, DOT&E (1) executes its approval authority over test and
evaluation master plans and operational test plans and (2) provides
independent annual and summary reports on the test and evaluation of
individual weapon systems to the Secretary of Defense and Congress.
DOT&E can reduce the risk that systems are not adequately tested
prior to the full-rate production decision. But DOT&E cannot ensure
that (1) only systems whose operational effectiveness and suitability
have been demonstrated through operational testing will proceed to
the full-rate production decision or (2) new fielded systems will
accomplish their missions as intended or that the fielded systems are
safe, survivable, and effective. Moreover, service and acquisition
officials have argued that DOT&E does not have the independent
authority to require and approve service-conducted follow-on
operational test and evaluation after full-rate production begins.
In addition, the Office is not currently required to report on
whether new systems are both operationally suitable and effective
before they are fielded.
DOT&E management must balance its oversight responsibilities for
operational testing with the broader acquisition priorities of
program managers and service test agencies. Though supportive of the
Office's mission and independence, program and service
representatives frequently considered the time, expense, and
resources expended to accommodate DOT&E concerns to be ill-advised.
Service officials contended that the additional testing requested by
DOT&E was either unnecessary for determining the operational
effectiveness or suitability of a program or unrealistic in light of
the limitations in the services' testing resources.
DOT&E must manage multiple oversight, advisory, and coordination
responsibilities. Several current trends may challenge DOT&E's
ability to manage its workload and its ability to impact operational
test and evaluation. These trends include (1) service challenges to
DOT&E's authority to require and oversee follow-on operational
testing and evaluation, (2) a decline in resources available for
oversight, (3) an expansion of DOT&E involvement in activities other
than oversight of major acquisition programs, (4) participation of
DOT&E in the acquisition process as a member of working-level
integrated product teams, and (5) greater integration of
developmental and operational testing. These trends make it
imperative that DOT&E prioritize its workload to achieve a balance
between the oversight of major defense acquisition programs and other
initiatives important to the quality of operational test and
evaluation.
--------------------
\7 Aspects of realism can include (1) equipment and personnel placed
under realistic stress and operational tempo, (2)
threat-representative forces, (3) end-to-end testing, (4) realistic
combat tactics, (5) operationally realistic environments and targets,
(6) countermeasured environments, (7) interfacing systems, (8)
terrain and environmental conditions, and (9) contractor involvement.
DOT&E ADVOCATES MORE THOROUGH
TESTING THAN THE SERVICES
------------------------------------------------------------ Letter :3
A frequent complaint among representatives of the services'
operational testing agencies was that DOT&E frequently demanded more
tests than were proposed by the operational test agencies in draft
master plans or test plans. Operational test agency representatives
contended that the additional testing was either unnecessary for
determining the operational effectiveness or suitability of a program
or unrealistic in light of the limitations in the services' testing
resources. However, our review indicated that DOT&E urged more
testing to reduce the level of risk and number of unknowns prior to
the decision to begin full production, while program and service
officials typically sought less testing and were willing to accept
greater risk when making production decisions. The additional
testing DOT&E advocated, often over the objections of service
testers, served to meet the underlying objectives of operational
testing--to reduce the uncertainty and risk that systems entering
full-rate production would not fulfill their requirements.
The impact of DOT&E oversight varies with the system under
development. Table 1 summarizes the types of impacts that DOT&E
advocated or facilitated in operational testing among the 13 cases we
studied. While the impacts vary, one consistent pattern in our case
studies was a reduction in uncertainty regarding the weapon systems'
suitability or effectiveness prior to the full-rate production
decision. Each of the impacts are discussed in more detail in tables
2-6 and in subsequent sections.
Table 1
Types of Impacts on the Operational
Testing of 13 Systems Due to DOT&E
Oversight
Follow-on
operational
More Enhancement DOT&E's test and
testing More s made in conclusion evaluation
advocated realism data deviated advocated and
and included in collection from the planned or
System conducted test design or analysis service's conducted
----------- ----------- ----------- ----------- ----------- ---------------
AH-64D X X X X X
Longbow
Apache
helicopter
ASPJ\a X X
jammer
C-17A X X
aircraft
E-3 AWACS\b X X
(RSIP\c)
F-22 X X X
fighter
Javelin X X X
missile
Joint X X X X
STARS\d
LPD-17 X X
assault
ship
M1A2 tank X X X
Sensor X X X X
fuzed
weapon
Standard X X
missile
Tomahawk X X
Weapon
System
V-22 X
aircraft
--------------------------------------------------------------------------------
Note: The absence of an "X" does not necessarily indicate the
absence of DOT&E impact. For example, blanks may occur where DOT&E
and the service agreed on issues; however, the deterrent effect of
DOT&E oversight is unquantifiable. In addition, blanks may occur
because the system has not yet progressed through the entire
acquisition process.
\a Airborne Self-Protection Jammer.
\b Airborne Warning and Control System.
\c Radar System Improvement Program.
\d Surveillance Target Attack Radar System.
DOT&E OVERSIGHT LED TO MORE
TESTING THAN PROPOSED BY THE
OPERATIONAL TEST AGENCIES
---------------------------------------------------------- Letter :3.1
Two of DOT&E's typical concerns in reviewing service test plans are
that the proposed test methodologies enable (1) comparisons of a
system's effectiveness through side-by-side testing between the
existing and modified systems and (2) assessments of a system's
reliability through a sufficient number of test repetitions. Table 2
illustrates examples of cases where additional testing was conducted
at DOT&E's insistence or with DOT&E's support to alleviate these and
other types of effectiveness and suitability concerns.
Table 2
Examples of Programs that Expanded
Testing Due to DOT&E Oversight
System Expanded testing Impact
------------------ ----------------------------- -----------------------------
AH-64D Longbow DOT&E insisted that the Army Testers were able to
Apache helicopter include a baseline AH-64A demonstrate the gunnery
company in gunnery and force- performance improvements of
on-force exercises to ensure the AH-64D. These
direct comparability with the improvements included that
Longbow. (1) the AH-64D had 300
instances of lethality
compared to 75 for the AH-
64A, (2) the AH-64D was
approximately 8 times more
survivable than the AH-64A,
and (3) the AH-64D had zero
fratricide instances compared
to 34 for the AH-64A.
ASPJ jammer In follow-on operational test The revised test plan enabled
and evaluation of the F-14D testers to address the
begun in 1995, DOT&E insisted critical operating issue--
that the scope of the test that the F-14D is more
plan address the ASPJ's survivable with the ASPJ as
contribution to the part of its electronic
aircraft's survivability-- warfare suite than without
not merely the jammer's it.
compatibility with the
aircraft's avionics. This
expansion of the scope
necessitated an additional 18
open air flight tests to
measure the ASPJ's
effectiveness against air-
to-air threats and a
requirement to gather
suitability data pertaining
to ASPJ, including its built-
in test equipment.\a
C-17 aircraft The ability to safely perform DOT&E forced testing that
a mass personnel airdrop confirmed operational
while flying in close limitations, and the Army has
formation is a key Air Force yet to approve mass airdrops
capability needed to conduct of personnel from C-17s
a strategic brigade airdrop. flying in close formation.
DOT&E insisted that an Operational tests identified
airdrop of a brigade slice of specific problems with the C-
personnel and equipment be 17's airdrop capability--
done. The Air Force's that with the air turbulence
position was that the airdrop created in the wake of the
was unnecessary before the aircraft, flying in close
full-rate production decision formation can cause the
and that the use of the parachutes dropping from
aircraft in airdrops would be aircraft to oscillate,
determined after the full- partially deflate, or
rate production decision. collapse. These conditions
could result in serious
injury or death to
paratroopers.
F-22 fighter DOT&E and the Air Force The use of multiple testing
agreed to a balanced approach and evaluation techniques
of open-air testing, full will reduce uncertainty in
mission simulation, and system effectiveness more
digital models against then- than the Air Force's initial
current and future threats in preference to use test
an overall F-22 and F-15 results to support evaluation
effectiveness analysis. by modeling.
Javelin missile DOT&E insisted that the The test provided additional
system undergo additional confidence that the weapon
operational testing prior to system's modifications had
the full-rate production not affected Javelin's
decision in 1997 because over suitability for combat.
50 design changes had been
made to the system since
initial operational test and
evaluation in 1993. The Army
claimed that successful
passage of technical tests
was adequate assurance of
suitability for combat and
did not originally intend to
conduct operational tests
until 1998, over a year after
the start of full-rate
production.\b
Javelin missile Based on data collected from Before the additional test
(con't) initial operational testing, was conducted, the Army
DOT&E disagreed with the modified components of the
Army's conclusion that the command launch unit to
Javelin was suitable for increase its reliability. The
combat and supported the subsequent test demonstrated
Army's operational test that the modifications were
agency in requiring the successful. The test also
program manager to conduct an provided two additional
operational test to confirm benefits. First, missile
the unit's reliability. failures during the test led
to discovery and correction
of a design flaw that
prevented the missiles from
leaving the launch tube when
the gunner pulled the
trigger. Second, while
developing the test plan,
DOT&E discovered that the
Army had no Javelin-specific
tactical doctrine and
recommended the Army study
this deficiency. As a result,
the Army developed
operational tactics to guide
officers in integrating
Javelin with other antitank
systems.
LPD-17 assault The originally proposed The expanded scope of the
ship operational test for the LPD- test plan will more closely
17 consisted of at-sea encompass the range of system
steaming and some landing requirements as well as
craft air cushion (LCAC) enhance the realism of the
operations. DOT&E forced the test scenario.
incorporation of full-scale
assault operations with
LCACs, aircraft, ground
assault equipment, and
personnel.
Sensor fuzed DOT&E insisted on a second The additional testing of
weapon phase of operational test and system issues not fully
evaluation before the full- tested in the first phase
rate production decision that (such as additional
the Air Force did not want to countermeasures, multiple
conduct. releases, and an alternate
target formation) reduced
uncertainty in system
effectiveness and
reliability.
Standard missile DOT&E insisted on and DOT&E's insistence on
SM-2 obtained five flight tests of additional testing has
the User Operational lowered the technical risk of
Evaluation System SM-2 block the program by providing for
IVA missile, a theater a series of tests to
ballistic missile defense establish system level
system. The Navy planned only validation. These tests will
two at-sea safety flights help to demonstrate the level
against nonthreat- of reliability and
representative targets. Some effectiveness of the SM-2
of the new flight tests will block IVA missile.
be conducted against threat-
representative targets from
the integrated AEGIS system.
--------------------------------------------------------------------------------
\a See Electronic Warfare (GAO/NSIAD-96-109R, Mar. 1, 1996).
\b See Army Acquisition: Javelin Is Not Ready for Multiyear
Procurement (GAO/NSIAD-96-199, Sept. 26, 1996).
DOT&E OVERSIGHT LED TO MORE
REALISTIC TESTING THAN
PROPOSED BY THE OPERATIONAL
TEST AGENCIES
---------------------------------------------------------- Letter :3.2
Table 3 illustrates examples where the design or conduct of
operational testing was modified at DOT&E's insistence or with
DOT&E's support to increase the realism of test conditions and reduce
the uncertainty of system suitability or effectiveness.
Table 3
Examples of Programs That Conducted More
Realistic Testing Due to DOT&E Oversight
Enhanced realism in tests Impact
System ------------------------- --------------------------
AH-64D Longbow DOT&E required a The testing revealed
Apache helicopter demanding air defense operational limitations of
network, directly the AH-64D variant without
intervening to ensure the fire control radar and
that a specific threat thereby raised the issue
would be present in the of the appropriate mix of
force-on-force trials. variants to procure. The
AH-64D variant with the
fire control radar was
unable to reduce the air
defense threat
sufficiently to allow the
variant without the fire
control radar to move into
battle positions without
significant possibility of
being engaged by those air
defense units.
E-3 AWACS (RSIP) DOT&E insisted that (1) Reduced uncertainty of
mission crews comprise a system effectiveness
cross section of typical because (1) AWACS
AWACS aircrew members, personnel from the
(2) RSIP be employed engineering and
against an array of developmental test sorties
actual Soviet and other were excluded, resulting
threats, and (3) the in the use of two test
system be used in eight crews comprised of a
different terrain typical ratio of U.S. and
combinations in both the Canadian deployment
United States and Europe. personnel and (2) actual
threats and realistic
environments were
incorporated.
F-22 fighter DOT&E was instrumental in The credibility of the
ensuring that a full full mission simulator
mission simulator was (used to compare relative
developed for comparison mission effectiveness of
testing using validated the F-15 and F-22) will be
software and hardware, enhanced.
insisting that the
functionality and
fidelity of the
simulation be validated
by open air flight data.
DOT&E insisted that the The confidence level of
test and evaluation the model's prediction is
master plan include high enhanced by introducing
tempo demonstrations to surge data from actual
test the required sortie operations.
generation rate.
Javelin missile DOT&E required Army The Army found that
troops to carry the missiles carried during
missile a representative the test failed to leave
distance during missions the launch tube because of
and prior to actual a faulty design of the
firings to ensure that external restraining pin-
the missile's reliability wiring harness. This
would not be affected by finding led the Army to
field handling. redesign the assembly,
which prevented potential
missile malfunctions in
combat situations.
Joint STARS In the development of the Deployment of the system
test plan, DOT&E to Bosnia precluded
encouraged participation testing at the National
of Air Force and Army Training Center, but the
testers in training test design precedent was
exercises at the National established.
Training Center as a way
to enhance test realism.
Sensor fuzed weapon During the second phase The enhanced realism of
of initial operational testing reduced
test and evaluation, uncertainty of system
DOT&E required an effectiveness at low
extensive validation of altitudes and confirmed
the infrared signature decreased effectiveness as
and the use of altitude, dive angle, and
countermeasures, insisted time of flight increase.
on all-weather and all-
altitude testing at
numerous test sites;
insisted on realistic and
comprehensive
countermeasures testing;
and ensured realistic
targets were made
available for testing.
Standard missile SM-2 During the review of the The need for realistic
Navy's draft test and aerial targets is a
evaluation master plan significant issue cutting
for the SM-2 block IV, across all Navy surface
DOT&E identified antiair warfare programs
inadequacies in aerial such as the Phalanx Close-
target programs and In Weapon System and the
required that threat- Rolling Airframe Missile,
representative targets be as well as the various SM-
available for operational 2 blocks.
testing.
Tomahawk Weapon System DOT&E was instrumental in The use of realistic
ensuring that only ship operators reduced
crews were used during uncertainty in system
the testing of the all- reliability and
up-rounds\a and the effectiveness.
Tomahawk Weapon Control
System. Support personnel
conducted testing, while
contract personnel
maintained the equipment
as they do in actual
operations.
V-22 aircraft DOT&E has emphasized the The test program has been
effects of the V-22 revised to conduct
downwash on personnel and downwash testing in 1997
material in the vicinity rather than 1999 to
of the hovering aircraft address the concerns of
and the need to test in DOT&E and others.
more realistic ship and
landing zone
environments.
--------------------------------------------------------------------------------
\a Each Tomahawk missile variant is contained within a pressurized
canister to form an all-up-round.
DOT&E OVERSIGHT LED TO
CHANGES IN THE DATA ANALYSIS
PLAN
---------------------------------------------------------- Letter :3.3
DOT&E can insist on or support changes in data analysis plans that
provide more meaningful analyses for decisionmakers. Table 4
illustrates instances in which DOT&E altered the proposed data
collection or analysis plans to enhance the reliability or utility of
the test data.
Table 4
Examples of Programs in Which Changes
Were Made in the Data Analysis Plan Due
to DOT&E Oversight
Changes in data analysis
plan Impact
System ------------------------- --------------------------
AH-64D Longbow DOT&E insisted on DOT&E input allowed
Apache helicopter performance criteria to testers to more accurately
assess the superiority of compare the AH-64D to the
the AH-64D over the AH- AH-64A in quantifiable
64A. The criteria--a 20- categories of lethality,
percent improvement--had survivability, and
not formally been fratricide.
included in the test and
evaluation master plan.
DOT&E required measures
that addressed the number
of targets killed and
helicopters lost.
ASPJ jammer DOT&E required the Navy The Navy determined that
to test the ASPJ against the ASPJ was effective
the type of missile that against that threat.
shot down an F-16 over
Bosnia in June 1995.\a
DOT&E was instrumental in Independent oversight of
establishing a ASPJ's suitability
requirement to gather assessment confirmed
suitability data on its ongoing concerns with
built-in test equipment. system reliability.
While the contractor
reported improvement in
previously unreliable
built-in test equipment,
DOT&E questioned the data
collection and
interpretation.
E-3 AWACS (RSIP) DOT&E insisted that Reduced uncertainty of
service personnel be system effectiveness
trained to operate because the contractor was
contractor data removed from data
extraction systems, processing ensuring test
thereby removing the integrity.
contractor from the
process and ensuring data
integrity. DOT&E reviewed
a major radar failure and
discovered an error in
the technical path
described by the service.
Joint STARS DOT&E insisted that the The change in test
Air Force modify its measures resulted in test
original technical data that were more
requirements to include operationally relevant to
measures of effectiveness system effectiveness.
that directly addressed
the missions of
surveillance, targeting,
and battlement
management. DOT&E
stressed differentiation
between user and system
requirements.
LPD-17 assault ship DOT&E insisted on The change in test
measures of effectiveness measures will result in
that addressed the test data that are more
movement of men and operationally relevant to
equipment ashore rather system effectiveness.
than the Navy's original
requirements that focused
on technical
specifications.
M1A2 tank DOT&E required that the Reduced uncertainty of
Army use credible data improved effectiveness and
for the determination of suitability of the M1A2
reliability in follow-on compared with the M1A1.
operational test and
evaluation. The Army
proposed the use of
failures and other
secondary measures that
would not provide a
credible basis for
reversing the results of
initial operational test
and evaluation. DOT&E
insisted that the
operational testing be
conducted to compare the
M1A2 with the M1A1.
Several improvements in
the M1A2 addressed
command and control that
could not be directly
measured. By conducting
several operations with
both tanks, the
difference in movements
and coordination could be
examined to determine the
value of the command and
control improvements. By
adding uncertainty to
test scenarios, DOT&E
enabled the Army
operational test agency a
means to identify
differences between the
M1A1 and M1A2 models.
Tomahawk Weapon System DOT&E was instrumental in More rigorous data
ensuring that the collection and validation
effectiveness of mission reduced uncertainty of
planning systems was system effectiveness.
validated using high-
fidelity models and
simulations and that bit-
by-bit checks were
conducted to validate the
effectiveness of
functional operations of
the planning system.
--------------------------------------------------------------------------------
\a See Airborne Self-Protection Jammer (GAO/NSIAD-97-46R, Jan. 29,
1997).
DOT&E INTERPRETED THE
RESULTS OF SOME TESTING LESS
FAVORABLY THAN THE
OPERATIONAL TEST AGENCIES
---------------------------------------------------------- Letter :3.4
DOT&E's independent analysis of service test data may confirm or
dispute the results and conclusions reported by the service. In the
cases described in table 5, DOT&E's analysis of service operational
test and evaluation data resulted in divergent, often less favorable
conclusions than those reached by the service.
Table 5
Examples of Programs in Which DOT&E and
Service Conclusions Differed
System Conflicting test results Impact
------------------ ----------------------------- -----------------------------
AH-64D Longbow DOT&E's independent analysis The Army will conduct a
Apache helicopter of the test data identified a series of simulations and
predominant firing technique additional missile firings to
that had not previously been determine the factors
identified as useful. Though affecting the overall
the technique was never effectiveness of the
anticipated to be used so technique and its relative
extensively and had not been effectiveness to the primary
considered in the development modes of engagement, thereby
of the Longbow's tactics, increasing certainty in
techniques, and procedures, system effectiveness.
DOT&E determined that over
half of the operational test
engagements were conducted
using this technique.
Nonetheless, this revelation
was not in the Army test
report.
Javelin missile DOT&E did not use reliability The Army made numerous design
data from the pre-initial changes to the launch unit
operational test and and round before the
evaluation period because the contractor initiated low-
data were not realistic; as a rate production.
result, DOT&E found the
command launch unit failed to
meet its reliability
criteria, differing from the
Army's report.
Joint STARS DOT&E disagreed with the Air DOT&E's Beyond-LRIP report
Force operational test indicated not only the Joint
agency's positive assessment STARS' disappointing test
of the operational results but also the need for
suitability and effectiveness extensive follow-on
of Joint STARS following its operational test and
deployment to Operation Joint evaluation. Subsequently, the
Endeavor. DOT&E concluded Joint STARS acquisition
that Joint STARS met one of decision memorandum required
three critical operational that the test and evaluation
effectiveness issues--with master plan be updated and
limitations, while the other that follow-on operational
two effectiveness issues test and evaluation address
could not be determined. the deficiencies identified
Overall, the Air Force's in initial operational test
conclusion was "suitable with and evaluation by DOT&E.
deficiencies"; DOT&E's
conclusion was "as tested is
unsuitable."\a DOT&E and the
Air Force operational test
agency also disagreed on how
to report data when terrain
masking occurred. DOT&E
objected to the Air Force's
phrasing "nothing significant
to report," when in fact
nothing could be seen.
M1A2 tank DOT&E evaluated the tank as Follow-on operational test
not operationally suitable-- and evaluation was conducted
a finding at odds with Army to determine if the Army's
testers. DOT&E determined design changes had improved
that the tank was unreliable the system. The suitability
and unsafe due to uncommanded problems persisted and the
turret movements, hot follow-on operational test
surfaces that caused contact and evaluation was suspended.
burns, and inadvertent firing New design changes were made
of the .50 caliber machine and a second follow-on
gun. operational test and
evaluation was conducted,
which determined that the
safety issues were resolved
and that the tank is now
operationally suitable.
Sensor fuzed Based on the results of the As a result of the unresolved
weapon first phase of operational issues in 1992, a second
test and evaluation ending in phase of operational test and
1992, the Air Force concluded evaluation was planned and
that the sensor fuzed weapon executed, leading DOT&E to
was "suitable and effective conclude in 1996 that the
for combat." In contrast, system was operationally
DOT&E concluded from the same suitable and effective--when
tests that the system was employed at low altitude
only "potentially using level or shallow angle
operationally effective and dive deliveries.
suitable."
--------------------------------------------------------------------------------
\a See Tactical Intelligence: Joint STARS Full-Rate Production
Decision Was Premature and Risky (GAO/NSIAD-97-68, Apr. 25, 1997).
DOT&E RECOMMENDED FOLLOW-ON
OPERATIONAL TEST AND
EVALUATION
---------------------------------------------------------- Letter :3.5
When DOT&E concludes that a weapon system has not fully demonstrated
operational suitability or effectiveness, or if new testing issues
arise during initial operational test and evaluation, it may
recommend that follow-on operational test and evaluation be done
after the full-rate production decision. Table 6 identifies
follow-on operational test and evaluation that DOT&E supported.
Table 6
Examples of Programs in Which DOT&E
Called for Follow-On Operational Test
and Evaluation
Advocated follow-on
operational test and
System evaluation Impact
------------------ ----------------------------- -----------------------------
AH-64D Longbow DOT&E sought follow-on The use of this technique was
Apache helicopter operational test and not fully anticipated prior
evaluation to characterize to initial operational test
the Hellfire missile's and evaluation. Its use
performance when using lock- provided an unexpected level
on before launch-inhibit of survivability for the AH-
technique. This method of 64D crews. This technique had
engagement enables crews to been subjected to little, if
immediately take cover after any, developmental testing.
target detection and fire at Further testing will
moving targets from those establish its probability of
covered locations. This hit. The Army operational
method was used in over half test agency plans to fire 8
of the operational test to 10 missiles in August
engagements, though it had 1998.
not been considered
sufficiently significant to
incorporate in the Longbow's
tactics, techniques, and
procedures.
C-17A aircraft DOT&E urged follow-on The Air Force has undertaken
operational test and further testing with the Army
evaluation to demonstrate the to overcome system
system's ability to meet deficiencies and demonstrate
operational readiness effectiveness. The Army is
objectives, including formulating a time
combination and brigade requirement of about 30
airdrops, and software minutes for completing a
maturity. strategic airdrop. The C-17
currently has a 5.5 minute
aircraft separation
restriction that essentially
prohibits formation flying
and therefore requires 2.5
hours to complete a strategic
airdrop. This resulted in
continuing efforts to resolve
these operational
limitations.
F-22 aircraft DOT&E insisted that the test The commitment to test these
and evaluation master plan capabilities is formally
require follow-on operational acknowledged.
test and evaluation on two
capabilities that will not be
released until after initial
operational test and
evaluation: employment of the
Joint Direct Attack Munition
and Cruise Missile Defense.
Joint STARS DOT&E stated in its Joint The Joint STARS acquisition
STARS Beyond-LRIP report that decision memorandum directed
only 18 of 71 performance additional testing to address
criteria tested were suitability deficiencies in
demonstrated by the system logistics and software.
and that further testing was
required for the remaining
53.
M1A2 tank The M1A2, during initial The Army executed a program
operational test and to correct the deficiencies
evaluation in 1993, failed to identified during initial
meet the combat mission operational test and
reliability threshold, evaluation and conducted
encountered an excessive follow-on operational test
number of battery failures, and evaluation in 1995.
consumed Suitability issues, such as
15 percent more fuel, uncommanded turret movement
exhibited uncommanded main and power loss, were again
gun/turret movements and experienced. The follow-on
inadvertent .50 caliber operational test and
machine-gun firing that made evaluation was put on hold
the tank unsafe. DOT&E, until additional corrective
through a Secretary of actions could be applied.
Defense letter accompanying Follow-on operational test
the M1A2 Beyond-LRIP report and evaluation resumed in
to Congress, required follow- July 1996. The safety
on operational test and problems were found to have
evaluation on M1A2 been addressed by the design
suitability issues when the changes, and there were no
Army claimed it was observed instances of the
unnecessary. problems experienced during
initial or beginning follow-
on operational test and
evaluation.
Sensor fuzed The test and evaluation Follow-on operational test
weapon master plan for the second and evaluation ensures
phase of operational test and further investigation of
evaluation specified a series system limitations known at
of follow-on operational test the time of the full-rate
and evaluations that would production decision.
address how well the addition
of the Wind Compensated
Munition Dispenser and the
preplanned product
improvements will rectify
system limitations.
--------------------------------------------------------------------------------
STRENGTHS AND WEAKNESSES OF
CURRENT ORGANIZATIONAL
FRAMEWORK
------------------------------------------------------------ Letter :4
The existence of a healthy difference of opinion between DOT&E and
the acquisition community is a viable sign of robust oversight. In
nearly all of the cases we reviewed, the services and DOT&E cited at
least one testing controversy. For example, services differ on how
they view the relationship between operational testing and their
development of tactics, techniques, and procedures. In addition,
DOT&E's ability to independently view the development and testing of
new systems across the services brings value to the context of
testing. However, several current trends have the potential to
adversely affect DOT&E's independence and its ability to affect
operational test and evaluation, including (1) service challenges to
DOT&E's authority to require and oversee follow-on operational
testing and evaluation, (2) declining resources available for
oversight, (3) the management of limited resources to address
competing priorities, (4) DOT&E's participation in the acquisition
process as a member of the program manager's working-level integrated
product teams, and (5) greater integration of developmental and
operational testing. DOT&E's impact on operational testing is
dependent upon its ability to manage these divergent forces while
maintaining its independence.
INDEPENDENCE IS THE KEY TO
DOT&E'S EFFECTIVENESS
---------------------------------------------------------- Letter :4.1
Although the acquisition community has three central
objectives--performance, cost, and schedule--DOT&E has but one:
operational testing of performance. These distinct priorities lead
to testing disputes. Characteristically, the disputes for each
system we reviewed revolved around questions of how, how much, and
when to conduct operational testing, not whether to conduct
operational testing. Conflicts encompassed issues such as (1) how
many and what types of tests to conduct; (2) when testing should
occur; (3) what data to collect, how to collect it, and how best to
analyze it; and (4) what conclusions were supportable, given the
analysis and limitations of the test program. The foundation of most
disputes lay in different notions of the costs and benefits of
testing and the levels of risk that were acceptable when making
full-rate production decisions. DOT&E consistently urged more
testing (and consequently more time, resources, and cost) to reduce
the level of risk and number of unknowns before the decision to
proceed to full-rate production, while the services consistently
sought less testing and accepted more risk when making production
decisions. Among our case studies, these divergent dispositions
frequently led to healthy debates about the optimal test program, and
in a small number of cases, the differences led to contentious
working relations.
In reviews of individual weapon systems, we have consistently found
that testing and evaluation is generally viewed by the acquisition
community as a requirement imposed by outsiders rather than a
management tool to identify, evaluate, and reduce risks, and
therefore a means to more successful programs. Developers are
frustrated by the delays and expense imposed on their programs by
what they perceive as overzealous testers. The program office
strives to get the program into production despite uncertainties that
the system will work as promised or intended. Therefore, reducing
troublesome parts of the acquisition process--such as operational
testing--is viewed as a means to reduce the time required to enter
production.
Nonetheless, the commanders and action officers within the service
operational test agencies were nearly unanimous in their support for
an independent test and evaluation office within OSD. For example,
the Commander of the Army's Operational Test and Evaluation Command
commended the style and orientation of the current DOT&E Director and
affirmed the long-term importance of the office and its independent
reporting responsibilities to Congress. The Commander of the Navy's
Operational Test and Evaluation Force stated that the independence of
both DOT&E and the operational test agency was an essential element
in achieving their common goal of ensuring that new programs pass
sufficiently rigorous and realistic operational testing prior to
fielding. The Commander of the Air Force's Operational Test and
Evaluation, while critical of DOT&E oversight of several major weapon
systems, said that the services were well served by DOT&E's potential
to independently report to Congress. Moreover, nearly all the
operational test agency action officers we interviewed participate in
the integrated product teams with the DOT&E action officers and
recognized the value of the Office's independent oversight role. The
action officers within the service testing organizations also have a
degree of independence that enables them to represent the future
users of systems developed in the acquisition community. These
action officers stated that their ability to voice positions
unpopular with the acquisition community was strengthened when DOT&E
separately supported their views.
In discussions with over three dozen action officers and analysts
responsible for the 13 cases we reviewed, the independence of DOT&E
emerged as the fundamental condition to enable effective and
efficient oversight. The foundation of interagency (i.e., DOT&E and
service operational test agencies) relations is based on the
independence of DOT&E, its legislative mandate, and its independent
reporting to Congress. DOT&E is outside the chain of command of
those responsible for developing and testing new systems. The
services need to cooperate with DOT&E primarily because the Office
must approve all test and evaluation master plans and operational
test plans. Moreover, DOT&E independently reports on the operational
suitability and effectiveness at a system's full-rate production
milestone, a report that is sent separately to Congress.
UNFAVORABLE REPORTS ON
OPERATIONAL TESTING DO NOT
ALWAYS INHIBIT FULL-RATE
PRODUCTION
---------------------------------------------------------- Letter :4.2
DOT&E's report on a system's operational suitability and
effectiveness is only one of several inputs considered before the
full-rate production decision is made. An unfavorable DOT&E report
does not necessarily prevent full-rate production. In each of the
cases cited below, an affirmative full-rate production decision was
made despite a DOT&E report concluding that the system had not
demonstrated during operational test and evaluation that it was both
operationally suitable and operationally effective:
-- Full-rate production of the M1A2 tank was approved despite
DOT&E's report that found the system unsuitable.
-- Full-rate production of Joint STARS was approved, though the
system demonstrated only limited effectiveness for "operations
other than war" and found "as tested is unsuitable." Only 18 of
the 71 performance criteria were met; 53 others required more
testing.
-- Full-rate production of the C-17 Airlifter was approved despite
a number of operational test and evaluation deficiencies,
including immature software and failure to meet combination and
brigade airdrop objectives.
SERVICES CONTEST DOT&E
OVERSIGHT OF FOLLOW-ON
OPERATIONAL TEST AND
EVALUATION
---------------------------------------------------------- Letter :4.3
The services contend that DOT&E does not have authority to insist on,
or independently approve the conduct of, follow-on operational test
and evaluation. However, in several of the systems we reviewed,
DOT&E overcame service opposition and monitored follow-on operational
test and evaluation. It used several means to achieve success, such
as (1) incorporating follow-on operational test and evaluation in
test and evaluation master plans developed and approved prior to the
full-rate production decision milestone; (2) persuading the Secretary
of Defense to specify follow-on operational test and evaluation, and
DOT&E's oversight role, in the full-rate production acquisition
decision memorandum; and (3) citing policy, based on title 10, that
entitles DOT&E to oversee operational test and evaluation whenever it
occurs in the acquisition process.\8
Nonetheless, DOT&E action officers stated that the service's
acceptance of DOT&E's role in follow-on operational test and
evaluation varies over time, by service and acquisition system, and
is largely dependent upon the convictions of executives in both the
services and DOT&E. Among the cases reviewed in this report, the
services offered a variety of arguments against DOT&E's having a role
in follow-on operational test and evaluation. They specifically
asserted the following:
-- DOT&E need not be involved because the scope of follow-on
operational test and evaluation is frequently less encompassing
than initial operational test and evaluation. Follow-on
operational test and evaluation has been characterized as
testing by the user to determine the strengths and weaknesses of
the system and to determine ways to compensate for, or fix,
shortcomings observed in initial operational test and
evaluation.\9
-- Title 10 provides DOT&E with the authority to monitor and
review--but not necessarily approve--service follow-on
operational test and evaluation plans.\10
-- Follow-on operational test and evaluation is unnecessary when a
system is found to be operationally effective and suitable
during initial operational test and evaluation--even though
DOT&E does not concur.\11
A clear distinction between DOT&E oversight in follow-on operational
test and evaluation versus initial operational test and evaluation is
that DOT&E is not required to report follow-on operational test and
evaluation results to Congress in the detailed manner of the
Beyond-LRIP report. Therefore, even if follow-on operational test
and evaluation is conducted to assess modifications to correct
effectiveness or suitability shortcomings reported to Congress in the
Beyond-LRIP report, there is no requirement that Congress receive a
detailed accounting of the impact of these modifications.
--------------------
\8 In March 1997 DOT&E issued the "Policy on DOT&E Oversight of
Systems in Follow-on Operational Test and Evaluation (FOT&E)." The
Director stated that 10 U.S.C. section 139 provides DOT&E with the
authority to oversee follow-on operational test and evaluation.
Specifically, DOT&E shall oversee follow-on operational test and
evaluation to (1) refine estimates made during operational test and
evaluation, (2) complete initial operational test and evaluation
activity, (3) verify correction of deficiencies, (4) evaluate
significant changes to design or employment, and (5) evaluate the
system to ensure that it continues to meet operational needs and
retains effectiveness in a substantially new environment or against a
new threat. The Director elaborated by specifying that normal DOD
5000.2R documental and approval requirements apply.
\9 In the case of Joint STARS, the acquisition decision memorandum
required the Air Force and the Army to update a test and evaluation
master plan for OSD approval--but did not require DOT&E approval.
Moreover, the Director of Air Force Test and Evaluation termed
post-milestone III testing as "regression testing" and emphasized
that DOT&E had no oversight role.
\10 In two case study systems, the C-17 and Joint STARS, the Air
Force provided DOT&E with a copy of its follow-on operational test
and evaluation test plans for review but did not allow sufficient
time and had no expectation that DOT&E would approve the plans prior
to the initiation of testing.
\11 The acquisition decision memorandum for the M1A2 tank required
the Army to conduct follow-on operational test and evaluation (with
DOT&E oversight) on safety and suitability shortcomings identified by
DOT&E in initial operational test and evaluation, though the Army had
already determined that the system was operationally suitable as
tested.
DOT&E'S RESOURCES ARE
DECLINING
---------------------------------------------------------- Letter :4.4
DOT&E's primary asset to conduct oversight--its cadre of action
officers--has decreased in size throughout the decade. This creates
a management challenge for the Office because at the same time staff
has decreased, the number of programs overseen by DOT&E has
increased. As illustrated in table 7, authorized staffing declined
from 48 in fiscal
year 1990 to 41 in fiscal year 1997, as did funding (in constant
dollars) from $12,725,000 in fiscal year 1990 to $11,437,000 in
fiscal year 1997. The decline in DOT&E funding is consistent with
the general decline in DOD appropriations during this period.
However, since fiscal year 1990, while the authorized staffing to
oversee operational test and evaluation has declined by 14.6 percent,
the number of systems on the oversight list has increased by 17.7
percent.
Table 7
DOT&E Staffing and Funding
(Dollars in thousands)
Fiscal year
--------------------------------------------------------------
1990 1991 1992 1993 1994 1995 1996 1997
---------------- ------ ------ ------ ------ ------ ------ ------ ------
Funding\a $12,72 $13,55 $12,83 $12,33 $11,45 $12,50 $12,18 $11,43
5 0 6 3 0 1 3 7
Authorized 48 46 44 44 43 43\b 42 41
staffing
Oversight 186 207 204 191 199 202 219 219
programs
--------------------------------------------------------------------------------
\a Funding for operational test and evaluation program element only;
funding provided for the live fire test and evaluation program
element assumed by DOT&E beginning in fiscal year 1995 is not
reflected in the funding data for fiscal years 1995-97.
\b The authorized end strength for DOT&E beginning in fiscal year
1995 increased by four a result of the congressionally directed
(Federal Acquisition Streamlining Act of 1994, P.L. 103-355) move of
live fire test and evaluation responsibilities to DOT&E. Since these
positions are dedicated to live fire testing and not operational
testing, their numbers are not reflected in this table.
DOT&E'S LIMITED RESOURCES
MUST ADDRESS COMPETING
PRIORITIES
---------------------------------------------------------- Letter :4.5
With declining resources, DOT&E must manage competing priorities
related to its oversight, advisory, and coordination
responsibilities. DOT&E must balance the continuing need to allocate
resources to these different priorities while not being perceived as
having lost any independence. DOT&E management has flexibility in
defining some portion of the scope of its oversight and has continued
to electively oversee a substantial number of nonmajor defense
acquisition programs and assumed a leading role in advocating an
examination of the modernization needs of the test and evaluation
infrastructure.
DOT&E CONTINUES TO
OVERSEE A SUBSTANTIAL
NUMBER OF NONMAJOR
PROGRAMS
-------------------------------------------------------- Letter :4.5.1
Between fiscal year 1990 and 1996, the number of nonmajor acquisition
programs overseen annually by DOT&E ranged between 19 and 43. In
fiscal year 1996, when the oversight list reached a peak of 219, 1 of
every 8 programs was listed at the discretion of DOT&E. Thus, during
this period when the resources to oversee operational testing
declined and acquisition reforms have placed additional burdens on
oversight staff, the directors of DOT&E continued to place extra
responsibility on their staff by augmenting the required oversight of
major acquisition programs with a substantial number of optional
systems.
Despite a relative decline in resources for oversight, DOT&E
management has also elected to assume "a larger role in test resource
management planning and leadership in an attempt to achieve
much-needed resource modernization."\12 Although the Director is
designated as the principal adviser to the Secretary of Defense and
the Under Secretary of Defense for Acquisition and Technology on
operational test and evaluation, including operational test
facilities and equipment,\13 assuming the larger role defined by
DOT&E may be at the expense of its testing oversight mission and
perception of independence. The DOT&E Director is now an adviser to
the Central Test and Evaluation Investment Program and previously
served as Chairman of the Test and Evaluation Committee. The
Committee is responsible for the investment program and presides over
the planning, programming, and budgeting for development and
operational test resources. When the Director served as chairman, we
questioned whether these ties created the perception that the
Director was not independent from developmental testing.\14 This
issue may resurface as DOT&E seeks a larger role in test resource
management planning. Also, as the emphasis, cost, and time for
operational test and evaluation are increasingly questioned in the
drive to streamline acquisition, and as oversight assets are
stretched, new DOT&E initiatives may stress the Office's capacity to
manage oversight effectively.
--------------------
\12 Director, Operational Test and Evaluation, FY'95 Report, March
1996, p. I-6.
\13 10 U.S.C. section 139 assigns six responsibilities to the
Director, the fifth of which is to "review and make recommendations
to the Secretary of Defense on all budgetary and financial matters
relating to operational test and evaluation, including operational
test facilities and equipment {emphasis added}, in the Department of
Defense.
\14 In Test and Evaluation: The Director, Operational Test and
Evaluation's Role in Test Resources (GAO/NSIAD-90-128, Aug. 27,
1990), we found that the Director's independence was jeopardized
because the Director had influence over the types of development test
assets used by the services. Responsibility for developmental test
resources rests with the services. In 1987 Congress amended DOT&E's
statute to emphasize the separation of operational testing from
functions associated with developmental testing by stating that "the
Director may not be assigned any responsibility for developmental
test and evaluation, other than the provision of advice to officials
responsible for such testing."
DOT&E PARTICIPATION IN
WORKING-LEVEL INTEGRATED
PRODUCT TEAMS HAS THE
POTENTIAL TO COMPLICATE
INDEPENDENCE
-------------------------------------------------------- Letter :4.5.2
In May 1995, the Secretary of Defense directed DOD to apply the
integrated product and process development concept--using integrated
product teams--throughout the acquisition process. The revised DOD
acquisition regulations (DOD 5000.2-R March 1996) also addressed the
use of empowered integrated product teams at the program office
level. DOT&E action officers participate as members of the
working-level integrated product teams, and the DOT&E Director is a
member of the overarching team. One objective of integrated product
teams, and DOT&E participation in particular, is to expedite the
approval process of test documents by reaching agreement on the
strategy and plan through the identification and resolution of issues
early, understanding the issues, and documenting a quality test and
evaluation master plan that is acceptable to all organizational
levels the first time. Integrated product teams are designed to
replace a previously sequential test and evaluation master plan
development and approval process and therefore enhance timeliness.
While this management tool could increase communication between
testers and the program managers, it also poses a challenge to DOT&E
independence. The challenge was recognized by the Department of
Defense Inspector General (DODIG) when after reviewing the conduct
of operational testing it subsequently recommended that "to meet the
intent of 10 U.S.C. 139, DOT&E should be a nonvoting member [of the
working-level integrated product team] so as to maintain his
independence."\15 {emphasis added} Though integrated product teams
were not used throughout the entire time period covered by this
report, several action officers noted that this management tool
created threats to their effectiveness other than having their
positions out-voted. One DOT&E action officer reported having the
lone dissenting opinion in a meeting of 30 participants seeking to
reach consensus and resolve issues early. The pressure of
maintaining independent, contrary positions in large working groups
can be a test. Several DOT&E representatives also noted that the
frequency of integrated product team meetings to cover the multiple
systems for which they were responsible made it impossible for them
to attend all, thereby lessening the possibility that testing issues
can be identified and resolved as early as possible.
Moreover, program managers and DOT&E pursue different objectives
through integrated product teams. The services and program managers
view the teams as a way to facilitate their program objectives for
cost, schedule, and performance; DOT&E's objective is oversight of
performance through operational testing. The program managers and
DOT&E share a desire to identify testing issues as early as possible.
However, the priority of the program manager to resolve these issues
as early as possible through the teams may conflict with DOT&E's
mission. DOT&E must remain flexible and react to unknowns as they
are disclosed during developmental testing, operational assessments,
and initial operational test and evaluation. Thus, DOT&E's
participation on the teams is a natural source of tension and a
potential impediment to the team's decision-making. The challenge
for DOT&E action officers is to maintain an independent and
potentially contrary position in an ongoing working group during the
life of a program, which may extend over several years.
--------------------
\15 See Department of Defense Office of the Inspector General,
Operational Testing Performed on Weapons Systems, Report No. 96-107,
May 6, 1996, p. 11.
INCREASED INTEGRATION OF
DEVELOPMENTAL AND
OPERATIONAL TESTING MAY
ATTENUATE INDEPENDENT
OVERSIGHT
---------------------------------------------------------- Letter :4.6
The objectives of developmental and operational testing are distinct.
Developmental testing determines whether a system meets its
functional requirements and contractual technical performance
criteria sufficiently to proceed with operational testing.
Operational testing determines whether the system meets the
operational requirements and will contribute to mission effectiveness
in relevant operational environments sufficiently to justify
proceeding with production. The integration of these two disparate
test activities is proposed to save the time and resources required
for testing and evaluation. The sentiment to more closely link
developmental and operational testing dates from at least the 1986
Blue Ribbon Commission on Defense Management (Packard Commission),
which found that "developmental and operational testing have been too
divorced, the latter has been undertaken too late in the cycle, and
prototypes have been used and tested far too little."\16 However,
both we and the DODIG have found that systems were regularly tested
before they were ready for testing. In its 1996 report, the DODIG
reported that "4 of 15 systems we examined for operational testing
were not ready for testing. This situation occurred because a
calendar schedule rather than system readiness often drove the start
of testing."\17 Similarly, we have observed numerous systems that
have been pushed into low-rate initial production without sufficient
testing to demonstrate that the system will work as promised or
intended. Our reviews of major system development in recent years
have found that because insufficient time was dedicated to initial
testing, systems were produced that later experienced problems during
operational testing and systems entered initial production despite
experiencing problems during early operational testing.\18
In 1996 the Secretary of Defense also urged the closer integration of
developmental and operational testing, and combined tests where
possible, in part to enhance the objectives of acquisition reform.
Combined developmental and operational testing is only one of many
sources of test data that DOT&E has used to foster more timely and
thorough operational test and evaluation. Other sources of
information include contractor developmental testing, builder's
trials, component testing, production lot testing, stockpile
reliability testing, and operational deployments. While DOT&E has
some influence over the quality of operational testing, by
independently reviewing the design, execution, analysis, and
reporting of such tests, it has no direct involvement or oversight of
these other sources of testing information. The use of alternative
sources of test data as substitutes for operational test and
evaluation will limit DOT&E's oversight mission, which was created to
improve the conduct and quality of testing.
--------------------
\16 A Quest for Excellence: Final Report by the President's Blue
Ribbon Commission on Defense Management, June 1986, p. xxiii.
\17 Office of the Inspector General, Department of Defense,
Operational Testing Performed on Weapons Systems, Report No. 96-107,
May 6, 1996, p. 16.
\18 See Weapons Acquisition: Low-Rate Initial Production Used to Buy
Weapon Systems Prematurely (GAO/NSIAD-95-18, Nov. 21, 1994).
CONCLUSIONS AND RECOMMENDATIONS
------------------------------------------------------------ Letter :5
DOT&E's challenge is to manage an expansion in independent oversight
while satisfying the efficiency goals of acquisition reform and
undergoing the economic pressures of downsizing. DOT&E oversight is
clearly affecting the operational testing of new defense systems.
DOT&E actions (such as the insistence on additional testing, more
realistic testing, more rigorous data analysis, and independent
assessments) are resulting in more assurance that new systems fielded
to our armed forces are safe, suitable, and effective. However,
DOT&E is not, by design or practice, the guarantor of effective and
suitable acquisitions. DOT&E oversight reduces, but does not
eliminate, the risk that new systems will not be operationally
effective and suitable. Affirmative full-rate production decisions
are made for systems that have yet to demonstrate their operational
effectiveness or suitability. Moreover, the services question
DOT&E's authority regarding follow-on test and evaluation of
subsequent corrective actions by the program office.
We recommend that the Secretary of Defense revise DOD's operational
test and evaluation policies in the following ways:
-- Require the Under Secretary of Defense for Acquisition and
Technology, in those cases where affirmative full-rate
production decisions are made for major systems that have yet to
demonstrate their operational effectiveness or suitability, to
(1) take corrective actions to eliminate deficiencies in
effectiveness or suitability and (2) conduct follow-on test and
evaluation of corrective actions until the systems are
determined to be operationally effective and suitable by the
Director, Operational Test and Evaluation.
-- Require the Director, Operational Test and Evaluation, to (1)
review and approve follow-on test and evaluation master plans
and specific operational test plans for major systems before
operational testing related to suitability and effectiveness
issues left unresolved at the full-rate production decision and
(2) upon the completion of follow-on operational test and
evaluation, report to Congress, the Secretary of Defense, and
the Under Secretary of Defense for Acquisition and Technology
whether the testing was adequate and whether the results
confirmed the system is operationally suitable and effective.
Further, in light of increasing operational testing oversight
commitments and to accommodate oversight of follow-on operational
testing and evaluation, we recommend that the Director, Operational
Test and Evaluation, prioritize his Office's workload to ensure
sufficient attention is given to major defense acquisition programs.
AGENCY COMMENTS AND OUR
EVALUATION
------------------------------------------------------------ Letter :6
In commenting on a draft of this report, DOD concurred with our first
and third recommendations and partially concurred with our second
recommendation. Concerning the recommendation with which it
partially concurred, DOD stated that system specific reports to the
Secretary of Defense and Congress are not warranted for every system
that requires follow-on operational test and evaluation. DOD pointed
out that for specific programs designated for follow-on oversight,
test plans are prepared to correct previously identified deficiencies
by milestone III, and DOT&E includes the results of follow-on testing
in its next annual report.
We continue to believe our recommendation has merit. We recommended
that the Secretary require DOT&E approval of follow-on test and
evaluation of corrective actions because during our review we found
no consensus within the defense acquisition community concerning
DOT&E's role in follow-on operational test and evaluation. In its
comments DOD did not indicate whether it intended to give DOT&E a
role in follow-on operational test and evaluation that is comparable
to its role in initial operational test and evaluation. Moreover, we
continue to believe that if a major system goes into full-rate
production (even though it was deemed by DOT&E not to be
operationally suitable and effective) based on the premise that
corrections will be made and some follow-on operational test and
evaluation will be performed, DOT&E should report, as promptly as
possible, whether or not the follow-on operational test and
evaluation results show that the system in question had improved
sufficiently to be characterized as both operationally suitable and
effective.
DOD's comments are reprinted in their entirety in appendix IV, along
with our specific evaluation.
As agreed with your offices, unless you publicly announce its
contents earlier, we plan no further distribution of this report
until 15 days after its date of issue. We will then send copies to
other congressional committees and the Secretary of Defense. We will
also make copies available to others upon request.
If you have any questions or would like additional information,
please do not hesitate to call me at (202) 512-3092 or the
Evaluator-in-Charge, Jeff Harris, at (202) 512-3583.
Kwai-Cheung Chan
Director of Special Studies and Evaluation
SCOPE AND METHODOLOGY
=========================================================== Appendix I
To develop information for this report, we selected a case study
methodology--evaluating the conduct and practices of DOT&E through an
analysis of 13 weapon systems. Recognizing that many test and
evaluation issues are unique to individual systems, we determined
that a case study methodology would offer the greatest probability of
illuminating the variety of factors that impact the value or
effectiveness of oversight at the level of the Office of the
Secretary of Defense (OSD). Moreover, with nearly 200 systems
subject to review of the Director, Operational Test and Evaluation
(DOT&E) at any one time, we sought a sample that would enable us to
determine if the Office had any impact as well as the ability to
examine the variety of programs overseen. Therefore, we selected a
judgmental sample of cases reflecting the breadth of program types.
As illustrated in table I.1,\1 we selected systems (1) from each of
the primary services, (2) categorized as major defense systems, and
(3) representing a wide array of acquisition and testing phases--from
early operational assessments through and beyond the full-rate
production decision. We studied both new and modified systems.
Table I.1
Characteristics of Weapon Systems Used
for Case Studies
Estimated or
actual New or
year of selected modification
Acquisition development of existing
System Service(s) category\a phase system
---------------- ---------- ---------------- ---------------- --------------
AH-64D Longbow Army 1D MS III (1995); Modification
Apache IOT&E (1995)
helicopter
Airborne Self- Navy 1D FOT&E (1995- New
Protection 96);
Jammer Bosnia (1995)
C-17A Airlifter Air Force 1D FOT&E (1996- New
98);
MS IIIB (1995)
E-3 AWACS Radar Air Force 1C AFSARC III Modification
System (1997);
Improvement IOT&E (1995-96)
Program
F-22 fighter Air Force 1D MS III (2003); New
aircraft IOT&E (2002);
LRIP (1999)
Javelin missile Army 1D MS III (1997); New
LUT (1996);
UE (1996)
Joint
Surveillance
Target Attack
Radar System
E-8 aircraft Air Force 1D FOT&E (1997); New
MS III (1996);
Bosnia (1996)
Common ground Army 1D MS III (1998); New
station IOT&E (1997-
98);
Bosnia (1995)
LPD-17 Navy 1D MS II (1996); New
Amphibious EOA-2 (1996);
Assault Ship EOA-1 (1994-95)
M1A2 tank Army 1D FOT&E (1995- Modification
96);
MS III (1994)
Sensor Fuzed Air Force 1D FOT&E-1 (1997- New
Weapon 98);
MS III (1996);
IOT&E-2 (1995-
96)
Standard Missile
SM-2
Block IIIB Navy II FOT&E (1997); Modification
version MS III (1996);
OPEVAL (1996)
Block IV version Navy 1D MS III (1997); Modification
DT/IOT&E (1994)
Tomahawk Weapon
System
Baseline III Navy 1C MS III (1998); Modification
OPEVAL (1998);
IOT&E (1997)
Baseline IV Navy 1C MS III (2000); Modification
OPEVAL (1999-
00);
IOT&E (1999)
V-22 Navy 1D OPEVAL (1999); New
OT-IIC (1996)
--------------------------------------------------------------------------------
Legend
AFSARC = Air Force Systems Acquisition Review Council
DT = developmental testing
EOA = early operational assessment
FOT&E = follow-on operational test and evaluation
IOT&E = initial operational test and evaluation
LRIP = low-rate initial production
LUT = limited user test
MS = milestone
OA = operational assessment
OPEVAL = operational evaluation
OT = operational testing
UE = user evaluation
\a The Under Secretary of Defense for Acquisition and Technology (USD
(A&T)) designates major defense acquisition programs as either
acquisition category 1D or 1C. The milestone decision authority for
category 1D programs is USD (A&T). The milestone decision authority
for category 1C programs is the Department of Defense (DOD) component
head or, if delegated, the DOD component acquisition executive.
Category I programs are major defense acquisition programs estimated
to require more than $355 million (fiscal year 1996 constant dollars)
for expenditures in research, development, test, and evaluation, or
more than $2.135 billion (fiscal year 1996 constant dollars) for
procurement. Category II programs are those that do not meet the
criteria for category I but do meet the criteria for a major system.
A major system is estimated to require more than $75 million in
fiscal year 1980 constant dollars (approximately $140 million in
fiscal year 1996 constant dollars) for expenditures in research,
development, test, and evaluation, or more than $300 million in
fiscal year 1980 constant dollars (approximately $645 million in
fiscal year 1996 constant dollars) for procurement.
DOT&E, the service operational test agencies, and the Institute for
Defense Analyses (IDA) personnel agreed that DOT&E was influential in
the testing done on these 13 systems. In several cases, the
participating agencies vehemently differed on the value of DOT&E's
actions; however, whether DOT&E had an impact on testing (be it
perceived as positive or negative) was not in dispute.
In conducting our 13 case studies, we assessed the strengths and
weaknesses of the organizational framework in DOD for operational
testing via test agency representatives, an assessment on the origins
and implementation (exemplified by the 13 cases) of the title 10
amendments creating and empowering DOT&E, and a review of the
literature.
To compile case study data, we interviewed current action officers in
both DOT&E and the appropriate operational test agency and reviewed
documentation provided by the operational test agencies, DOT&E, and
IDA. Using structured questionnaires, we interviewed 12 DOT&E and 27
operational test agency action officers responsible for the 13
selected systems as well as managers and technical support personnel
in each organization. In addition, we interviewed the commanders of
each of the service testing agencies and DOT&E. When possible, we
corroborated information obtained from interviews with documentation,
including test and evaluation master plans, beyond low-rate initial
production reports, defense acquisition executive summary status
reports, defense acquisition memoranda, and interagency
correspondence.
In Washington, D.C., we obtained data from or performed work at the
Office of the Director of Operational Test and Evaluation, OSD;
Deputy Under Secretary of Defense for Acquisition Reform; Directorate
of Navy Test and Evaluation and Technology Requirements, Office of
the Chief of Naval Operations; Test and Evaluation Management Agency,
Director of Army Staff; Air Force Test and Evaluation Directorate;
and the DOD Office of the Inspector General. We also reviewed data
and interviewed officials from the Army Operational Test and
Evaluation Command and the Institute for Defense Analyses,
Alexandria, Virginia; the Navy Commander, Operational Test and
Evaluation Force, Norfolk, Virginia; and the Air Force Operational
Test and Evaluation Command, Kirtland Air Force Base, New Mexico.
The use of a systematic case study framework enabled us to identify
and categorize the types of impacts attributable to DOT&E among the
systems studied. In addition, this framework enabled us to identify
trends among factors that correlate with DOT&E effectiveness.
However, we were unable to generalize to all systems subject to
OSD-level oversight. In light of this limitation, we included only
major (high-cost) systems and systems identified by DOT&E and the
lead operational test agency as having been affected by DOT&E
initiatives. Moreover, while our methodology and data collection
enabled us to qualitatively assess the impact of DOT&E, it was not
sufficiently rigorous either to evaluate the cost-effectiveness of
DOT&E actions or to determine the deterrent effects, if any, the
Office exerts over the acquisition and testing process. Finally, our
methodology did not enable an assessment of whether the additional
testing requested by DOT&E was necessary to provide full-rate
production decisionmakers the essential information on a system's
operational effectiveness and suitability or whether the additional
data was worth the time, expense, and resources necessary to obtain
it.
Our review was performed from June 1996 through March 1997 in
accordance with generally accepted government auditing standards.
--------------------
\1 Table I.1 lists the lead service, program size, and acquisition or
testing phase for each of the case study systems, as well as whether
the program is a development effort or a modification of an existing
system.
DESCRIPTION OF 13 CASE STUDY
SYSTEMS
========================================================== Appendix II
AH-64D LONGBOW APACHE
HELICOPTER
-------------------------------------------------------- Appendix II:1
The AH-64D Longbow Apache is a remanufactured and upgraded version of
the AH-64A Apache helicopter. This Army system is equipped with a
mast-mounted fire control radar, fire-and-forget radio frequency
Hellfire missile, and airframe improvements (i.e., integrated
cockpit, improved engines, and global positioning system navigation).
AIRBORNE SELF-PROTECTION JAMMER
-------------------------------------------------------- Appendix II:2
The Airborne Self-Protection Jammer is a defensive electronic
countermeasures system using reprogrammable deceptive jamming
techniques to protect tactical aircraft from radar-guided weapons.
This Navy system is intended to protect Navy and Marine Corps F-18
and F-14 aircraft.
C-17A AIRLIFTER
-------------------------------------------------------- Appendix II:3
The C-17A Airlifter provides strategic/tactical transport of all
cargo, including outsized cargo, mostly to main operational bases or
to small, austere airfields, if needed. Its four-engine turbofan
design enables the transport of large payloads over intercontinental
ranges without refueling. This Air Force aircraft will replace the
retiring C-141 aircraft and augment the C-130 and C-5 transport
fleets.
E-3 AWACS RADAR SYSTEM
IMPROVEMENT PROGRAM
-------------------------------------------------------- Appendix II:4
The Air Force's E-3 AWACS consists of a Boeing 707 airframe modified
to carry a radome housing a pulse-Doppler radar capable of detecting
aircraft and cruise missiles, particularly at low altitudes. The
Radar System Improvement Program replaces several components of the
radar to improve detection capability and electronic countermeasures
as well as reliability, availability, and maintainability.
F-22 AIR SUPERIORITY FIGHTER
-------------------------------------------------------- Appendix II:5
The F-22 is an air superiority aircraft with a capability to deliver
air-to-ground weapons. The most significant features include
supercruise, the ability to fly efficiently at supersonic speeds
without using fuel-consuming afterburners, low observability to
adversary systems with the goal to locate and shoot down the F-22,
and integrated avionics to significantly improve the pilot's
battlefield awareness.
JAVELIN MISSILE
-------------------------------------------------------- Appendix II:6
The Javelin is a man-portable, antiarmor weapon developed for the
Army and the Marine Corp to replace the aging Dragon system. It is
designed as a fire-and-forget system comprised of a missile and
reusable command launch unit.
JOINT SURVEILLANCE TARGET
ATTACK RADAR SYSTEM
-------------------------------------------------------- Appendix II:7
The Joint Surveillance Target Attack Radar System is designed to
provide intelligence on moving and stationary targets to Air Force
and Army command nodes in near real time. The system comprises a
modified Boeing 707 aircraft frame equipped with radar,
communications equipment, and the air component of the data link,
computer workstations, and self-defense suite as well as ground
station modules mounted on Army vehicles.
LPD-17 AMPHIBIOUS ASSAULT SHIP
-------------------------------------------------------- Appendix II:8
The LPD-17 will be an amphibious assault ship capable of launching
(1) amphibious assault craft from a well deck and (2) helicopters or
vertical takeoff and landing aircraft from an aft flight deck. It is
intended to transport and deploy combat and support elements of
Marine expeditionary brigades as a key component of amphibious task
forces.
M1A2 ABRAMS MAIN BATTLE TANK
-------------------------------------------------------- Appendix II:9
The M1A2 Abrams main battle tank is an upgrade of the M1A1 and is
intended to improve target acquisition and engagement rates and
survivability while sustaining equivalent operational suitability.
Specifically, the modified tank incorporates a commander's
independent thermal viewer, a position navigation system, and an
intervehicle command and control system.
SENSOR FUZED WEAPON
------------------------------------------------------- Appendix II:10
The Sensor Fuzed Weapon is an antiarmor cluster munition to be
employed by fighter, attack, or bomber aircraft to achieve multiple
kills per pass against armored and support combat formations. Each
munition contains a tactical munitions dispenser comprising 10
submunitions containing a total of 40 infrared sensing projectiles.
High-altitude accuracy is to be improved through the incorporation of
a wind-compensated munition dispenser upgrade.
STANDARD MISSILE-2
------------------------------------------------------- Appendix II:11
The Standard Missile-2 is a solid propellant-fueled, tail-controlled,
surface-to-air missile fired by surface ships. It was originally
designed to counter high-speed, high-altitude antiship missiles in an
advanced electronic countermeasures environment. The block IIIA
version provides improved capacity against low-altitude targets with
an improved warhead. The block IIIB adds an infrared seeker to the
block IIIA to enhance the missile's capabilities against specific
threats. These improvements are being made to provide capability
against theater ballistic missiles while retaining its capabilities
against antiair warfare threats.
TOMAHAWK WEAPON SYSTEM
------------------------------------------------------- Appendix II:12
The Tomahawk Weapon System is a long-range subsonic cruise missile
for land and sea targets. The baseline IV upgrade is fitted with a
terminal seeker, video data link, and two-way digital data link. The
primary
baseline IV configuration is the Tomahawk multimission missile; a
second variant is the Tomahawk hard target penetrator.
V-22 OSPREY
------------------------------------------------------- Appendix II:13
The V-22 is a tilt rotor vertical/short takeoff and landing,
multimission aircraft developed to fulfill operational combat
requirements in the Marine Corps and Special Operations Forces.
DOT&E'S AND DOD'S SYSTEM
ACQUISITION PROCESS
========================================================= Appendix III
DOT&E's role in the system acquisition process does not become
prominent until the latter stages. As weapon system programs
progress through successive phases of the acquisition process, they
are subject to major decision points called milestones. The
milestone review process is predicated on the principle that systems
advance to higher acquisition phases by demonstrating that they meet
prescribed technical and performance thresholds. Figure III.1
illustrates DOD's weapon system acquisition process.
Figure III.1: DOD's Weapon
System Acquisition Process
(See figure in printed
edition.)
Per DOD directive, test and evaluation planning begins in phase 0,
Concept Exploration. Operational testers are to be involved early to
ensure that the test program for the most promising alternative can
support the acquisition strategy and to ensure the harmonization of
objectives, thresholds, and measures of effectiveness in the
operational readiness document and the test and evaluation master
plan. Early testing of prototypes in phase I, Program Definition and
Risk Reduction, and early operational assessments are to be
emphasized to assist in identifying risks. A combined developmental
and operational test approach is encouraged to save time and costs.
Initial operational test and evaluation is to occur during phase II
to evaluate operational effectiveness and suitability before the
full-rate production decision, milestone III, on all acquisition
category I and II programs. For all acquisition category I programs
and other programs designated for OSD test and evaluation oversight,
a test and evaluation master plan is prepared and submitted for
approval prior to first milestone review (excluding milestone 0).\1
The master plan is to be updated at milestones when the program has
changed significantly. DOT&E must approve the test and evaluation
master plan and the more specific operational test plans prior to
their execution. This process and the required plan approvals
provide DOT&E opportunities to affect the design and execution of
operational testing throughout the acquisition process.
(See figure in printed edition.)Appendix IV
--------------------
\1 Master plans for acquisition category I programs are to be
submitted to the Director, Test Systems Engineering and Evaluation,
30 days prior to the first milestone. For all other programs
designated for OSD oversight, the plans must be submitted 90 days
prior to the first milestone.
COMMENTS FROM THE DEPARTMENT OF
DEFENSE
========================================================= Appendix III
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
The following are GAO's comments on the September 19, 1997, letter
from the Department of Defense.
GAO COMMENTS
1. In prior reviews of individual weapon systems, we have found that
operational testing and evaluation is generally viewed by the
acquisition community as a costly and time-consuming requirement
imposed by outsiders rather than a management tool for more
successful programs. Efforts to enhance the efficiency of
acquisition, in general--and in operational testing, in
particular--need to be well balanced with the requirement to
realistically and thoroughly test operational suitability and
effectiveness prior to the full-rate production decision. We
attempted to take a broader view of acquisition reform efficiency
initiatives to anticipate how these departures from past ways of
doing business could impact both the quality of operational testing
and the independence of DOT&E.
2. We were asked to assess the impact of DOT&E on the quality and
impact of testing and reported on the Secretary of Defense
initiatives only to the extent they may pose a potential impact on
DOT&E's independence or effectiveness. Moreover, we did not
recommend or suggest that testers wait until milestone III to
discover problems that could have been learned and corrected earlier.
Since its inception, DOT&E has been active in test integration and
planning working groups and test and evaluation master plan
development during the earliest phases of the acquisition process.
In fact, we have long advocated more early testing to demonstrate
positive system performance prior to the low-rate initial production
decision. DOT&E's early involvement in test planning is appropriate,
necessary, and required by DOD regulations. In this report we do not
advocate the elimination of DOT&E participation during the early
stages of the acquisition process; rather, we merely observe that
DOT&E participation through the vehicle of working-level program
manager integrated product teams has the potential to complicate
independence and may be increasingly difficult to implement with
declining resources and increasing oversight responsibilities
following milestone III.
3. We did not recommend or suggest that DOT&E ignore its statutory
responsibility to review and make recommendations to the Secretary of
Defense on budgetary and financial matters related to operational
test facilities and equipment. We only observed that in an era of
declining resources, earlier participation, and extended oversight
responsibilities, a decision to assume a larger role in test resource
management planning and leadership is likely to result in tradeoffs
in other responsibilities--the largest being oversight.
4. We made this recommendation because DOT&E, the services, and the
program offices did not necessarily agree on the degree to which
system performance requirements have been met in initial operational
test and evaluation. Furthermore, there was no consensus within the
acquisition community concerning DOT&E's authority to oversee
follow-on operational test and evaluation conducted to ensure that
proposed corrections to previously identified deficiencies were
thoroughly tested and evaluated.
5. Under 10 U.S.C. 2399, DOT&E is required to independently report
to Congress whether a major acquisition system has proven to be
operationally suitable and effective prior to the full-rate
production decision. When follow-on operational test and evaluation
is necessary to test measures intended to correct deficiencies
identified in initial operational test and evaluation, Congress does
not receive an equivalent independent report from DOT&E that
concludes, based on required follow-on operational test and
evaluation, whether or not a major system has improved sufficiently
to be considered both operationally suitable and effective.
RELATED GAO PRODUCTS
============================================================ Chapter 0
Tactical Intelligence: Joint STARS Full-Rate Production Decision Was
Premature and Risky (GAO/NSIAD-97-68, Apr. 25, 1997).
Weapons Acquisition: Better Use of Limited DOD Acquisition Funding
Would Reduce Costs (GAO/NSIAD-97-23, Feb. 13, 1997).
Airborne Self-Protection Jammer (GAO/NSIAD-97-46R, Jan. 29, 1997).
Army Acquisition: Javelin Is Not Ready for Multiyear Procurement
(GAO/NSIAD-96-199, Sept. 26, 1996).
Tactical Intelligence: Accelerated Joint STARS Ground Station
Acquisition Strategy Is Risky (GAO/NSIAD-96-71, May 23, 1996).
Electronic Warfare (GAO/NSIAD-96-109R, Mar. 1, 1996).
Longbow Apache Helicopter: System Procurement Issues Need to Be
Resolved (GAO/NSIAD-95-159, Aug. 24, 1995).
Electronic Warfare: Most Air Force ALQ-135 Jammers Procured Without
Operational Testing (GAO/NSIAD-95-47, Nov. 22, 1994).
Weapons Acquisition: Low-Rate Initial Production Used to Buy Weapon
Systems Prematurely (GAO/NSIAD-95-18, Nov. 21, 1994).
Acquisition Reform: Role of Test and Evaluation in System
Acquisition Should Not Be Weakened (GAO/T-NSIAD-94-124, Mar. 22,
1994).
Test and Evaluation: The Director, Operational Test and Evaluation's
Role in Test Resources (GAO/NSIAD-90-128, Aug. 27, 1990).
Adequacy of Department of Defense Operational Test and Evaluation
(GAO/T-NSIAD-89-39, June 16, 1989).
Weapons Testing: Quality of DOD Operational Testing and Reporting
(GAO/PEMD-88-32BR, July 26, 1988).
*** End of document. ***
|
NEWSLETTER
|
|
Join the GlobalSecurity.org mailing list
|
|
|
