Defense Acquisition: Advanced Concept Technology Demonstration Program Can Be Improved (Letter Report, 10/15/98, GAO/NSIAD-99-4)
Pursuant to a congressional request, GAO reviewed the current Advanced
Concept Technology Demonstration (ACTD) program, focusing on: (1)
whether the selection process includes criteria that are adequate to
ensure that only mature technologies are selected for ACTD prototypes;
(2) whether guidance on transitioning to the normal acquisition process
ensures that a prototype appropriately completes product and concept
development and testing before entering production; and (3) the
Department of Defense's (DOD) current practice of procuring more ACTD
prototypes than needed to assess the military utility of a mature
technology.
GAO noted that: (1) through the determination of military value of
mature technologies and their use in the acquisition process, ACTDs have
the potential to reduce the time to develop and acquire weapon systems;
(2) however, several aspects of the ACTD program can be improved; (3)
DOD's process for selecting ACTD candidates does not include adequate
criteria for assessing the maturity of the proposed technology and has
resulted in the approval of ACTD projects that included immature
technology; (4) DOD has improved its guidance on the maturity of the
technologies to be used in ACTD projects but the revised guidance
describes several types of exceptions under which immature technologies
may be used; (5) where DOD approves immature technologies as ACTD
program candidates and time is spent conducting developmental
activities, the goal of reduced acquisition cycle time will not be
realized; (6) further, guidance on entering technologies into the normal
acquisition process is not sufficient to ensure that a prototype
completes product and concept development and testing before entering
production; (7) the guidance does not mention the circumstances when
transition to development may be appropriate or the kinds of
developmental activities that may be appropriate; (8) while commercial
items that do not require any further development could proceed directly
to production, many ACTDs may still need to enter the engineering and
manufacturing development phase to proceed with product and concept
development and testing before production begins; (9) through the ACTD
early user demonstration, DOD is expected to obtain more detailed
knowledge about its technologies before entering into the acquisition
process; (10) however, in the one case in which an ACTD has proceeded
into production, DOD made that decision before completing product and
concept development and testing, thereby accepting programmatic risks
that could offset the schedule and other benefits gained through the
ACTD process; (11) DOD's current practice of procuring prototypes beyond
those needed for the basic ACTD demonstration and before completing
product and concept development and testing is unnecessarily risky; and
(12) this practice risks wasting resources on the procurement of items
that may not work as expected or may not have sufficient military
utility and risks a premature and excessive commitment to production.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: NSIAD-99-4
TITLE: Defense Acquisition: Advanced Concept Technology
Demonstration Program Can Be Improved
DATE: 10/15/98
SUBJECT: Weapons systems
Concurrency
Cost effectiveness analysis
Defense cost control
Defense procurement
Procurement practices
Testing
IDENTIFIER: DOD Advanced Concept Technology Demonstration Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** <info@www.gao.gov> **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Chairman, Subcommittee on Military Research and
Development, Committee on National Security, House of Representatives
October 1998
DEFENSE ACQUISITION - ADVANCED
CONCEPT TECHNOLOGY DEMONSTRATION
PROGRAM CAN BE IMPROVED
GAO/NSIAD-99-4
Defense Acquisition
(707299)
Abbreviations
=============================================================== ABBREV
ACTD - Advanced Concept Technology Demonstration
DOD - Department of Defense
Letter
=============================================================== LETTER
B-278429
October 15, 1998
The Honorable Curt Weldon
Chairman, Subcommittee on Military
Research and Development
Committee on National Security
House of Representatives
Dear Mr. Chairman:
The Advanced Concept Technology Demonstration (ACTD) program was
initiated by the Department of Defense (DOD) to determine the extent
to which a given mature technology will improve military capabilities
before entering the normal acquisition process and, by using mature
technology, reduce the length of time to develop and acquire weapon
systems. According to the ACTD program documentation, the ACTD
program was established in 1994 in response to recommendations of the
1986 Packard Commission\1 and a 1991 Defense Science Board study.\2
The Packard Commission concluded that major improvements could be
made in defense acquisition by emulating the practices of successful
commercial companies. Accordingly, the Commission recommended, among
other things, building and testing prototypes to assess military
utility and provide a basis for realistic cost estimates before
committing to acquisition. The Defense Science Board's study
discussed the need for early dialogue between the potential system's
user and producer for a proper analysis of cost, risk, and
operational capability.
ACTDs have been the subject of congressional interest since the
program's inception. Congressional committees have expressed
concerns about the validity of several technologies selected for the
ACTD program and the number of assets procured for the projects. At
your request, we assessed the current ACTD program. Specifically, we
determined whether (1) the selection process includes criteria that
are adequate to ensure that only mature technologies are selected for
ACTD prototypes and (2) guidance on transitioning to the normal
acquisition process ensures that a prototype appropriately completes
product and concept development and testing before entering
production. Finally, we assessed DOD's current practice of procuring
more ACTD prototypes than needed to assess the military utility of a
mature technology.
--------------------
\1 The Packard Commission was created by former President Reagan in
1985 to review the defense acquisition system to determine how
military acquisitions could be made quicker and at lower costs.
\2 Report of the Defense Science Board, 1991 Summer Study on Weapon
Development and Production Technology, November 1991.
BACKGROUND
------------------------------------------------------------ Letter :1
Through fiscal year 1998, about $172 million has been allocated to
the ACTD program and 48 projects have been approved. DOD's budget
request for fiscal year 1999 for the ACTD program is $116.4 million.
An additional 10 to 15 projects are expected to be funded in fiscal
year 1999.
Under the current ACTD program, DOD builds prototypes to assess the
military utility of mature technologies, which are used to reduce or
avoid the time and effort usually devoted to technology development.
Demonstrations that assess a prototype's military utility are
structured to be completed within 2 to 4 years and require the
participation of field users (war fighters).
ACTD projects are not acquisition programs. The ACTD program seeks
to provide the war fighter with the opportunity to assess a
prototype's capability in realistic operational scenarios. From this
demonstration, the war fighter can refine operational requirements,
develop an initial concept of operation, and make a determination of
the military utility of the technology before DOD decides whether the
technology should enter into the normal acquisition process.
Not all projects will be selected for transition into the normal
acquisition process. The user can conclude that the technology (1)
does not have sufficient military utility and that acquisition is not
warranted or (2) has sufficient utility but that additional
procurement is not necessary. Of the 11 ACTD projects completed as
of August 1998, 2 were found to have insufficient utility to proceed
further, 8 were found to have military utility but no further
procurement was found to be needed at the time,\3 and 1 was found to
have utility and has transitioned to the normal acquisition process.
ACTD funding is to be used to procure enough prototypes to conduct
the basic demonstration of military utility. At the conclusion of
the basic demonstration, ACTD projects are expected to provide a
residual operational capability for the war fighter. Under the
current practice, ACTD funding is also to be available to support
continued use of ACTD prototypes that have military utility for a
2-year, post-demonstration period. The
2 years of funding is to support continued use by an operational unit
and provide the time needed to separately budget for the acquisition
of additional systems. Further, if the ACTD prototypes--such as
missiles--will be consumed during the basic demonstration, additional
prototypes are to be procured.
As stated in the ACTD guidance, a key to successfully exploiting the
results of the demonstration is to enter the appropriate phase of
acquisition without loss of momentum. ACTDs are intended to shorten
the acquisition cycle by reducing or eliminating technology
development and maturation activities during the normal acquisition
process. Further, DOD can concentrate more on technology integration
and demonstration activities. Time and effort usually devoted to
technology development can be significantly reduced or avoided and
the subsequent acquisition process reduced accordingly, if the
project is deemed to have sufficient military utility.
ACTD candidates are nominated from a variety of sources within the
defense community, including the Commanders in Chief, the Joint
Chiefs of Staff, the Office of the Secretary of Defense agencies, the
services, and the research and development laboratories. The
candidates are then reviewed and assessed by staff from the Office of
the Deputy Under Secretary of Defense (Advanced Technology). After
this initial screening, the remaining candidates are further assessed
by a panel of technology experts. The best candidates are then
submitted to the Joint Requirements Oversight Council, which assesses
their priority. The final determination of the candidates to be
funded is made within the Office of the Deputy Under Secretary of
Defense (Advanced Technology), with final approval by the Under
Secretary of Defense (Acquisition and Technology).
--------------------
\3 Three of the projects in this category are primarily software, for
which production would not be appropriate.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :2
Through the determination of military value of mature technologies
and their use in the acquisition process, ACTDs have the potential to
reduce the time to develop and acquire weapon systems. However,
several aspects of the ACTD program can be improved.
DOD's process for selecting ACTD candidates does not include adequate
criteria for assessing the maturity of the proposed technology and
has resulted in the approval of ACTD projects that included immature
technology. DOD has improved its guidance on the maturity of the
technologies to be used in ACTD projects but the revised guidance
describes several types of exceptions under which immature
technologies may be used. Where DOD approves immature technologies
as ACTD program candidates and time is spent conducting developmental
activities, the goal of reduced acquisition cycle time will not be
realized.
Further, guidance on entering technologies into the normal
acquisition process is not sufficient to ensure that a prototype
completes product and concept development and testing before entering
production. According to the ACTD guidance, if the prototype is
found to have military utility, ACTD can directly enter production.
The guidance does not mention the circumstances when transition to
development may be appropriate or the kinds of developmental
activities that may be appropriate. While commercial items that do
not require any further development could proceed directly to
production, many ACTDs may still need to enter the engineering and
manufacturing development phase to proceed with product and concept
development and testing before production begins. Through the ACTD
early user demonstration, DOD is expected to obtain more detailed
knowledge about its technologies before entering into the acquisition
process. However, in the one case in which an ACTD has proceeded
into production, DOD made that decision before completing product and
concept development and testing, thereby accepting programmatic risks
that could offset the schedule and other benefits gained through the
ACTD process.
DOD's current practice of procuring prototypes beyond those needed
for the basic ACTD demonstration and before completing product and
concept development and testing is unnecessarily risky. This
practice risks wasting resources on the procurement of items that may
not work as expected or may not have sufficient military utility--as
would be determined in the basic demonstration--and risks a premature
and excessive commitment to production.
INADEQUATE GUIDANCE FOR
ASSESSING MATURITY OF
TECHNOLOGY
------------------------------------------------------------ Letter :3
By limiting consideration to prototypes that feature mature
technology, the ACTD program avoids the time and risks associated
with technology development, concentrating instead on technology
integration and demonstration activities. The information gained
through the demonstration of the mature technology could provide a
good jump start to the normal acquisition process, if the
demonstration shows that the technology has sufficient military
value. Time and effort usually devoted to technology development
could be reduced or avoided and the acquisition process shortened
accordingly.
Program officials stated that they have a mechanism in place to
ensure that only those projects using mature technology are allowed
to become ACTDs. These officials explained that an ACTD candidate's
technology is assessed by high-ranking representatives from the
services and the DOD science and technology community before
candidates are selected. Program personnel stated that determining
technology maturity is important before a candidate is selected
because ACTD program funding is not intended to be used for
technology development. According to program guidance, the ACTD
funding is to be used for (1) costs incurred when existing technology
programs are reoriented to support ACTD, (2) costs to procure
additional assets for the basic ACTD demonstration, and (3) costs for
technical support for 2 years of field operations following the basic
ACTD demonstration. We were told that no ACTD money was to be used
for technology development activities.
However, the project selection process does not ensure that only
mature technologies enter the ACTD program. We found examples where
immature technologies were selected and technology development was
taking place after the approval and start of the ACTD program. The
current operations manager of the Combat Identification project,
which began in fiscal year 1996, told us that one of his major
concerns has been that some of the ACTD funding was being used for
technology development, and not exclusively used for designing and
implementing the assessment. However, during the ACTD project,
technical or laboratory testing was still necessary to evaluate the
acceptability of many of the 12 technologies included in the initial
project. Eventually, 6 of the 12 technologies had to be terminated.
According to the demonstration manager, 2 of the 6 technologies were
terminated because they were immature. According to the manager,
that is one of the reasons the project is currently behind schedule.
Another example of the inclusion of immature technology occurred in
the Outrider Unmanned Aerial Vehicle project. According to the
management plan for the project, one of the individual technologies
to be incorporated into the vehicle was a heavy fuel engine.
According to a program official, it was later deemed that this
individual technology was too immature and an alternate technology
had to be used. However, trying to use this immature technology has
already caused schedule slippage and cost overruns in the ACTD
project.
INSUFFICIENT GUIDANCE FOR
TRANSITIONING ACTDS TO THE
NORMAL ACQUISITION PROCESS
------------------------------------------------------------ Letter :4
To complete the basic demonstration within the prescribed 2 to 4 year
period, ACTDs typically use early prototypes. If the demonstrated
technology is deemed to have sufficient military utility, many ACTD
projects will still need to enter the normal acquisition process to
complete product and concept development and testing to determine,
for example, whether the system is producible and can meet the user's
suitability needs.\4 These attributes of a system go beyond the
ACTD's demonstration of military utility to address whether the item
can meet the full military requirement. Commercial items that do not
require any further development could proceed directly to production.
However, other non-software related ACTDs should enter the
engineering and manufacturing development phase to proceed with
product and concept development and testing.
According to ACTD guidance, if further significant development is
needed, a system might enter the development portion of the
engineering and manufacturing development phase. However, the
guidance states that, if the capability is adequate, the ACTD can
directly enter production. The guidance does not specifically define
what is considered an "adequate capability" to allow an ACTD system
to enter low-rate production.
In 1994, we reported on numerous instances of weapon systems that
began production prematurely and later experienced significant
operational effectiveness or suitability problems.\5 In our best
practices report, we reported that typically DOD programs allowed
much more technology development to continue into the product
development phase than is the case in commercial practices.\6
Turbulence in program outcomes--in the form of production problems
and associated cost and schedule increases--was the predictable
consequence of DOD's actions. In contrast, commercial firms gained
more knowledge about a product's technology, performance, and
producibility much earlier in the product development process.
Commercial firms consider not having this type of knowledge early in
the acquisition process an unacceptable risk. In responding to that
report, the Secretary of Defense stated that DOD is vigorously
pursuing the adoption of such business practices. Specifically, he
stated that DOD has taken steps to separate technology development
from product development through the use of ACTDs. The ACTD guidance
and DOD's current practice do not appear to reflect this emphasis.
In the case of the Predator ACTD, the one ACTD that has proceeded
into production, DOD decided to enter the technology into production
before proceeding with product and concept development and testing,
thereby accepting programmatic risks that could offset the schedule
and other benefits gained through the ACTD process. In the early
operational assessment of the Predator's ACTD demonstration, the
Director, Operational Test and Evaluation, did not make a
determination of the system's potential operational effectiveness or
suitability. However, the system was found to be deficient in
several areas, including mission reliability, documentation, and
pilot training. The assessment also noted that the ACTD
demonstration was not designed to evaluate several other areas such
as system survivability, supportability, target location accuracy,
training, and staffing requirements.
The basic ACTD demonstration may have clarified the Predator's
military utility but it did not demonstrate its system requirements
or its suitability. Thus, instead of using the knowledge acquired
during the demonstration to complete the Predator's development
through the product and concept development and testing stages of
acquisition, DOD allowed it to directly enter production.
--------------------
\4 "Suitability" involves factors such as maintainability,
reliability, safety, and supportability.
\5 Weapons Acquisition: Low-Rate Initial Production Used to Buy
Weapon Systems Prematurely (GAO/NSIAD-95-18, Nov. 21, 1994).
\6 Best Practices: Successful Application to Weapon Acquisitions
Requires Changes in DOD's Environment (GAO/NSIAD-98-56, Feb. 24,
1998).
PROCURING ACTD PROTOTYPES
BEYOND THOSE NEEDED FOR BASIC
DEMONSTRATION IS UNNECESSARILY
RISKY
------------------------------------------------------------ Letter :5
DOD's practice is to procure sufficient ACTD prototypes to provide a
2-year residual capability. When it determines that the original
prototypes will be consumed during the basic demonstration,
additional prototypes are procured for potential use after the basic
ACTD demonstration. However, these additional assets--like the basic
demonstration prototypes--have not been independently tested to
determine their effectiveness and suitability. Procuring additional
ACTD prototypes before product and concept development and testing is
completed risks wasting resources on the procurement of items that
may not work as expected or may not have sufficient military utility.
Representatives from the service test agencies did not support this
practice and agreed that it had the potential for problems. Without
a meaningful independent assessment of a product's suitability,
effectiveness, and survivability, users cannot be assured that it
will operate as intended and is supportable.
Congress has expressed concern about the amount of equipment being
procured beyond what is needed to conduct the basic ACTD
demonstration. Its concern is that DOD is making an excessive
commitment to production before military utility is demonstrated and
before appropriate concepts of operation are developed. For example,
DOD plans to procure 192 Enhanced Fiber Optic Guided missiles at an
estimated cost of $27 million and 144 Line-of-Sight Anti-Tank
missiles at an estimated cost of $28 million beyond the quantities of
missiles required for the ACTD demonstrations--64 and 30 missiles,
respectively. The production of these additional missiles will
follow the production of the missiles needed for the basic
demonstration and will continue on a regular basis throughout the
2-year, post-demonstration period. If the prototypes are deemed to
have sufficient military utility, the service involved will be
expected to fund the production of additional missiles beyond these
quantities. By establishing a regular pattern of procurement in this
way, DOD risks committing to a continuing production program before a
determination is made about the technology's military utility and
before there is assurance that the system will meet validated
requirements and be supportable.
CONCLUSIONS
------------------------------------------------------------ Letter :6
The strength of the ACTD program is in conducting basic
demonstrations of mature technology in military applications before
entering the normal acquisition process. This practice could
significantly reduce or eliminate the time and effort needed for
technology development from the acquisition process. For this to
occur, it is essential that DOD use only mature technology in its
ACTDs. DOD's criteria for selecting technologies for ACTD candidates
should be clarified to ensure the selection of mature technology with
few, if any, exceptions.
Further, ACTDs may not, by themselves, result in an effective and
safe deployment of military capability. It is important that product
and concept development as well as test and evaluation processes be
allowed to proceed before the service commits to the production of
the demonstrated technology. If an ACTD project is shown to have
military value, the normal acquisition processes can and should be
tailored--but not bypassed--before DOD begins production. Lastly,
emphasizing the need to complete concept and product development and
testing before procuring more items than needed for the basic
demonstration would reduce the risk of prematurely starting
production.
RECOMMENDATIONS
------------------------------------------------------------ Letter :7
We recommend that the Secretary of Defense clarify the ACTD program
guidance to (1) ensure the use of mature technology with few, if any,
exceptions and (2) describe when transition to the development phase
of the acquisition cycle is necessary and the types of development
activity that may be appropriate. Further, we recommend that the
Secretary of Defense limit the number of prototypes to be procured to
the quantities needed for early user demonstrations of mature
technology until the item's product and concept development and
testing have been completed.
AGENCY COMMENTS AND OUR
EVALUATION
------------------------------------------------------------ Letter :8
In commenting on a draft of this report, DOD partially concurred with
each of our recommendations. DOD said that it continues to refine
the ACTD selection and implementation process. DOD agreed that the
ACTD program should focus on mature technologies and stated that it
had improved its definitions of mature technology. DOD's new
guidance on the maturity of technology to be used in ACTDs states
". . . new technologies proposed for incorporation into an
ACTD should not be in the 6.1 (basic research) or 6.2 (applied
research) budget categories. Furthermore, the technologies must
have been successfully demonstrated at the subsystem or
component level and at the required performance level prior to
the start of the ACTD."
While this guidance is improved over previous versions, the new
guidance permits the selection of immature technology--even as the
primary or core technology-- provided that it is demonstrated prior
to the ACTD demonstration. Also, some recent ACTD projects have been
approved without the technologies having been identified. Moreover,
the new guidance goes on to describe several types of exceptions
under which immature technologies may be permitted to be used in an
ACTD. As our report states, the use of immature technologies has
delayed programs and we continue to believe DOD needs to focus the
ACTD program on the use of mature technology with few, if any
exceptions.
DOD also agreed that some but not all ACTDs may require additional
product and concept development before proceeding into production.
DOD states that a mandatory engineering and manufacturing development
phase would not be appropriate for all ACTD projects. We agree,
however, the existing ACTD guidance focuses on the transition
directly to production and provides too little guidance concerning a
possible transition to development. As stated in our recommendation,
the guidance should specify when a transition to development may be
appropriate and the kinds of developmental activities that may be
appropriate.
Finally, DOD agreed that the number of ACTD prototypes to be procured
should be limited until the Under Secretary can confirm that
sufficient testing has been satisfactorily completed to support any
additional procurement. We agree with DOD that test results should
form the basis for starting limited procurement. However, DOD's
equating a determination of military utility (based on an ACTD
demonstration) with a determination of a system's readiness to begin
production is inappropriate because production decisions require more
testing data. We have long held the view and have consistently
recommended that DOD use extreme caution to avoid premature
commitments to production.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :9
To determine the adequacy of the ACTD program's selection criteria in
assessing technology maturity and guidance for transitioning to the
normal acquisition process, we reviewed existing program guidance,
published reports, the Office of the Inspector General's April 1997
ACTD report, and the recommendations of the 1986 Packard commission
and the 1996 Defense Science Board. We discussed selection criteria,
transitioning to the acquisition process, and all 34 of the
individual ACTD programs approved through fiscal year 1997 with
representatives from the Office of the Deputy Under Secretary of
Defense (Advanced Technology), Washington, D.C.; the Army's Deputy
Chief of Staff for Operations and Plans, Office of Science and
Technology Programs, Washington, D.C.; the Air Force's Director for
Operational Requirements, Rosslyn, Virginia; the Navy's Requirements
and Acquisition Support Branch, Washington, D.C.; the Marine Corps'
Combat Development Command Office of Science and Innovation,
Quantico, Virginia; the Joint Staff's Acquisition and Technology
Division and Requirements Assessment Integration Division,
Washington, D.C.; and the Office of the Commander in Chief, U.S.
Atlantic Command, Norfolk, Virginia.
We discussed the issue of procuring additional residual assets for
early deployment with representatives from DOD's Office of the
Director, Operational Test and Evaluation, Washington, D.C.; the
Army's Test and Evaluation Management Agency, Washington, D.C.; the
Army's Operational Test and Evaluation Command, Alexandria, Virginia;
the Marine Corps' Operational Test and Evaluation Activity, Quantico,
Virginia; the Air Force's Test and Evaluation Directorate,
Washington, D.C.; and the Navy's Commander, Operational Test and
Evaluation Force, Norfolk, Virginia.
We conducted our review from September 1997 to July 1998 in
accordance with generally accepted government auditing standards.
---------------------------------------------------------- Letter :9.1
As agreed with your office, unless you publicly announce the contents
of this report earlier, we plan no further distribution of this
report until
30 days from its issue date. At that time, we will send copies to
other interested congressional committees; the Secretaries of
Defense, the Army, the Air Force, and the Navy; the Commandant of the
Marine Corps; the Director, Office of Management and Budget; and
other interested parties. We will also make copies available to
others upon request.
Please contact me at (202) 512-4841, if you or your staff have any
questions concerning this report. The major contributors to this
report were
Bill Graveline, Laura Durland, and John Randall.
Sincerely yours,
Louis J. Rodrigues
Director, Defense Acquisitions Issues
(See figure in printed edition.)APPENDIX I
COMMENTS FROM THE DEPARTMENT OF
DEFENSE
============================================================== Letter
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
The following are GAO's comments on the Department of Defense's (DOD)
letter, dated August 31, 1998.
GAO COMMENTS
1. DOD's guidance is improved over previous versions. DOD's revised
guidance on the maturity of technology to be used in the Advanced
Concept Technology Demonstration (ACTD) states that
". . . new technologies proposed for incorporation into an
ACTD should not be in the 6.1 (basic research) or 6.2 (applied
research) budget categories. Furthermore, the technologies must
have been successfully demonstrated at the subsystem or
component level and at the required performance level prior to
the start of the ACTD."
The new guidance, however, permits the selection of immature
technology--even as the primary or core technology--provided that it
is demonstrated prior to the ACTD demonstration. The guidance goes
on to describe several types of exceptions under which immature
technologies may be permitted to be used in an ACTD. Moreover, the
revised guidance would permit--as has occurred in some cases--the
approval of ACTD projects without the technologies having been
identified. Our point is that DOD should continue to focus the ACTD
program on the use of mature technology with few, if any, exceptions.
2. For the Combat Identification ACTD, the management plan for the
project shows that the technologies in question were integral parts
of the capabilities being evaluated. The demonstration manager told
us that the involvement of immature technologies was one of the
reasons that the project was behind schedule.
3. According to the Outrider project's management plan, the heavy
fuel engine was the primary technology to be used, not the
alternative. Also, the involvement of this immature technology was
one of the causes of schedule slippage and cost overruns on the
project.
4. DOD's most current ACTD guidance states
". . .Strategies and approaches are described to facilitate
transitioning from an ACTD to the acquisition process as defined
in DOD 5000.2R. The suggested approaches are based on lessons
learned. The focus of the suggestions are ACTDs that are
planned--if successful--to enter the acquisition process at the
start of LRIP."
Although there is a basic recognition that the transition to
development may be possible, the bulk of the guidance is on how and
when to transition to production. As pointed out in the report, the
guidance does not describe when a transition to development or what
types of development activity may be appropriate. In our view, the
guidance needs to be more balanced between the possibility of
transition to development and the transition of ACTD projects
directly to production.
5. As discussed in the report, the independent operational testing
agencies are observers in the ACTD demonstrations and not active
participants. While the Office of the Director of Operational Test
and Evaluation was an observer during the Predator demonstration, a
determination was not made that Predator was potentially effective
and suitable.
6. We agree that ACTDs address the technology's suitability.
However, the ACTD focus on suitability is in a very general sense and
extensive data is not collected on the system's reliability,
maintainability, and other aspects of suitability needed to support
production decisions.
7. As our report states, the Predator was rushed into low-rate
initial production prematurely given the limited amount of testing
conducted at that time and the problems that were uncovered during
that limited testing.
8. DOD's equating a determination of military utility (based on an
ACTD demonstration) with a determination of a system's readiness to
begin production is inappropriate because production decisions
require more testing data. During our review, we noted that
sufficient information was not obtained from an ACTD demonstration to
make a commitment to limited production. Commercial practice would
dictate that much more information be obtained about a product's
effectiveness, suitability, producibility, or supportability before
such a commitment is made.\1 We believe the ACTD guidance needs to be
more balanced and should anticipate that ACTD prototypes may need to
conduct more product and concept development and testing prior to
production. We have long held the view and have consistently
recommended that DOD use extreme caution to avoid premature
commitments to production.
9. We are not suggesting that a lengthy development phase be
conducted on all ACTD products nor, as DOD appears to suggest, that
an ACTD prototype may be ready to start limited production
immediately after its basic demonstration. As DOD stated in its
intent to establish the ACTD program, we believe the benefit of the
ACTD process is in eliminating or reducing technology development,
not in making early commitments to production or in postponing
product and concept development and testing activities until after
production starts.
10. While ACTD demonstrations are performed in operational
environments, they are not operational tests. During the course of
our work, we held several discussions with officials from the
operational test community. Those officials were in favor of the
user demonstrations featured in the ACTD program, but none considered
those demonstrations as substitutes for operational testing because
of their informality, lack of structure, and the lack of a defined
requirement by which to measure performance.
11. DOD appears not to recognize the very real possibility that the
ACTD demonstration may find the technology in question to have little
or no military utility or to be unaffordable in today's budgetary and
security environment. In fact, due to budget constraints, the Army
was forced to prioritize its procurement programs, and the planned
procurement funding for Enhanced Fiber Optic Guided missiles has been
reallocated.
12. While we agree with DOD that test results should form the basis
for starting limited procurement, the testing needed goes beyond the
basic demonstration of military utility provided by the ACTD program.
--------------------
\1 Best Practices: Successful Application to Weapon Acquisitions
Requires Changes in DOD's Environment (GAO/NSIAD-98-56, Feb. 24,
1998).
*** End of document. ***
|
NEWSLETTER
|
|
Join the GlobalSecurity.org mailing list
|
|
|
