Defense IRM: Poor Implementation of Management Controls Has Put Migration
Strategy at Risk (Chapter Report, 10/20/97, GAO/AIMD-98-5).
Pursuant to a congressional request, GAO provided information on the
Department of Defense's (DOD) Corporate Information Management (CIM)
initiative, focusing on: (1) the number and cost of systems designated
for migration, the number of legacy systems already terminated or
scheduled for termination, and savings resulting from terminations of
legacy systems; and (2) whether DOD's management control and oversight
processes for migration systems are ensuring that the investment in
CIM-related technology is economically sound and in compliance with its
technical and data standards.
GAO noted that: (1) from fiscal years 1995 through 2000, DOD plans to
spend at least $18 billion on migration; (2) DOD has selected 363
migration systems and has targeted 1,938 legacy systems for potential
termination; (3) despite this substantial investment, DOD did not adhere
to decisionmaking and oversight processes it established to ensure that
the economical and technical risks associated with migration projects
have been mitigated; (4) DOD's primary corporate-level control mechanism
for ensuring that sound management and development practices were
followed for each migration investment has not been effective; (5) DOD
does not have assurance that the migration systems being developed will
help achieve its technology goals and that sound business decisions were
made in selecting the systems; (6) DOD's traditional acquisition
oversight processes also did not support the migration effort because
they have not fully ensured that economic analyses for migration
projects are prepared and reviewed and that the systems comply with
technical and data standards; (7) had there been more rigorous attention
to established oversight procedures and sound business practices, DOD
might well have avoided the migration problems GAO identified in
previous reviews, which unnecessarily cost it hundreds of millions of
dollars; (8) in implementing the migration strategy, DOD did not ensure
that it had adequate departmentwide visibility over status, costs, and
progress; (9) as a result, it could not provide accurate and reliable
information, as requested, on the number and cost of systems designated
for migration, and the numbers of systems terminated and scheduled to be
terminated; (10) DOD has not been able to convincingly demonstrate
whether the migration strategy has been successful, and it has not been
able to provide the Congress with accurate and reliable information
needed for decisionmaking purposes; (11) DOD has taken a positive step
to turn around its information technology investment process as part of
its implementation of the Clinger-Cohen Act of 1996; (12) the Secretary
of Defense outlined his expectations for improvements in management
processes and information resources related to information technology;
and (13) these expectations incorporate some of the most important
elements of the Clinger-Cohen Act and are an excellent starting point
for bringing meaningful change to the current information technology
management process.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: AIMD-98-5
TITLE: Defense IRM: Poor Implementation of Management Controls Has
Put Migration Strategy at Risk
DATE: 10/20/97
SUBJECT: Information resources management
Information systems
Defense cost control
Defense procurement
Strategic information systems planning
Systems conversions
Systems design
Privatization
Cost effectiveness analysis
ADP procurement
IDENTIFIER: DOD Corporate Information Management Initiative
Defense Data Dictionary System
Defense Integration Support Tools Database
CIM
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** <info@www.gao.gov> **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Ranking Minority Member, Committee on Governmental
Affairs, U.S. Senate
October 1997
DEFENSE IRM - POOR IMPLEMENTATION
OF MANAGEMENT CONTROLS HAS PUT
MIGRATION STRATEGY AT RISK
GAO/AIMD-98-5
Defense Migration Strategy
(511355)
Abbreviations
=============================================================== ABBREV
ASDC3I - Assistant Secretary of Defense for Command, Control,
Communications and Intelligence
CIO - Chief Information Officer
CIM - Corporate Information Management initiative
DOD - Department of Defense
DAB - Defense Acquisition Board
DIICOE - Defense Information Infrastructure Common Operating
Environment
DISA - Defense Information Systems Agency
DIST - Defense Integration Support Tools
FY - fiscal year
IRM - information resources management
JTA - Joint Technical Architecture
MAISRC - Major Automated Information System Review Council
OMB - Office of Management and Budget
OSD - Office of the Secretary of Defense
PSA - Principal Staff Assistant
PA&E - Program Analysis and Evaluation
SFFAS - Statements of Federal Financial Accounting Standards
STARS - Standard Accounting and Reporting System
TAFIM - Technical Architecture Framework for Information Management
Letter
=============================================================== LETTER
B-275670
October 20, 1997
The Honorable John Glenn
Ranking Minority Member
Committee on Governmental Affairs
United States Senate
Dear Senator Glenn:
As you know, in 1989, the Department of Defense started its Corporate
Information Management (CIM) initiative in an effort to save billions
of dollars by streamlining operations and deploying standard
information systems to support common business operations. A key
part of this initiative is Defense's migration effort, which involves
replacing its functionally duplicative and inefficient automated
information systems with the Department's best existing information
systems. Concerned that the billions of dollars projected to be
spent on CIM-related technology efforts are at high risk, you asked
that we provide information on the status and progress of Defense's
migration effort and assess whether Defense has effective controls in
place to manage and oversee the initiative. This report also
highlights the implications of our findings for Defense as it
responds to investment management requirements of the Clinger-Cohen
Act of 1996.
We are sending copies of this report to the Chairman of the Senate
Committee on Governmental Affairs and to the Chairmen and Ranking
Minority Members of the Senate Committee on Armed Services;
Subcommittee on Defense, Senate Committee on Appropriations; House
Committee on National Security; Subcommittee on National Security,
House Committee on Appropriations; and Senate and House Committees on
the Budget; the Secretary of Defense; the Acting Assistant Secretary
of Defense for Command, Control, Communications and Intelligence; the
Acting Under Secretary of Defense (Comptroller); and the Director,
Office of Management and Budget. Copies will also be made available
to others upon request.
If you have any questions about this report, please call Mickey
McDermott, Assistant Director, at (202) 512-6240. Other major
contributors to this report are listed in appendix V.
Sincerely yours,
Jack L. Brock, Jr.
Director, Defense Information and
Financial Management Systems
EXECUTIVE SUMMARY
============================================================ Chapter 0
PURPOSE
---------------------------------------------------------- Chapter 0:1
In 1989, the Department of Defense started its Corporate Information
Management (CIM) initiative in an effort to save billions of dollars
by streamlining operations and deploying standard information systems
to support common business operations. However, 8 years after
beginning CIM and making substantial investments, Defense has not met
its savings goal because it has not fully implemented sound
management practices to carry out this initiative. GAO placed the
effort on its high-risk list in February 1995, in part because it
found that CIM-related technology investments, which are expected to
total billions of dollars each year, are vulnerable to waste and
mismanagement.
A key part of CIM is Defense's migration effort, which involves
replacing its functionally duplicative and inefficient automated
information systems with the best existing systems. Defense believes
that migration can cut costs associated with developing and
maintaining disparate systems supporting the same functions. It also
believes that if done properly, migration can help standardize
business processes. Concerned that billions of dollars projected to
be spent on CIM-related technology efforts are at high risk, the
Ranking Minority Member of the Senate Committee on Governmental
Affairs asked GAO to (1) provide information on the number and cost
of systems designated for migration, the number of legacy systems
already terminated or scheduled for termination, and savings
resulting from terminations of legacy systems and (2) determine
whether Defense's management control and oversight processes for
migration systems are ensuring that the investments are economically
sound and in compliance with its technical and data standards.
BACKGROUND
---------------------------------------------------------- Chapter 0:2
When it initiated CIM, Defense believed that the thousands of
automated information systems supporting its business
operations--which include such functions as logistics,
communications, personnel, health affairs, and finance--were
redundant and inefficient. These operations were traditionally
carried out in isolation by the individual military services and
Defense agencies. As such, the information systems supporting these
discrete operations were developed independently even though they may
well have served similar purposes.
DOD's CIM initiative included several aspects: (1) corporate
policy/planning, (2) process and data modeling, (3) process
improvement, (4) performance measurement, (5) standard information
systems, and (6) computing and communications infrastructure. As CIM
was implemented, Defense emphasized two ways of achieving process
improvements and addressing problems associated with its disparate
and stovepiped information technology environment: (1) reengineering
business processes first and then applying technology to the new
processes and (2) selecting the best DOD information systems from
pools of existing, or legacy, systems that provide the same automated
support services and eventually replacing the duplicative systems
with the best systems. The second approach is known as migration.
In October 1993, Defense embarked on an "accelerated" migration
strategy, which placed more emphasis on the second improvement
approach. As part of this strategy, it asked its managers to select
migration systems in 6 months and develop and deploy them
departmentwide in the following 3 years. Defense believed that
instilling pressure to select and deploy migration systems would reap
savings much quicker than streamlining or reengineering the business
processes and acquiring systems after those processes were
reengineered.
The consequence of the increased emphasis on migration was that the
dramatic gains that could be achieved through reengineering would be
postponed. This impaired the chances of CIM achieving its objectives
for dramatic improvements and cost savings in several respects.
First, it kept Defense from focusing on redesigning core business
processes, which promised order-of-magnitude improvements. Second,
it increased the risk that bad business processes would be
perpetuated. Third, it would make future reengineering efforts more
difficult by entrenching inefficient and ineffective work processes.
As a result, reengineering never took hold in many of DOD's
functional areas. The migration arm of CIM, however, is still
active.
RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3
From fiscal years 1995 through 2000, Defense plans to spend at least
$18 billion on migration. It has selected 363 migration systems and
has targeted 1,938 legacy systems for potential termination. Despite
this substantial investment, Defense did not adhere to
decision-making and oversight processes it established to ensure that
the economical and technical risks associated with migration projects
have been mitigated.
Defense's primary corporate-level control mechanism for ensuring that
sound management and development practices were followed for each
migration investment has not been effective. This control requires
that all migration system selections be approved by the Assistant
Secretary of Defense for Command, Control, Communications and
Intelligence
(ASD C3I). Despite this requirement, about 67 percent of migration
selections were never submitted for approval, and many of those that
were submitted were approved in the absence of critical technical and
programmatic support. Consequently, Defense does not have assurance
that the migration systems being developed will help achieve its
technology goals and that sound business decisions were made in
selecting the systems.
Because this control mechanism did not support the migration effort,
GAO also assessed whether Defense's traditional acquisition oversight
processes, which are designed to help ensure that individual major
information system investments are economically and technically
sound, were helping to ensure that the risks associated with
migration were mitigated. However, these processes also did not
support the migration effort because they have not fully ensured that
economic analyses for migration projects are prepared and reviewed
and that the systems comply with technical and data standards. For
example, economic analyses for 12 of 43 major migration systems--for
which Defense has invested hundreds of millions of dollars in
total--have not yet been submitted for independent review. Delaying
the preparation of an economic analysis to the later stages of
development defeats the purpose of the analysis--to demonstrate that
a proposal to invest in a new system is valid before that investment
is made.
Had there been more rigorous attention to established oversight
procedures and sound business practices, Defense might well have
avoided the migration problems GAO identified in previous reviews,
which unnecessarily cost the Department hundreds of millions of
dollars. One functional area, for example, embarked on and later
abandoned a substantially flawed effort to develop a standard suite
of migration systems for materiel management after spending over $700
million without strong oversight. In addition, some functional areas
did not account for various categories of significant costs when
making their migration decisions or adequately consider alternatives
to developing systems in-house.
In implementing the migration strategy, Defense did not ensure that
it had adequate departmentwide visibility over status, costs, and
progress. As a result, it could not provide accurate and reliable
information, as requested, on the number and cost of systems
designated for migration, and the numbers of systems terminated and
scheduled to be terminated. Furthermore, the Department has not been
able to convincingly demonstrate whether the migration strategy has
been successful or not, and it has not been able to provide the
Congress with accurate and reliable information needed for
decision-making purposes.
Defense has taken a positive step to turn around its information
technology investment process as part of its implementation of the
Clinger-Cohen Act of 1996 (Division E of Public Law 104-106). This
legislation requires federal agencies to have processes and
information in place to help ensure that information technology
projects (1) are being implemented at acceptable costs, within
reasonable and expected time frames, and (2) are contributing to
tangible, observable improvements in mission performance. In June
1997, the Secretary of Defense outlined his expectations for
improvements in management processes and information resources
related to information technology. These expectations incorporate
some of the most important elements of the Clinger-Cohen Act and are
an excellent starting point for bringing meaningful change to the
current information technology management process. Additionally, the
Department's Task Force on Defense Reform is currently examining the
current structure of the Chief Information Officer (CIO) position to
ensure that the CIO can devote full attention to reforming
information resources management within the Department.
In implementing the Clinger-Cohen Act, the Department faces a
formidable challenge in successfully implementing real change across
an organizational structure that has clearly defined roles and
responsibilities for the three individual services to execute
national defense policy and objectives. The separation of budget
authority, program execution, and functional authority have all
contributed to an environment that has fostered stovepipe systems
within each service and has made departmentwide oversight difficult.
For Clinger-Cohen implementation to make a difference, any new
processes or requirements must be successfully accomplished within
this environment.
PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4
MANAGEMENT CONTROL PROCESSES
OVER INDIVIDUAL MIGRATION
EFFORTS HAVE BROKEN DOWN
-------------------------------------------------------- Chapter 0:4.1
Defense's primary corporate-level control mechanism for ensuring that
sound management and development practices are followed for each
migration investment has not been effective. This control requires
the functional area manager, known as the Principal Staff Assistant\1
(PSA), to submit all systems selected for migration for approval by
the Assistant Secretary of Defense for Command, Control,
Communications and Intelligence. This approval is important because
the Assistant Secretary, as the Department's Chief Information
Officer, needs to ensure that DOD's technology-related goals are met.
However, this control mechanism has broken down in two respects.
First, even though all system selections should have been submitted
for review and approval by the Assistant Secretary, 245 of the 363
selections were never submitted for this oversight. Second, most of
the systems that were submitted for this review were approved in the
absence of critical technical and programmatic support.
Specifically, about 19 percent were approved without supporting
technical justification and about 54 percent were approved without
documents showing that the functional area had evaluated different
options for improving a business area, such as reengineering.
Because all of the migration selections submitted for this oversight
have been approved--whether adequately supported with technical and
programmatic justification or not--this control has essentially
become meaningless.
Because the oversight from the ASD C3I that was specifically
established for migration selections has been marginal in trying to
ensure that sound management and development practices are followed
for migration selection, GAO assessed whether the Department's
traditional major information systems acquisition oversight processes
were doing so for 43 of the major migration systems.\2 Major systems
are Defense's most expensive and critical information systems and can
cost up to hundreds of millions of dollars to develop. These review
processes are designed to assess whether projects are affordable and
financial and operational risks have been minimized.
However, these processes did not provide sufficient assurance that
the systems (1) were economically justified or (2) complied with
Defense's technical and data standards--which are intended to help
pave the way toward an interoperable systems environment. For
example, economic analyses or updates of previously prepared economic
analyses for 12 of the 43 systems had not been submitted for
independent review. These 12 systems are under development or
modernization and Defense has invested hundreds of millions of
dollars in them. In addition, even though the Department recognizes
that economic analyses play a critical role in assessing whether
system development efforts will be cost-effective and beneficial, it
has not yet published official guidance to standardize the
methodology and analytic techniques for preparing economic analyses.
GAO's previous reviews of migration systems identified a number of
problems that could have been prevented had there been better
oversight by the Major Automated Information System Review Council
(MAISRC) and the ASD C3I. For example, GAO's review of the materiel
management migration strategy showed that a functional area was able
to embark and spend over $700 million pursuing a substantially flawed
effort--which was later abandoned--without rigorous department-level
oversight. In addition, in previous reviews of migration efforts in
depot maintenance, transportation, and finance, GAO found that
functional areas failed to account for various categories of
significant costs when making their migration decisions--including
costs related to interfacing migration systems with other systems,
project management, and the system selection process. More rigorous
oversight and guidance could have helped ensure that these costs were
included.
GAO also found a lack of good departmentwide visibility over the
migration effort in terms of project costs and progress. For
example, DOD has not been tracking key performance issues, such as
cost savings resulting from migration or management and staff
productivity improvements. Further, it does not have a complete
picture of the costs of all migration projects and its scheduling
information is inaccurate and unreliable. As a result, the
Department has not provided its own department-level decisionmakers
with information needed to manage the effort, and it has not provided
the Congress with complete and accurate information on migration.
Moreover, in the absence of good performance measures for migration,
the Department has not been able to show whether the overall strategy
has been successful or not.
--------------------
\1 The PSA is the senior executive-level manager who is responsible
for the management of a defined function or functions within the DOD.
\2 There are 49 major migration systems; however, GAO's review
focused on the 43 major migration systems under the oversight of the
Major Automated Information System Review Council. The remaining six
systems are under the oversight of the Defense Acquisition Board.
REFORMING DOD'S INFORMATION
TECHNOLOGY INVESTMENT
PROCESS REQUIRES EFFECTIVE
IMPLEMENTATION OF THE
CLINGER-COHEN ACT
-------------------------------------------------------- Chapter 0:4.2
Defense recognizes the need for a better information technology
investment environment and has taken steps to implement the
Clinger-Cohen Act of 1996. The Congress passed the act in an effort
to put an end to poorly managed and wasteful information technology
projects. Among other things, it requires agencies to adopt an
investment process that provides for the continual identification,
selection, control, life-cycle management, and evaluation of
information technology projects. As a first step in implementing the
act, the Secretary of Defense directed the ASD C3I, as the
Department's Chief Information Officer, to take the lead in
implementing an investment process as well as a performance- and
results-based management strategy for information technology.
This is an important move toward bringing meaningful change to the
current decision-making and oversight environment for migration.
However, the current structure of the CIO position in the Department
seriously limits the CIO's ability to effectively serve as a bridge
between top management, line management, and information management
support officials and to identify opportunities to use information
technology to enhance performance. At present, the ASD C3I also
serves as the Department's CIO. Asking the same individual to serve
in both capacities prevents the CIO from devoting full attention to
reforming information resources management within the Department. It
also means that the ASD C3I will continue to be responsible for
providing oversight for the same systems that he is responsible for
selecting, developing, and implementing in his capacity as a PSA.
This shortchanges much-needed independent oversight for about 45
percent of the Department's migration system investment.
Finally, effective implementation of the Clinger-Cohen Act will not
occur unless Defense's information and system investment control
processes successfully address the challenge posed by the prevailing
organizational structure and culture found throughout the Department.
This condition has promoted stovepipe systems solutions in each
component agency and has made it difficult to implement
departmentwide oversight or visibility over information resources.
This same condition has contributed to the difficulty that has
limited the Department in modernizing business processes and
implementing corporate information systems across service and agency
lines. This is most evident in the perceived failure of the CIM
initiative, which was intended to reengineer business processes
throughout the Department. By doing so, the Department expected to
save billions by having more efficient, effective business processes
running across service and component lines. However, these benefits
have yet to be widely achieved after 8 years of effort. Without the
Secretary's strong and continued support for management processes and
controls designed to improve information management initiatives,
Clinger-Cohen implementation could well suffer similar results.
RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5
To ensure that continued investment in migration systems provides
measurable improvements in mission-related and administrative
processes, GAO is recommending that the Secretary of Defense require
Defense components to rank development/modernization systems
justifications and complete them on an expedited basis. GAO is
further recommending that the Chief Information Officer certify that
these justifications include the following:
-- An analysis of operational alternatives that clearly demonstrate
that continuing with migration is the best solution for
improving performance and reducing costs in the functional area.
-- An economic analysis showing a return on investment or other
mission benefits that justify further investment.
-- Documentation showing that the system currently complies with
applicable Defense technical standards and uses standard data.
GAO's recommendations are also aimed at (1) correcting weaknesses
within the current life-cycle management environment and (2) ensuring
the successful reengineering of processes and implementing of
corporate information systems for functional areas.
AGENCY COMMENTS
---------------------------------------------------------- Chapter 0:6
The Acting Assistant Secretary of Defense for Command, Control,
Communications and Intelligence provided written comments on a draft
of this report. Defense concurred with five recommendations and
partially concurred with two recommendations. Defense did not concur
with the remaining five recommendations and expressed concerns about
certain aspects of the report. Generally, the Department concurred
with GAO's recommendations that it revise or develop internal
policies and procedures to conform to the Clinger-Cohen Act, develop
and implement performance measures, and improve the performance of
internal tracking systems.
The Department did not agree to a proposal in the draft report that
it limit further investments in ongoing migration systems to those
that meet critical needs until they are independently determined to
be economically, functionally, and technically justified. Defense
believed that such a limitation would not only adversely affect
military readiness, system development, and contract obligations, but
also increase system obsolescence. GAO understands DOD's concerns
regarding readiness and contract obligations. That is why the
recommendation in the draft report recognized that critical needs
still need to be met. GAO's point is that greater management
attention needs to be placed on the migration decision-making process
for approving and funding the development and modernization of
migration systems to achieve intended benefits and minimize
unnecessary costs. In order to better clarify this position, GAO
reworded its recommendation to focus on strengthening the controls
for the decision-making process so that these investments can be
better considered in the Department's budget process.
Defense also did not concur with a proposal in the draft report that
it consider separating the CIO and the ASD C3I positions so that CIO
can devote full attention to departmentwide information resource
management issues and provide independent oversight. It noted,
however, that all ASD C3I functions are being reviewed by the
Department's Task Force on Defense Reform as part of its review of
the Office of the Secretary of Defense's organizational structure.
GAO withdrew the proposal because DOD is now considering this matter.
Nevertheless, the concern that the current structure of the CIO
position does not allow the CIO to devote full attention to critical
IRM issues--such as computer security, the Year 2000 problem, and the
need to develop and implement an integrated information technology
architecture--remains valid.
Defense also stated that the report negatively portrays Defense's
efforts to streamline its acquisition processes. GAO disagrees.
Acquisition streamlining allows the Department to greatly simplify
its acquisition processes for information technology purchases.
However, acquisition streamlining was not intended to excuse DOD from
exercising management controls to ensure that those purchases and
related system development efforts made good business sense.
Clinger-Cohen implementation should allow the Department an
opportunity to develop more appropriate controls.
Defense's comments are discussed in chapter 4 and reprinted in
appendix I.
INTRODUCTION
============================================================ Chapter 1
BACKGROUND
---------------------------------------------------------- Chapter 1:1
The Department of Defense (DOD) began its Corporate Information
Management initiative (CIM) in October 1989 to help meet the
challenge of effectively managing its diverse operations as it
downsized its forces and activities. At the time, the Department
believed that the thousands of systems and numerous administrative
and mission-related processes\1 supporting DOD functions were
redundant and inefficient and that they should be standardized and
made more efficient. To address these problems, Defense planned to
-- simplify and improve business processes;
-- centralize responsibility and authority in functional areas,
such as finance, personnel, communications, health affairs,
logistics, command and control, and intelligence; and
-- develop an integrated communications and data processing
infrastructure based on departmentwide standards.
From the outset, DOD recognized that implementing CIM would be
difficult because the basic tenet of the initiative--managing and
implementing business improvements and building corporate systems
along functional lines--represented a major shift in the way Defense
traditionally did business. Whereas each military and Defense agency
had historically managed its own business functions and information
systems, CIM called on senior functional officials, known as
Principal Staff Assistants (PSAs),\2 together with their Defense
component counterparts to be responsible for implementing
improvements and information systems within the Department's business
functions across service and agency lines.
As CIM was implemented, Defense emphasized two ways of achieving
process improvements and addressing problems associated with its
disparate and stovepiped information technology environment.\3 The
first was to improve, or reengineer, business processes first and
then apply technology to the new processes. The second involved
selecting the best DOD information systems from pools of existing, or
legacy, systems that provide similar automated support services and
eventually replacing the duplicative systems with the best systems.
A few years into the CIM effort, this became known as migration.\4
Defense believed that if done properly, migration could cut costs
associated with developing and maintaining disparate systems
supporting the same functions. It also believed that migration would
help to standardize business processes and allow Defense to achieve
savings that could be put to better use in advancing warfighting
capabilities. Figure 1.1 illustrates how migration would work in one
functional area.
Figure 1.1: An Example of
Migration in Practice
(See figure in printed
edition.)
--------------------
\1 Also known as business processes.
\2 PSAs are the Under Secretaries of Defense, the Director of Defense
Research and Engineering, the Assistant Secretaries of Defense, the
Director of Operational Test and Evaluation, the General Counsel of
DOD, the Inspector General, the Assistants to the Secretary of
Defense and the Office of the Secretary of Defense (OSD) Directors or
equivalents who report directly to the Secretary or the Deputy
Secretary of Defense. In essence, they are top executives at the
department level who, together with their Defense component
counterparts, are charged with developing corporate systems and
reengineering business processes within their respective functional
areas.
\3 The CIM initiative included several aspects, such as (1) corporate
policy/planning, (2) process and data modeling, (3) process
improvement, (4) performance measurement, (5) standard information
systems, and (6) computing and communications infrastructure. DOD
considered all the aspects of CIM to be important to effectively
field information systems that support its mission.
\4 In DOD, migration can also involve developing or acquiring a new
system, rather than choosing from existing systems. However, most
migration efforts have involved choosing from existing systems.
MIGRATION HAS BECOME THE FOCAL
POINT FOR CIM
---------------------------------------------------------- Chapter 1:2
While Defense began CIM emphasizing the need to improve, or
reengineer, processes before applying technology, it ended up placing
more priority on obtaining quick savings through an accelerated
migration strategy, which began in October 1993. This strategy
called for selection of migration systems in 6 months and
departmentwide transition to selected systems in the following 3
years. Defense believed that setting tight time frames for migration
with some potential slippage would allow it to "harvest the
low-hanging fruit" of potential savings before reengineering. By
default, this meant that the increased emphasis on migration would
postpone the dramatic gains that could be achieved through
reengineering.\5
The risks involved with postponing reengineering efforts were
significant. Reengineering identifies, analyzes, and redesigns an
organization's core business processes, aiming to achieve dramatic
improvements in critical areas of performance, such as cost, quality,
service, and speed. It focuses on redesigning the business process
as a whole in order to achieve the greatest possible benefits to an
organization and its customers. Migration, on the other hand,
focuses on standardizing existing processes and information systems.
This can yield some benefits, such as reducing the need to maintain
disparate systems that support the same functions, but seldom yields
dramatic improvements. In addition, if the process is inefficient or
outmoded, migration only serves to perpetuate a bad process.
Finally, choosing migration before reengineering may cause future
reengineering efforts to be more difficult by entrenching inefficient
and ineffective work processes.
We raised these concerns both before\6 and after\7 Defense decided to
embark on the accelerated migration strategy. We believed that the
shift in emphasis toward migration seriously endangered CIM's chances
for success. For this reason, and because CIM-related technology
investments, which are expected to total billions of dollars each
year, are vulnerable to waste and mismanagement, we designated CIM as
a high-risk government information technology initiative.\8
Nevertheless, Defense proceeded to concentrate on migration even in
areas where it had recognized that there was a significant need to
change business processes. One such area was transportation. We
reported in 1996 that even though Defense recognized that its
military transportation processes were fragmented, outdated,
inefficient, and costly, it focused on technology solutions rather
than the need to identify and correct the root causes of its
transportation problems.\9
Because reengineering took a backseat to migration in other business
areas as well, CIM has never been able to achieve the level of cost
savings and process improvements originally intended. A recent
report\10 conducted by RAND, a consulting organization, notes that
"The CIM effort is today widely viewed as a failure in most quarters
of the DOD. It has not resulted in either significant process
reengineering or visible savings in the hardware and software
required to support all the varied information systems in the Defense
infrastructure."
Defense is still planning to invest at least $18 billion in its
migration effort from fiscal years 1995 through 2000. Because this
investment is significant, we were asked for information on the
status and progress of migration and whether Defense has significant
controls in place to manage and oversee the effort.
--------------------
\5 In an October 1993 memorandum, the Deputy Secretary of Defense
announced a near-term strategy that focused on migration and data
standardization. The memorandum stated that completion of other
initiatives were not to be prerequisites of implementation of
migration systems and data standards. While some functional areas
may have continued process reengineering initiatives, our work
indicates that many others gave priority to the migration strategy.
\6 Defense ADP: Corporate Information Management Must Overcome Major
Problems (GAO/IMTEC-92-77, September 14, 1992).
\7 Defense Management: Stronger Support Needed for Corporate
Information Management Initiative to Succeed (GAO/AIMD/NSIAD-94-101,
April 12, 1994) and Defense Management: Impediments Jeopardize
Logistics Corporate Information Management (GAO/NSIAD-95-28, October
21, 1994).
\8 Our high-risk effort, which began in 1990, identifies those
federal program areas we consider high risk because they are
especially vulnerable to waste, fraud, abuse, and mismanagement. See
High-Risk Series: Information Management and Technology
(GAO/HR-97-9, February 1997) and High-Risk Series: An Overview
(GAO/HR-95-1, February 1995) for a discussion on CIM.
\9 Defense Transportation: Migration Systems Selected Without
Adequate Analysis (GAO/AIMD-96-81, August 29, 1996).
\10 Strategic Appraisal 1997: Strategy and Defense Planning for the
21st Century, February 1997, RAND's Project Air Force; edited by
Zalmay M.Khalilzad and David A Ochmanek.
STATUS OF THE MIGRATION
EFFORT
-------------------------------------------------------- Chapter 1:2.1
Defense reported that it has selected 363 systems for migration.
Forty-nine of the 363\11
migration systems are large-scale, or major, systems\12 that are
expected to cost at least $13 billion--or about 72 percent of the
total $18 billion cost estimate--to develop, deploy, and maintain
over fiscal years 1995 through 2000. Defense has identified 1,938
legacy systems for potential termination. As of April 1996, its
Defense Integration Support Tools (DIST) database showed that 281 had
already been terminated, 886 were scheduled for termination by the
year 2000, and 771 more were identified for potential elimination
after the year 2000.
Table 1.1 provides the number of systems selected by each functional
area, the number of legacy systems identified for potential
termination, and DOD's estimated costs of developing, deploying, and
maintaining the migration systems. We excluded costs for 27 systems
for which DOD collected costs because the systems were classified.
And costs for the remaining 121 of 363 systems were not included in
the table because the Office of the ASD C3I had not collected the
costs for these systems. Table 1.2 provides the name, status, and
cost for each of the 49 major systems.
Table 1.1
Status Reported to GAO on Defense
Migration Systems Selections and Costs
(Dollars in millions)
Number of Total cost
legacy systems Number of Number of reported to GAO
identified for migration/ migration/ for fiscal
Functional potential interim systems interim systems years 1995
area termination selected reporting costs through 2000
----------- -------------- ---------------- ---------------- ---------------
Command and 67 35 34 4,493.2
Control
Intelligenc 304 69 46 572.6\a
e
Mission
Support
Atomic 7 8 2 12.7
Energy
Communicati 17 2 2 3,483.2
ons
Environment 286 11 6 33.1
al
Security
Finance\b 267 52 31 1,417.1
Health 62 55 49 2,533.8
Human 198 20 10 933.1
Resources
Information 21 5 0 0.0
Management
Inspector 8 2 1 15.8
General
Logistics 496 63 53 4,459.1
Meteorology 33 33 0 0.0
and
Oceanograp
hy
Policy 3 7 7 73.9
Procurement 67 1 1 282.1
Science and 102\c 0\c 0\c \c
Technology
================================================================================
Total 1,938 363 242 $18,309.7
--------------------------------------------------------------------------------
Note: We did not independently verify information provided in this
table.
\a Costs were not included for 27 systems for which DOD collected
cost data because the systems were classified.
\b The information presented for finance is not consistent with
information presented in figure 1.1 of this chapter because the
counts of systems occurred at different points in time.
\c The science and technology functional area has not yet selected
migration systems and has not specified which, if any, of its legacy
systems may be terminated.
Source: Information obtained from the DIST, DOD's April 1996 Report
to the Congress, discussions with Defense functional area managers,
and the ASD C3I. Cost data were provided as of December 1996.
Table 1.2
Major Migration and Interim Systems/
Applications Selected for Each
Functional Area
(Dollars in millions)
Total cost data
reported to GAO
Current for fiscal years
milestone\ 1995 through
System name Acronym a 2000
-- ---------------------- ---------------------- ---------- ----------------
Systems being reviewed by
--------------------------------------------------------------------------------
Command and Control sys
--------------------------------------------------------------------------------
1 Advanced Field AFATDS 3 $ 320.6
Artillery Tactical
Data System
2 Combat Service Support CSSCS 2 104.9
Control System
3 Forward Area Air FAADC2I 3 216.4
Defense C2
Intelligence System
4 Maneuver Control MCS 3 253.7
System
Intelligence systems
--------------------------------------------------------------------------------
5 All Source Analysis ASAS 3 347.9
System
6 Joint Service Imagery JSIPS 2 \b
Processing System
Systems being reviewed by systems):
--------------------------------------------------------------------------------
Command and Control sys
--------------------------------------------------------------------------------
1 Air Force Mission AFMSS 3 351.9
Support System
2 Army Global Command AGCCS 3 254.6
and Control System
3 Counter Narcotics CN/CMS 3 112.2
Command Management
Control System
4 Global Command and GCCS 3 422.1
Control System
5 Naval Aviation NALCOMIS 3 93.5
Logistics Command
Management Information
System
6 Naval Tactical Command NTCS-A 3 314.1
System -Afloat
7 Operations Support OSS 3 174.1
System
8 Shipboard Non- SNAP III 3 401.4
Tactical ADP Program
III
9 Strategic War Planning SWPS 3 343.4
System
10 Tactical Support TSC 3 154.3
Center
Intelligence systems
--------------------------------------------------------------------------------
11 Exploitation Support ESS 0 \b
System
12 High Performance HPCMP 2 \b
Computing
Modernization Program
Mission support systems
--------------------------------------------------------------------------------
Communications
--------------------------------------------------------------------------------
13 Defense Information DISN 0 2,556.2
System Network
14 Defense Message System DMS 2 927.0
Finance
--------------------------------------------------------------------------------
15 Defense Joint Military DJMS 1/2 114.9
Pay System
16 Defense Procurement DPPS 1/2 68.5
Payment System
17 Standard Accounting STARS 2/3 263.7
And Reporting System
Health
--------------------------------------------------------------------------------
18 Ambulatory Data System ADS 3 90.3
19 Composite Health Care CHCS 3 785.6
System
20 Composite Health Care CHCS II 0 1.4
System II
21 Corporate Executive CEIS 0/1/2 18.2
Information System
Human resources
--------------------------------------------------------------------------------
22 Defense Civilian DCPDS 1 58.5
Personnel Data System
23 Defense Commissary DCIS 2 120.8
Information System
24 Defense Commissary POS 3 262.4
Point of Sales System
25 Joint Recruiting JRISS 0 207.0
Information Support
System
26 Navy Standard NSIPS 0 \b
Integrated Personnel
System
27 Reserve Component RCAS 2/3 \b
Automation System
28 Standard Installation/ SIDPERS-3 2 \b
Division Personnel
System -3
Logistics
--------------------------------------------------------------------------------
29 Ammunition Management AMSS 0 55.7
Standard System
30 Command and Control C2IPS 3 311.8
Information Processing
System
31 Defense Automatic DAAS 3 12.7
Addressing System
32 Defense Medical DMLSS 2 129.0
Logistics Standard
Support System
33 Department of Army DAMMS-R 3 43.9
Movements Management
System -Redesigned
34 Depot Maintenance DMS 0\c 438.6
System
35 Distribution Standard DSS 3 270.2
System
36 Global Transportation GTN 2 179.5
Network
37 Joint Computer-Aided JCALS 2 789.0
Acquisition and
Logistics System
38 Joint Engineering Data JEDMICS 3 243.7
Management Information
and Control System
39 Materiel Management MMS 0\c 1,149.2
System
40 Transportation TC-AIMS II 0 15.1
Coordinators'
Automated Information
for Movement System II
41 Transportation TOPS 3 95.4
Operational Personal
Property System
Meteorology & Oceano
--------------------------------------------------------------------------------
42 Primary Oceanographic POPS 3 \b
Prediction System
Procurement
--------------------------------------------------------------------------------
43 DOD Standard SPS 1 282.1
Procurement System
================================================================================
Total $13,355.5
--------------------------------------------------------------------------------
Note: We did not verify the information provided in this table. As
discussed in chapter 2, these costs may be inaccurate.
\a There are four milestones in the major system acquisition review
process. These are milestone 0--approval to conduct concept studies;
milestone 1--approval to begin a new acquisition program; milestone
2--approval to enter engineering and manufacturing development; and
milestone 3--approval to produce, field, or deploy the system.
Multiple milestones may be shown for some projects because the
projects' subsystems or applications are in different phases of the
acquisition process.
\b The Office of the ASD C3I did not collect costs for this system.
\c Both of these programs were redirected and are no longer targeted
to be standard migration systems.
Source: Information DOD reported to the Congress in April 1996 and
to GAO in the course of this audit.
--------------------
\11 The numbers of migration systems selected and legacy systems
identified for potential termination are subject to change as PSAs
make additional selections and reevaluate previous ones. Also, the
PSAs classified 138 of the 363 migration systems as interim systems.
\12 In DOD, major information systems projects are those that (1)
have estimated program costs in excess of $30 million in any 1 year,
(2) have estimated program costs of over $120 million in total, (3)
have total life-cycle costs of over $360 million, or (4) are
designated as being of special interest. DOD periodically revises
these dollar thresholds.
OBJECTIVES, SCOPE, AND
METHODOLOGY
---------------------------------------------------------- Chapter 1:3
We were asked (1) for information on the number and cost of systems
designated for migration, the number of legacy systems already
terminated or scheduled for termination, and savings resulting from
terminations of legacy systems and (2) whether Defense's management
control and oversight processes for migration systems are ensuring
that the investments are economically sound and in compliance with
Defense's technical and data standards.
To assess the status of Defense's overall migration strategy and
obtain available information on the number and cost of systems
designated for migration, the number of legacy systems already
terminated and scheduled for future termination, and savings
resulting from terminations of legacy systems, we analyzed a Defense
April 1996 report to the Congress containing information on the
schedule, cost, and status of the migration systems. Defense
prepared this report\13 in response to a requirement in Section 366
of the National Defense Authorization Act for fiscal year 1996. We
also obtained and analyzed a copy of the DIST database as of April
1996 that Defense used to develop the report to the Congress, as well
as to provide its own senior managers information on migration system
selections and legacy system terminations. Additionally, we
reviewed--but did not independently verify--cost information that
Defense reported to Congress as part of the April 1996 Section 366
report. We also interviewed senior DOD officials to determine if
additional cost and performance information on migration systems
existed at the Office of the Secretary of Defense (OSD) level.
We assessed the reliability of the data presented to the Congress in
two ways. First, we identified missing and conflicting information
in the April 1996 Section 366 report to the Congress and requested
that the ASD C3I staff to clarify conflicting information. Second,
we compared the data reported to the Congress for three functional
activities (business areas)--Transportation, Civilian Personnel, and
Clinical Health--against data provided to us by the functional area
managers at these activities. We then established and analyzed a
database containing schedule and available cost information for the
migration and legacy systems and modified the database to reflect
updated schedule, costs, and other descriptive data Defense provided
to us during the course of the audit.
To assess Defense's management control and oversight processes for
migration systems to determine whether the investments are
economically justified and comply with Defense technical and data
standards, we reviewed the department-level management and oversight
processes for approving the PSAs' migration system selections and for
overseeing major migration systems' acquisitions by the Major
Automated Information System Review Council (MAISRC).\14 Our review
did not focus on the process by which the PSAs select, develop, and
manage their migration systems. Nor did we examine acquisition
review processes within the individual functional areas employed for
nonmajor system projects.
To analyze Defense's oversight processes for reviewing and approving
the PSAs' migration system selections, we reviewed department-level
approval of the PSAs' selections. We obtained a list of migration
systems that PSAs had selected for their functional areas. We then
visited the Office of the ASD C3I and the Defense Information Systems
Agency (DISA) and reviewed the business case analysis documentation,
technical analysis documentation, and other documentation provided to
those offices for department-level approval of the PSA's migration
and interim system selections. We also interviewed officials from
the Office of the ASD C3I, DISA, and the offices of selected PSAs
regarding the documentation supporting the selections.
To review Defense's acquisition oversight processes for major
migration systems, we first identified the major migration systems,
as defined by Defense regulations. We then obtained and analyzed the
progress reports and other documentation provided to MAISRC for those
systems. We also interviewed Defense representatives and officials
responsible for information systems oversight in the Office of the
ASD C3I and offices of DISA, the Defense Acquisition Board (DAB),
MAISRC, and Program Analysis and Evaluation (PA&E). We also
interviewed representatives of the program offices or oversight
offices for selected systems for which oversight had been delegated
by MAISRC. Through document reviews and interviews, we determined
whether economic analyses for major migration systems had been
independently reviewed and if so, whether the reviews had identified
problems in the analyses. We also determined whether DOD guidance
existed on preparing economic analyses for information systems. A
GAO economist met with PA&E analysts, reviewed problems that were
identified with economic analyses, and verified the validity of the
problems reported.
Additionally, we determined whether MAISRC had information on
alternative analyses performed for major migration system selections.
Lastly, we determined whether MAISRC has sufficient oversight
information to ensure that major migration systems are complying with
applicable technical and data standards that are necessary to achieve
an interoperable systems environment.
We reviewed Defense's policies and guidance for migration systems to
ensure that information technology is acquired, managed, and used in
the most efficient and effective manner. Our assessment included
analyzing the National Defense Authorization Acts, Committee reports,
and Conference reports for fiscal years 1993 through 1997; Defense
Office of Inspector General reports; our prior studies of CIM and the
migration system strategy, other available evaluations of the CIM and
the migration strategy, and existing legislation affecting these
efforts. The major studies reviewed included studies by RAND and the
Defense Science Board.\15 Pertinent existing legislation includes the
Clinger-Cohen Act, the Government Performance and Results Act, the
Paperwork Reduction Act, and the Chief Financial Officers Act.
We conducted our review from August 1996 through July 1997 in
accordance with generally accepted government auditing standards. We
requested comments on a draft of this report from the Department of
Defense. The Acting Secretary for Command, Control, Communications
and Intelligence provided us with written comments. These comments
are discussed in chapter 4 and reprinted in appendix I.
--------------------
\13 As noted in this chapter, this report did not include costs for
121 information systems.
\14 MAISRC provides acquisition oversight for information systems
that (1) are anticipated to cost $30 million or more a year, (2) have
estimated program costs in excess of $120 million, (3) have estimated
life-cycle costs of more than $360 million, or (4) are designated for
review by the ASD C3I, who also chairs the MAISRC. MAISRC reviews
such matters as whether a proposed system is being developed in
accordance with Defense policies, procedures, and regulations and
whether system managers took steps to minimize the cost of a new
system.
\15 Achieving an Innovative Support Structure for 21st Century
Military Superiority: Higher Performance at Lower Costs, 1996 Summer
Study, Defense Science Board, Department of Defense. The Defense
Science Board is a Federal Advisory Committee established to provide
independent advice to the Secretary of Defense. Statements,
opinions, conclusions, and recommendations in their reports do not
necessarily represent the official position of DOD.
MANAGEMENT CONTROL PROCESSES OVER
INDIVIDUAL MIGRATION EFFORTS HAVE
BROKEN DOWN
============================================================ Chapter 2
Embarking on the migration strategy was an extremely risky endeavor
for the Department of Defense. First, in developing standard systems
within functional areas, the Department had to carefully consider
complex technical issues, such as interfacing migration systems with
other systems and contending with nonstandard data formats and
definitions. For Defense, such complexities were compounded by its
sheer size and the numbers of disparate systems. We believe an even
tougher obstacle facing Defense, however, was the prevailing culture,
which was based on decentralized department organizational structure,
nonstandard processes and procedures. In general, each military
service and Defense agency has historically managed its own business
functions and information technology projects, whereas CIM and
migration required business improvements and information systems to
be managed on a Departmentwide, or corporate basis.
Therefore, to ensure the strategy's success, it was vital for Defense
to establish an effective decision-making and oversight environment
for making migration investments. It would need controls and
processes that ensured inefficient administrative and mission-related
work processes were modernized before significant technology
investments were made to support them. It would also need controls
that ensured that the migration projects themselves were effectively
managed as investments so that the Department could target resources
and attention to priority areas and stop those projects that failed
to meet their goals. Finally, Defense needed life cycle management
controls that ensured, on a system-by-system basis, that sound
management and development practices were followed.
However, Defense's primary corporate-level control mechanism for
ensuring that sound management and development practices were
followed for each migration investment has not been effective. This
control requires that all selections to be submitted for approval by
the Assistant Secretary of Defense for Command, Control,
Communications and Intelligence (ASD C3I), who is also the
Department's Chief Information Officer (CIO). In approving the
systems, the Assistant Secretary is to review data, technical, and
programmatic factors relating to the selection. We found, however,
that most migration selections never came in for this approval and
those that did were approved in the absence of critical technical and
programmatic supporting documents. (A description of the migration
decision process is provided in appendix II.)
Because this control was playing a marginal role in ensuring the
success of migration, we assessed whether Defense's acquisition
oversight processes--which are designed to augment management
oversight for the Department's most expensive information
systems--were helping to ensure that the major migration investments
were economically and technically sound. These processes have also
broken down where migration is concerned. Had there been more
rigorous attention to oversight in both areas, Defense may well have
avoided the migration problems we identified in previous reviews that
cost the Department hundreds of millions of dollars.
In implementing the migration strategy, Defense did not ensure that
it had adequate visibility over status, costs, and progress. As a
result, it has not been able to (1) demonstrate whether the migration
strategy has been successful, (2) provide the Congress with accurate
and reliable information needed for oversight purposes, and (3)
provide its own decisionmakers at the headquarters level with
information needed to oversee the strategy.
MOST SYSTEMS NOT APPROVED BY
DEFENSE'S SENIOR TECHNOLOGY
MANAGER
---------------------------------------------------------- Chapter 2:1
After selecting a migration system, the Principal Staff Assistant
(PSA)\1 responsible for a functional area is to submit the system for
approval by the Assistant Secretary of Defense for Command, Control,
Communications and Intelligence--the Senior Information Management
Official and Chief Information Officer in DOD. In his memorandum\2
setting forth the requirements for selecting and reviewing migration
systems, the ASD C3I stated that in approving the selections, he
would consider data, technical, and programmatic factors. For
example, the ASD C3I review would include such issues as whether the
migration systems will lend themselves to data sharing and whether
they conform to DOD's technical standards--which act as a set of
"building codes" for constructing systems to ensure they will be
compatible with its information infrastructure and technically
interoperable with each other.\3 This approval is important because,
as the Department's Chief Information Officer, the Assistant
Secretary needs to ensure that the migration systems help facilitate
the sharing of information across service and agency lines.
However, both the PSAs and the ASD C3I did not adhere to this
oversight process. First, 245 selections, or about 67 percent, were
never submitted for this oversight even though all 363 system
selections should have
been.\4 \, \5 Of the 49 major, or large scale, migration systems,
only 16 were submitted for this review. Thus, for the bulk of
migration systems, the Assistant Secretary was unable to ensure that
Defense technical standards would be met and that the best system
development practices were being followed.
Second, most of the systems that were submitted for this review were
approved in the absence of critical technical and programmatic
business case support. For example, 23 of the 118 selections that
were submitted for this oversight, about 19 percent, were approved
without documents stating that the functional area complied or
planned to comply with technical standards. In addition, 64 of the
118 selections, about 54 percent, were approved without documents
showing that the PSA responsible for the functional area performed
some type of functional economic analysis or other similar business
case analysis. This analysis is the means by which Defense managers
are supposed to evaluate different options for improving business
areas, such as outsourcing, reengineering, or migration. Because all
of the migration selections submitted for this oversight have been
approved--whether adequately supported by technical and business case
justification or not--the ASD C3I oversight has essentially become a
meaningless process.
--------------------
\1 The PSA is the senior executive-level manager who is responsible
for the management of a defined function or functions within DOD.
\2 Requirements for selecting and reviewing migration systems under
DOD's accelerated migration strategy are described in two
memorandums. The first memorandum was issued by the Deputy Secretary
of Defense in October 1993, and the second was issued by the
Assistant Secretary of Defense for Command, Control, Communications
and Intelligence in November 1993.
\3 Defense's technical and data standards are designed to enable
systems to easily interoperate and transfer information. Its
standard definitions for data elements are intended to ensure that
users of all Defense systems define the same data in the same way and
have a common understanding of their meaning. Defense has developed
or is in the process of defining technical standards in the Technical
Architecture Framework for Information Management (TAFIM), the Joint
Technical Architecture (JTA), and the Defense Information
Infrastructure Common Operating Environment (DIICOE). The Defense
Information Systems Agency (DISA) is responsible for developing,
obtaining from commercial sources, and maintaining the compilation of
Defense Information Infrastructure technical standards, and it is
responsible for maintaining a Defense data dictionary system as a
repository of data requirements and for facilitating the
cross-functional coordination and approval of standard formats,
definitions, etc. PSAs, the military services, Defense agencies, and
Joint Chiefs of Staff are responsible for reaching agreement on the
standards and approving them as DOD standard data elements. DISA is
then responsible for disseminating the approved standard data
elements for use throughout the Department.
\4 Nine of the systems that were submitted for approval were
grandfathered into approval because development for them was well
underway at the time the accelerated migration system strategy began
in 1993.
\5 Forty-two of the systems that were approved by the ASD C3I as
migration systems did not follow DOD's migration system approval
process described in appendix II. Instead, the ASD C3I approved
these 42 Command and Control systems based on reviews by the Military
Communications Electronics Board.
ACQUISITION OVERSIGHT PROCESS
DID NOT ENSURE SYSTEM
SELECTIONS WERE ECONOMICALLY
AND TECHNICALLY SOUND
---------------------------------------------------------- Chapter 2:2
Because the ASD C3I oversight control over the migration strategy
played a marginal role in ensuring that sound management and
development practices are followed, we assessed whether Defense's
traditional major information systems acquisition oversight processes
were doing so for 43 of the major migration systems.\6 These review
processes are designed to assess whether projects are affordable and
financial and operational risks have been minimized.
Once under acquisition oversight, systems are normally reviewed and
approved at each of four milestones.\7 The reviews involve assessing
such matters as whether the proposed system is being developed in
accordance with Defense policies, procedures, and regulations;
whether the systems' program managers took steps to minimize the cost
of a new system by ensuring full and open competition; and whether
the program managers will effectively use advanced system design and
software engineering technology to minimize software and maintenance
costs.
However, these reviews have not been effective for the migration
effort. As discussed in the following section, MAISRC could not
provide sufficient assurance that the major migration systems are
economically justified and comply with Defense's technical and data
standards.
--------------------
\6 As noted earlier, there are 49 major migration systems. Six of
these are under the oversight of the Defense Acquisition Board (DAB).
This is Defense's senior-level forum for advising the Under Secretary
of Defense (Acquisition and Technology) on critical decisions
concerning major weapon systems and large-scale information systems,
principally in the communication, command, control, and intelligence
functional areas. Criteria for DAB oversight are (1) estimated
expenditures for research, development, test, and evaluation of more
than $355 million or (2) estimated procurement costs of more than
$2.135 billion. One difference between the DAB and MAISRC oversight
processes is that systems reviewed by DAB are required to have
independent cost estimates rather than economic analyses. According
to the DAB analyst responsible for coordinating oversight of these
systems, Program Analysis and Evaluation (PA&E) analysts performed
independent cost estimates for all six migration systems under DAB
oversight.
\7 The four milestones are: milestone 0, approval to conduct concept
studies; milestone 1, approval to begin a new acquisition program;
milestone 2, approval to enter engineering and manufacturing
development; and milestone 3, approval to produce, field, or deploy
the system.
INEFFECTIVE REVIEW AND
PREPARATION OF ECONOMIC
ANALYSES
-------------------------------------------------------- Chapter 2:2.1
A principal tool of acquisition oversight is the economic analysis
because it helps to ensure that the system chosen for development is
cost-effective. If done properly, it can enable reviewers to
determine whether DOD has sufficient funds in its budget to pay for
the system, that is, whether the system is affordable. It can also
reveal potential conflicts between available funding and the planned
schedule for deployments. To arrive at these conclusions, the
economic analysis establishes baseline life-cycle costs and benefit
estimates for the project and calculates the project's return on
investment.
The importance of developing complete and accurate economic analyses
is underscored by several governmentwide requirements. For example,
the Office of Management and Budget's (OMB) Circular A-130,
Management of Federal Information Resources, calls on agencies "to
conduct benefit-cost analyses to support ongoing management oversight
processes that maximize return on investment and minimize financial
and operational risks for investments in major information systems on
an agencywide basis." Likewise, OMB's Circular A-11, Part 3, Planning
Budgeting and Acquisition of Fixed Assets, (July 16, 1996), and its
Bulletin No. 95-03, Planning and Budgeting for the Acquisition of
Fixed Assets, state that "the planning for fixed asset acquisitions
should be based on a systematic analysis of expected benefits and
costs."
However, even though economic analyses play a critical role in
assessing whether system development efforts will be cost-effective
and beneficial, Defense has not yet established a set of minimum
standards that an economic analysis must meet to be considered valid.
In addition, it has not published official guidance for preparing an
economic analysis. PA&E developed and published an unofficial
economic analysis guide that recommended, but did not require,
standard methods and formats for preparing an economic analysis for
an information system development/modernization project. According
to a PA&E representative, this unofficial guide is no longer adequate
because it does not require the degree of standardization in
methodology and analytic techniques needed to support a portfolio
management approach for managing information technology investments,
as required by the Clinger-Cohen Act of 1996. For example, the guide
does not require that returns on investment for systems
development/modernization projects be calculated in a standard manner
using a standard definition. This lack of standardization results in
economic analyses for different systems that are not comparable
enough for DOD managers to have a good basis for deciding which
systems offer the highest payoffs. The PA&E official stated that in
conjunction with DOD's implementation of a portfolio management
approach, it plans to officially publish appropriate documents and
provide minimum standards and guidance for the economic analyses
process.
Our review also found that DOD decisionmakers do not view economic
analyses as key tools for deciding whether to invest in an
information system development or modernization project. As a
result, DOD often lacks complete and accurate information on system
development/modernization projects' estimated costs and benefits at
the time decisions are made to invest in the projects. Specifically,
as the following examples show, we found that (1) economic analyses
for many systems have not been submitted for independent review and
(2) a significant portion of those that were submitted were
inadequately prepared. Thus, many of the benefits that could be
derived from this tool have not been realized.
-- Twelve of the 43 major migration systems have not yet submitted
an economic analysis, or an update of a previously prepared
economic analysis, for independent review.\8 These systems were
under development or modernization and DOD had invested hundreds
of millions of dollars in them in total. These included 5
systems that were under direct MAISRC oversight and 7 systems
for which MAISRC delegated oversight responsibility to a service
or agency. These 12 systems, or components of them, are all in
the second development phase or beyond.\9 Delaying the
preparation or updating of a previously prepared economic
analysis to the later stages of development for these 12 systems
defeats the purpose of the economic analysis, which is to
demonstrate that a proposal to invest in a new system is valid
before that investment is made.
-- Four systems that were under direct MAISRC oversight, were in
the first development phase--concept exploration--and were,
therefore, not yet required by DOD's acquisition regulations to
submit an economic analysis or an update of a previously
prepared economic analysis for independent review. Although
these four systems were technically in compliance with Defense's
acquisition regulations, DOD had already made major investments
in them without the benefit of knowing whether returns on
investment are going to be acceptable.
-- Even though Defense has not established standards for the
required analyses, Program Analysis and Evaluation (PA&E) staff
who review the economic analyses that are under direct MAISRC
oversight told us that 10 of the 19 economic analyses reviewed
had problems.\10
These problems included the following:
Understating or omitting the costs of standardizing data,
implementing the standard data in the systems, and developing system
interfaces.
Failing to estimate the amount of savings expected for terminating
duplicative legacy systems.
Relying on professional judgment to make unsupported assumptions
rather than making objective analyses to estimate the value of
benefits and costs. For example, in one case, the economic analysis
estimated a cost avoidance associated with replacing an old system by
assuming that both the old and the replacement systems' software
maintenance costs could be accurately estimated using two different
rates per line of code, but the analysis provided no data supporting
the validity of either rate.
After identifying problems with these 10 economic analyses, PA&E
staff worked with MAISRC analysts and the systems' program managers
to address the problems. However, DOD continued to develop the
systems in spite of the fact that the investments had not been
justified by complete and accurate economic analyses.
PA&E analysts told us that there were other problems that impeded
their review and verification of the economic analyses. For example,
economic analyses are often not updated and provided to PA&E for
review after major changes occur in the project, such as significant
cost growth or redirection of the project. A second problem is that
economic analyses are often not supported by analyses of alternatives
that weigh the cost and benefits of various technical options, such
as whether to buy commercial off-the-shelf software or develop a
system in-house.\11 This analysis would help Defense decisionmakers
make sound decisions on whether the proposed alternatives offer
sufficient military or economic benefits to be worth their cost, and
to determine which alternative is the best approach. It would also
identify alternatives that DOD may want to reconsider at a later time
if the selected approach runs into difficulties.
--------------------
\8 Of the 43 major migration systems under MAISRC review, 27 were
under direct review by MAISRC and 16 were delegated to other Defense
oversight organizations, with MAISRC retaining responsibility for
ensuring adequate oversight.
\9 The responsibility for independently reviewing economic analyses
for the five systems that were under direct MAISRC oversight rests
with DOD's Program Analysis and Evaluation staff, while the
responsibility for reviewing economic analyses for the seven
delegated systems rests with the various Defense organizations to
which MAISRC delegated oversight responsibility.
\10 We did not identify concerns or problems with the economic
analyses for the remaining 8 of the 43 major migration systems that
were under direct MAISRC oversight for which MAISRC had delegated
oversight responsibility to another DOD organization.
\11 The analysis of technical alternatives would normally precede the
economic analysis. In Defense, an economic analysis weighs the
costs, benefits, and risks associated with maintaining the status quo
versus the chosen technical solution.
DOD'S ACQUISITION OVERSIGHT
ORGANIZATIONS LACK ASSURANCE
THAT MAJOR SYSTEMS COMPLY
WITH TECHNICAL AND DATA
STANDARDS
-------------------------------------------------------- Chapter 2:2.2
Defense has established several sets of standards that are designed
to ensure that systems developed are compatible with its
communications and computing infrastructure and that they are
technically interoperable with each other. Some Defense leaders
consider systems interoperability and the ability to exchange data
across functional lines to be the most important consideration in
migration system development, transcending economic benefits. These
standards include the Technical Architecture Framework for
Information Management (TAFIM), Defense Information Infrastructure
Common Operating Environment (DIICOE), and DOD standard data.\12 DOD
system acquisition directives call on MAISRC and DAB to ensure that
program managers comply with the Department's policies and procedures
and use best practices in developing and modernizing individual
information systems. These best practices include building systems
and databases that comply with applicable technical standards and use
DOD standard data.
However, MAISRC and DAB do not have adequate assurance that the major
migration systems are complying with applicable technical
standards\13 and are using standard data. For example, out of 43
major systems under MAISRC oversight, program managers reported to
MAISRC that (1) only 19 were in compliance with the TAFIM standards
or had plans to comply with the TAFIM standards and (2) 9 systems
were using DOD standard data or had plans to use standard data.\14 We
found similar results for compliance with DIICOE standards.
Specifically, program managers reported that only 25 systems were in
compliance or had plans to be in compliance with DIICOE. These
self-reports by program managers are questionable because they are
not independently verified.
In addition to obtaining information on technical and data standards
directly from program managers, MAISRC and DAB also obtain such
information from the Defense Information Systems Agency (DISA). DISA
supports MAISRC and DAB oversight of major systems by performing
technical reviews of system documentation to determine if it
indicates that interoperability issues are being addressed or will be
addressed as the system is developed. According to a DISA
representative, DISA reviewed system documentation that had been
prepared for 37 of the 43 MAISRC migration systems and all 6 of the
DAB systems and found that the documentation indicated
interoperability issues were planned to be addressed for each of
them.
DISA also provides feedback, advice, and assistance on
interoperability and related issues during integrated product team
meetings with MAISRC, DAB, and system program managers. During these
meetings and on other occasions when it is asked to do so, DISA
provides MAISRC and DAB general information on compliance with
standards, such as whether the major systems' program managers have
contacted DISA and appear to be making reasonable attempts to bring
their systems into compliance with applicable standards. DISA also
assists program managers in developing strategies and cost estimates
for achieving compliance with standards.
However, DISA does not regularly report detailed information to
MAISRC and DAB that would enable these oversight organizations to
ensure that each major system is adequately complying with applicable
technical and data standards.\15 For example, DISA does not regularly
report to MAISRC and DAB such detailed information as (1) each
system's current compliance with applicable standards--that is, each
system's current status relative to the eight levels that DOD has
defined for the DIICOE, and each system's number and percentage of
data elements that have been approved as DOD data standards in the
Defense Data Dictionary System, (2) whether each program office
prepared sound strategies, schedules, and cost estimates for
achieving compliance with applicable standards, (3) whether each
system's compliance strategy and cost estimate were approved by DISA,
(4) each system's current schedule and cost status for achieving
compliance compared with its baseline schedule and cost estimate, and
(5) whether each system has been independently certified by DISA's
Joint Interoperability Testing Command to be in compliance with the
technical and data standards. MAISRC analysts responsible for
overseeing the major migration systems confirmed that they did not
know many of the systems' current status regarding compliance with
applicable technical standards and use of DOD standard data, or
whether the program managers had developed and were following sound
strategies for bringing the systems into compliance.
The technical and data standards are supposed to help pave the way to
an interoperable systems environment. However, without complete and
accurate data on individual systems' compliance with standards,
MAISRC and DAB cannot assure Defense's Chief Information Officer that
the Department's major information systems are complying with the
standards. Without this information, as well as standards compliance
information from the managers of DOD's nonmajor systems, the CIO
cannot gauge the Department's progress toward achieving its goal of
an interoperable systems environment. Additionally, this information
is critical if the CIO is to successfully implement an integrated
technical architecture for the Department as required by the
Clinger-Cohen Act of 1996.
--------------------
\12 See footnote 3 for more information on technical and data
standards.
\13 All DOD technical standards may not apply to all systems and
initiatives. For example, those standards that apply to application
software and data would not apply to a system or initiative that
included only computer equipment or hardware. To illustrate,
Defense's High Performance Computing Modernization Program initiative
largely involves the purchase of computer equipment and not the
development of software. Therefore, application software-related
standards may not apply.
\14 In addition, information available to DAB indicated that all six
migration systems under DAB program managers have already prepared
plans to bring their systems into compliance with DOD technical
standards.
\15 A DISA representative stated that program managers are supposed
to input data into DOD's DIST database that could be used by MAISRC
and DAB to track each individual system's progress in complying with
DIICOE. For example, the representative noted that program managers
are supposed to input compliance data into DIST, including (1) their
systems' current level of compliance with DIICOE, (2) information on
their strategies for achieving compliance, and (3) the expected date
and cost to achieve compliance. However, as discussed in appendix
IV, our analysis of the data in the DIST database showed they are
incomplete and inaccurate. Until DOD corrects these deficiencies,
MAISRC and DAB cannot rely on DIST for tracking compliance with
standards for individual systems.
BETTER OVERSIGHT COULD HAVE
HELPED TO PREVENT PROBLEMS
IDENTIFIED IN PREVIOUS GAO
REVIEWS
---------------------------------------------------------- Chapter 2:3
The lack of rigorous oversight for the migration strategy has
increased the risk that Defense will pursue flawed system strategies.
In fact, our previous reviews of migration systems identified a
number of problems that could have been prevented had there been
better oversight by MAISRC and the ASD C3I. For example, in several
reviews, we found that functional areas did not account for various
categories of significant costs when making their migration
decisions. These findings are highlighted as follows.
-- In reviewing the depot maintenance standard system migration
effort, we found that Defense did not address the full costs of
developing interfaces needed to allow system components to
exchange data with information systems currently used by the
services to accomplish their missions. One official estimated
that this represented $70 million in costs.\16
-- In analyzing what it would cost to develop its Standard
Accounting and Reporting System (STARS), we reported that
Defense neglected to consider internal project management costs
and costs to enhance all of the STARS components to bring them
into compliance with DOD's standard general ledger, key
accounting requirements, and the standard budget and accounting
classification code.\17
-- In reviewing the transportation migration effort, we found that
Defense did not include all costs associated with its evaluation
of in-house systems when analyzing costs and savings for its 28
migration systems. These included $16 million for its analysis
of candidate migration systems and $2 million for maintaining
migration system hardware. We also found that if these costs
were included in its systems selection analyses, Defense would
have found that the overall return on investment would have
decreased and that it may actually lose money on its
investment.\18
In previous reviews, we also found that Defense functional areas did
not adequately consider other alternatives to developing systems
in-house. For example, while the transportation area reviewed
commercial off-the-shelf transportation software projects for some
transportation business areas, this review was inadequate because it
did not (1) analyze the degree to which unmodified software could
meet unique Defense requirements, (2) identify the expected cost to
make necessary software modifications, (3) determine the time
required to make the modifications, and (4) provide for a hands-on
view of the software in operation. In addition, Defense concluded
that software packages that could provide some degree of
transportation functionality would require modifications that were
too costly. However, Defense could not provide documented analysis
to support this conclusion. Further, Defense planned on making $13
million worth of software modifications to just five of its in-house
selections. We believe better oversight by the CIO (then referred to
as the Senior Information Management Official) may well have forced
greater consideration of commercial packages in the transportation
area.
Finally, our review\19 of Defense's effort to develop a standard
suite of migration systems for materiel management showed that the
Department spent hundreds of millions of dollars without achieving
the expected benefits because it did not adequately anticipate and
mitigate risks. From 1992 to late 1995, Defense spent about $714
million developing standard systems with minimal results. During
that time, there were dramatic changes in the goals and expectations
for the program and only one application was partially deployed.
Because of changes in objectives and scheduling and problems in
development, prospects for achieving the original objective of
implementing a standard suite of integrated materiel management
systems appeared dim.
In 1996, Defense finally abandoned its strategy to develop a standard
suite of materiel management systems because of funding cuts, cost
overruns, and schedule delays and embarked on a new strategy that
involved individual deployment of nine system applications at
selected sites as the applications were developed. However, the
decision to drastically change the course of the strategy was
initiated without first conducting critical economic and risk
assessments that would estimate the costs, benefits, and risks of
alternative strategies and having the analyses independently reviewed
by MAISRC and PA&E.\20 The need for department-level review of
analyses supporting this decision was important, given the fact that
the new strategy represented a departure from Defense's goal of
eliminating redundant legacy systems and varied business
processes.\21
--------------------
\16 Defense Management: Selection of Depot Maintenance Standard
System Not Based on Sufficient Analyses (GAO/AIMD-95-110, July 13,
1995).
\17 DOD Accounting Systems: Efforts to Improve System for Navy Need
Overall Structure (GAO/AIMD-96-99, September 30, 1996).
\18 Defense Transportation: Migration Systems Selected Without
Adequate Analysis (GAO/AIMD-96-81, August 29, 1996).
\19 Defense IRM: Critical Risks Facing New Materiel Management
Strategy (GAO/AIMD-96-109, September 6, 1996).
\20 The materiel management standard system was originally projected
to cost $5.3 billion to develop and deploy. The threshold for major
information system projects is $360 million in total life-cycle
costs.
\21 Under the revised strategy, the military services would be
allowed to keep their legacy systems longer than anticipated, and
some legacy systems would not be shut down.
DEFENSE LACKS VISIBILITY OVER
MIGRATION EFFORT
---------------------------------------------------------- Chapter 2:4
A successful information technology investment process cannot operate
without accurate, reliable, and up-to-date data on project costs,
benefits, and risks. It is the basis for informed decision-making.
However, in implementing the migration strategy, Defense neglected to
provide visibility over progress and costs. The absence of reliable
and accurate performance, cost, and schedule information on the
migration effort has been debilitating in several respects. First,
it has impaired the Department's ability to demonstrate whether the
migration strategy has been successful. Second, it has kept Defense
from providing the Congress\22 with accurate and reliable information
needed for oversight purposes. Third, it has prevented Defense from
providing its own senior managers with information needed to oversee
the migration effort.
--------------------
\22 For example, Defense reported to the Congress that it had
selected 363 migration and interim systems, using DIST as the source
of the data. Defense also provided cost data that were collected by
the Office of the ASD C3I based on a different list of 245 migration
and interim systems. Since the ASD C3I's staff relied largely on an
official list of 245 systems that ASD C3I issued in July 1995 to
provide cost information to the Congress, that information was
incomplete. In addition, the ASD C3I's staff did not reconcile the
two lists of systems in its report to the Congress.
KEY PERFORMANCE ISSUES HAVE
NOT BEEN TRACKED
-------------------------------------------------------- Chapter 2:4.1
Since it embarked on the migration strategy, Defense has not tracked
overall departmentwide savings or validated improvements to
operations resulting from the migration strategy. In particular, it
has not been systematically tracking such key performance issues as
(1) administrative and operational cost savings resulting from
elimination of redundant systems, (2) cost reductions resulting from
improved information systems support to functional areas, (3)
management or staff productivity improvements, and (4) benefits
accruing to mission effectiveness that are attributable to
information technology support. Having this type of information is
the only means of ensuring that the billions of dollars being spent
on the migration are producing sufficient returns and achieving
departmentwide progress.\23 Without it, Defense cannot justify
whether the migration strategy has been a worthwhile investment or
support the need to continue pursuing migration.
At one point in the migration effort, as directed by the Congress,\24
Defense developed performance measures. However, it did not
regularly collect data on all the measures, and the accuracy of much
of the information it collected was questionable because of the data
integrity problems plaguing the database Defense relies on for
migration inventory and schedule-related information. In 1995, the
House Committee on National Security directed Defense\25 to
reevaluate its measures. In March of 1997, a Defense working group
proposed a revised set of performance measures relating to migration
systems and other information technology issues within the
Department. However, senior Defense management did not approve the
measures and instead tasked the working group to redo them to ensure
that they are linked to the Department's strategic information
technology management goals. The working group has not established a
schedule for completing this effort and finalizing the measures.
Having performance measures will help to validate individual reports
of migration successes, such as the migration systems characterized
as successful in Defense's report to the Congress pursuant to Section
381 of the National Defense Authorization Act for Fiscal Year 1995.
These are the Standard Procurement System, the Distribution Standard
System, the Defense Civilian Personnel Data System, and the Defense
Medical Logistics Standard Support System.
--------------------
\23 According to a DOD representative, the Department performed
intense reviews of migration systems during the last three cycles of
its planning, programming, and budgeting process. During these
reviews, DOD asked program managers and other senior managers to
provide performance information on their systems. However, the
representative stated that the performance information provided was
limited because DOD does not routinely track this information for its
information systems.
\24 National Defense Authorization Act for Fiscal Year 1995.
\25 H. Rep. No. 104-131, p. 161.
COST INFORMATION IS
INCOMPLETE AND MAY BE
SIGNIFICANTLY UNDERSTATED
-------------------------------------------------------- Chapter 2:4.2
We could not determine the full cost of the migration effort because
Defense's migration cost information is incomplete and may be
significantly understated. For example, when Defense provided
migration cost data to us in December 1996, it could only provide
costs for 242 of 363 migration systems. We did not include costs in
this report for 27 of 242 systems for which DOD collected costs
because the systems were classified. However, the remaining 121
systems were not included because the Office of the ASD C3I had not
collected the costs for these systems. When we obtained cost
information directly from three of the functional areas, we found
that at least $1.6 billion in costs had been excluded from the
approximately $18 billion total. This included about $1.4 billion
for clinical health systems, $56 million for civilian personnel
systems, and $111 million for transportation systems.
We also found that the $18 billion total cost estimate does not
account for some very important costs related to developing,
deploying, and maintaining migration systems both before and after a
project has been initiated. For example, the transportation
functional area has an office designated to oversee the development
and deployment of its migration systems. But, when accounting for
costs for the systems, DOD does not factor in the cost to maintain
this oversight responsibility. In addition, our previous reviews of
finance and logistics migration efforts have found that DOD did not
account for costs for these projects relating to such activities as
project management and developing interfaces with other systems. Our
detailed analysis of DOD's migration cost information is provided in
appendix III.
SCHEDULE INFORMATION IS
UNRELIABLE
-------------------------------------------------------- Chapter 2:4.3
We could not accurately determine how many legacy systems have been
terminated and how many are scheduled for termination because the
database Defense uses to track information systems is plagued with
data integrity problems.
Defense's own analyses of the database have shown that it cannot
readily provide simple descriptive information for many systems. For
example, a February 1997 DOD analysis of DIST showed that 55 percent
of the migration systems in the database had incomplete information
on interfaces with other systems and 77 percent had incomplete data
on installations where the systems operated.
Our analysis of DIST found that 61 percent the migration system
implementation dates and 32 percent of legacy system termination
dates were questionable. In addition, when we compared DIST
scheduling information to information maintained by three functional
areas we reviewed, we found that most of the migration systems
implementation dates and legacy system termination dates in DIST were
incorrect. For example, DIST showed that 92 legacy systems were
terminated by April 1996 in the clinical health, civilian personnel,
and transportation areas, while functional area managers told us that
only 43 had actually been terminated. Also, for the three functional
areas, DIST showed that 53 legacy systems were scheduled for future
termination while functional area managers told us that 91 were
slated for future termination.
We also found that Defense has not ensured that the data definitions
and formats used in DIST are fully compatible with data maintained in
other Defense information systems that track and report on systems.
While DOD is attempting to address this issue, DISA provided us
information showing that only 66 of the 218 data elements contained
in DIST have been approved as standard data elements in the Defense
Data Dictionary System. Without standard definitions and data
formats, data cannot be easily transferred to DIST from the other
systems that may be used by functional area managers and other
decisionmakers.
Defense officials acknowledge that DIST is incomplete and inaccurate,
and the Department has begun efforts to make the database more
accurate and more user friendly. However, Defense officials also
stated that they still used DIST to generate reports to the Congress
because it was the only departmentwide database containing schedule
and other descriptive data on all Defense migration and legacy
systems. A detailed analysis of DIST's integrity problems is
provided in appendix IV.
REFORMING DOD'S INFORMATION
TECHNOLOGY INVESTMENT PROCESS
REQUIRES EFFECTIVE IMPLEMENTATION
OF THE CLINGER-COHEN ACT
============================================================ Chapter 3
Defense has begun implementing Division E of the Clinger-Cohen Act of
1996 (Public Law 104-106), which the Congress passed in an effort to
put an end to problems associated with the development and
implementation of government information systems--such as those
evident in the migration effort. Under this legislation, Defense is
required to establish additional controls to ensure that more
attention is devoted to the selection, control, and evaluation of
information technology projects and that the Department's technology
investments are managed from a portfolio perspective.
Implementing the Clinger-Cohen Act in DOD can bring meaningful reform
to the management of migration and other information technology
investments. However, the current structure of the Chief Information
Officer (CIO) position in the Department will not permit the CIO to
devote full attention to reforming information resources management
within DOD. And it will continue to impose responsibilities on the
CIO that conflict with the CIO's obligation to provide independent
oversight over the development and implementation of information
systems.
THE CLINGER-COHEN ACT
---------------------------------------------------------- Chapter 3:1
The Clinger-Cohen Act recognizes that the key to successfully
managing information technology projects is ensuring that investment
processes provide for the continued identification, selection,
control, life-cycle management, and evaluation of information
technology investments. Best practices on which the act is based
have shown that to ensure an investment process is successful, top
executives need to periodically assess all major projects--proposed,
under development, and operational--then prioritize them and make
funding decisions based on factors such as cost, risk, return on
investment, and support of mission-related outcomes. Once projects
are selected for funding, executives need to monitor them
continually, taking quick actions to resolve development problems and
mitigate risks. After a project is implemented, executives should
evaluate actual versus expected results and revise their investment
management process based on lessons learned.
As our guide, Assessing Risks and Returns: A Guide for Evaluating
Federal Agencies' Information Technology Investment
Decision-making,\1 points out, a key to success in this type of
management is considering all the major technology investments that
are vying for funding at a designated level (departmental, functional
area, or service/agency level) as a total package, or portfolio, of
possible technology investments. Once this perspective is adopted,
an organization can focus scarce information technology resources on
the projects with the greatest impact on mission and concentrate
management attention on those high-impact projects that become
troubled projects. It can also establish performance goals and stop
or replace those systems or initiatives that fail to meet those
goals.
This type of decision-making process can be applied to almost any
organization, even one that is as highly decentralized as DOD.
According to our investment guide, separate information technology
investment decision-making processes can exist at various levels. In
DOD, such processes could exist at the departmental level and the
functional area or service/agency level, provided that the Department
can identify which major projects and initiatives should be managed
at each level. Criteria for determining projects that should be
managed at the various levels may include the dollar amount of the
investment, the degree of risk associated with the project, and
whether the system is to be shared across functional area lines or
service/agency lines.
The Clinger-Cohen Act contains the following additional requirements
that are important for DOD to implement in order to address problems
evident in its migration strategy.
-- Agencies are to determine whether their administrative and
mission-related business processes should be improved before
investing in major information systems to support them.
-- The investment process is to provide a means for senior
management to obtain timely information regarding progress (at
established milestones) in terms of cost, capability of the
system to meet requirements, timeliness, and quality. It should
also provide for the evaluation of the results of information
technology investments.
-- Performance measures are to be prescribed for information
technology used by or to be acquired for the agency.
-- The CIO is to monitor the performance of information technology
programs; evaluate the performance of those programs on the
basis of applicable performance measures; and advise the agency
head regarding whether to continue, modify, or terminate the
program or project.
-- The CIO is to be responsible for providing advice and other
assistance to agency heads and senior managers to ensure that
information technology is acquired and information resources are
managed for the agency in a manner that implements the policies
and procedures of the act and the priorities of the agency head.
-- The CIO is to develop, maintain, and facilitate the
implementation of a sound and integrated information technology
architecture for the agency. The architecture is an integrated
framework for evolving or maintaining existing information
technology and acquiring new information technology to achieve
the agency's strategic and information resources management
(IRM) goals.
--------------------
\1 GAO/AIMD-10.1.13 February 1997, Version 1.
DOD IMPLEMENTATION OF THE
CLINGER-COHEN ACT
---------------------------------------------------------- Chapter 3:2
As a first step in implementing the Clinger-Cohen Act, on June 2,
1997, the Secretary of Defense outlined his expectations for
improvements in information technology-related management processes
and information resources.\2 We believe the expectations identified
by the Secretary are an excellent starting point for implementing the
Clinger-Cohen Act and bringing meaningful change to the current
information technology management process. For example, the
Secretary has called on the CIO to design and implement a process for
maximizing the value and assessing and managing the risks of DOD
information technology acquisitions. This process is to
-- provide for the selection of information technology investments
to be made by the Department, the management of such
investments, and the evaluation of the results of such
investments;
-- be integrated with processes for making budget, financial, and
program management decisions;
-- include minimum criteria to be applied in considering whether to
undertake a particular investment in information systems,
including criteria related to the quantitatively expressed
projected net, risk-adjusted return on investment, and specific
quantitative and qualitative criteria for comparing and
prioritizing alternative information system investment projects;
-- identify, for each proposed investment, quantifiable
measurements for determining the net benefits and risks of the
investment; and
-- provide the means for senior managers to be able to obtain
timely information regarding the progress of an investment in an
information system, including the milestones for measuring
progress on an independently verifiable basis, in terms of cost,
capability of the system to meet specified requirements,
timeliness, and quality.
In addition, the Secretary has called on the CIO to institutionalize
performance-based and results-based management for information
technology. In doing so, the CIO is to work with the Chief Financial
Officer, the Principal Staff Assistants (PSAs), and the Defense
components. The CIO is to establish goals for improving the
efficiency and effectiveness of DOD operations and issue instructions
to functional areas on performance measurements. The CIO is also to
monitor the performance of information technology programs, evaluate
the performance of those programs on the basis of applicable
performance measurements, and advise the Secretary of Defense
regarding whether to continue, modify, or terminate programs or
projects. Additionally, the Secretary established a Chief
Information Officer Council for DOD to serve as the principal forum
for discussing improvements in DOD practices for the management of
information technology.
In outlining his expectations, the Secretary noted that the act poses
questions that should be answered before investing in information
technology, including:
-- What functions are we performing and are they consistent with
the mission?
-- If we should be performing particular functions, could they be
performed more effectively and at lower cost by the private
sector?
The Secretary further stated that if a function should indeed be
performed by the Department, the law requires that the function be
examined and redesigned or reengineered before applying new
technology.
--------------------
\2 Memorandum from the Secretary of Defense to the Secretaries of the
Military Departments, Chairman of the Joint Chiefs of Staff, Under
Secretaries of Defense, and others on the Implementation of
Subdivision E of the Clinger-Cohen Act of 1996, dated June 2, 1997.
CHALLENGES CONFRONTING DOD IN
IMPLEMENTING CLINGER-COHEN
---------------------------------------------------------- Chapter 3:3
Proper implementation of the Clinger-Cohen Act would help to address
a number of weaknesses that are currently standing in the way of
Defense's ability to provide a good decision-making and oversight
environment for information technology projects. For example, with
full implementation of the act, Defense could begin considering
information technology investments as a total package of possible
projects so that it can target resources to those projects having the
greatest impact on mission and concentrate management attention on
troubled areas. It could also strengthen visibility over project
performance, costs, and schedules so that senior managers can begin
comparing the results being achieved against projected, costs,
benefits, and risks and to identify actual or potential managerial,
organizational, or technical problems.
Moreover, in implementing the act, Defense has an opportunity to (1)
begin enforcing compliance with data and technical standards to
ensure that DOD's goals for interoperability and the sharing of
information are met and (2) increase oversight for functional area
assessments of whether to reengineer, migrate, or undertake some
other path toward improvement.
However, the current structure of the CIO position in Defense will
not permit the CIO to effectively serve as a bridge between top
management, line management, and information management support
officials and identify opportunities to use information technology to
enhance performance.
The Clinger-Cohen Act requires that information resources management
be the CIO's primary responsibility and that the CIO be involved in
key decisions regarding the application of information technology in
support of the agency's missions. Currently, the Assistant Secretary
of Defense for Command, Control, Communications and Intelligence (ASD
C3I) also acts as the CIO. By asking the CIO to also shoulder a
heavy load of programmatic responsibility, Defense has made it
difficult, if not impossible, for the CIO to devote full attention to
IRM issues. These issues go well beyond the development and
modernization of information systems.
For example, Defense needs its CIO to be heavily involved with
implementing a more aggressive and proactive computer security
program. As we reported in May 1996,\3 attackers have seized control
of entire Defense systems, many of which support critical functions,
such as weapons systems research and development, logistics, and
finance. Attackers have also stolen, modified, and destroyed data
and software. In addition, Defense needs its CIO to help ensure that
Year 2000\4 corrections are made to all of DOD's information systems.
If systems are not corrected on time, the impact on Defense
operations could be widespread, costly, and debilitating to important
warfighting and administrative operations. While DOD has delegated
year 2000 responsibility to its components, it still needs to ensure,
at the department level, that sufficient priority and resources are
being devoted to the problem and that all systems have been
identified and corrected.\5
There is also a direct conflict of responsibilites between the
oversight and programmatic obligations associated with the two
positions. The ASD C3I serves as the Principal Staff Assistant for
command, control, communications, and intelligence systems. These
systems represent about 45 percent of the migration system
investment--about $8.5 billion of the total $18 billion migration
investment. This poses a conflict for both control mechanisms we
assessed. For the first control--the ASD C3I approval process--the
same individual is responsible for both selecting a system and
approving the selection. For the second control--acquisition
oversight--Defense's Acquisition Executive is also responsible for
developing 14 of the 43 major automated information systems under the
Major Automated Information System Review Council (MAISRC).\6
A second challenge confronting DOD in implementing the Clinger-Cohen
Act is the prevailing organizational structure and embedded culture
found throughout the Department. Specifically, the three military
services have clearly defined roles and responsibilities and separate
budget authority, program execution, and functional authority for the
enforcement of national defense policy and objectives. As we have
reported\7 throughout the CIM initiative, this environment has
promoted stovepipe systems solutions in each component agency and has
made it difficult to implement departmentwide oversight or visibility
over information resources. This same condition has contributed to
the difficulty that has limited the Department in modernizing
business processes and implementing corporate information systems
across service and agency lines. This is most evident in the
perceived failure of the Corporate Information Management initiative
(CIM), which was intended to reengineer business processes throughout
the Department. In doing so, the Department expected to save
billions by having more efficient, effective business processes
running across service and component lines. However, these benefits
have yet to be widely achieved after 8 years of effort. Without the
Secretary's strong and continued support for management processes and
controls designed to improve information management initiatives,
Clinger-Cohen Act implementation could well suffer similar results.
--------------------
\3 Information Security: Computer Attacks at Department of Defense
Pose Increasing Risks (GAO/AIMD-96-84, May 22, 1996).
\4 The Year 2000 problem is rooted in the way dates are recorded and
computed in automated information systems. For the past several
decades, systems have typically used two digits to represent the year
in order to conserve on electronic data storage and reduce operating
costs. With this two-digit format, however, the year 2000 is
indistinguishable from 1900, 2001 from 1901, etc. As a result of
this ambiguity, system or application progr