[House Hearing, 111 Congress]
[From the U.S. Government Printing Office]
[H.A.S.C. No. 111-63]
MEASURING PERFORMANCE:
DEVELOPING GOOD
ACQUISITION METRICS
__________
HEARING
BEFORE THE
PANEL ON DEFENSE ACQUISITION REFORM
OF THE
COMMITTEE ON ARMED SERVICES
HOUSE OF REPRESENTATIVES
ONE HUNDRED ELEVENTH CONGRESS
FIRST SESSION
__________
HEARING HELD
MAY 19, 2009
[GRAPHIC(S) NOT AVAILABLE IN TIFF FORMAT]
U.S. GOVERNMENT PRINTING OFFICE
50-297 WASHINGTON : 2010
-----------------------------------------------------------------------
For sale by the Superintendent of Documents, U.S. Government Printing
Office Internet: bookstore.gpo.gov Phone: toll free (866) 512-1800; DC
area (202) 512-1800 Fax: (202) 512-2104 Mail: Stop IDCC, Washington, DC
20402-0001
PANEL ON DEFENSE ACQUISITION REFORM
ROBERT ANDREWS, New Jersey, Chairman
JIM COOPER, Tennessee K. MICHAEL CONAWAY, Texas
BRAD ELLSWORTH, Indiana DUNCAN HUNTER, California
JOE SESTAK, Pennsylvania MIKE COFFMAN, Colorado
Andrew Hunter, Professional Staff Member
John Wason, Professional Staff Member
Alicia Haley, Staff Assistant
C O N T E N T S
----------
CHRONOLOGICAL LIST OF HEARINGS
2009
Page
Hearing:
Tuesday, May 19, 2009, Measuring Performance: Developing Good
Acquisition Metrics............................................ 1
Appendix:
Tuesday, May 19, 2009............................................ 23
----------
TUESDAY, MAY 19, 2009
MEASURING PERFORMANCE: DEVELOPING GOOD ACQUISITION METRICS
STATEMENTS PRESENTED BY MEMBERS OF CONGRESS
Andrews, Hon. Robert, a Representative from New Jersey, Chairman,
Panel on Defense Acquisition Reform............................ 1
Coffman, Hon. Mike, a Representative from Colorado, Panel on
Defense Acquisition Reform..................................... 2
WITNESSES
Fitch, David P., USN (Retired), Director, AT&L Leadership
Learning Center of Excellence, Defense Acquisition University.. 7
Nussbaum, Dr. Daniel A., Visiting Professor, Department of
Operations Research, Naval Postgraduate School................. 9
Patterson, J. David, USAF (Retired), Executive Director, National
Defense Business Institute, University of Tennessee............ 5
APPENDIX
Prepared Statements:
Andrews, Hon. Robert......................................... 27
Coffman, Hon. Mike........................................... 28
Dillard, John T., Senior Lecturer, Graduate School of
Business and Public Policy, Naval Postgraduate School...... 77
Fitch, David P............................................... 57
Nussbaum, Dr. Daniel A....................................... 66
Patterson, J. David.......................................... 32
Documents Submitted for the Record:
[There were no Documents submitted.]
Witness Responses to Questions Asked During the Hearing:
Mr. Andrews.................................................. 83
Questions Submitted by Members Post Hearing:
[There were no Questions submitted post hearing.]
MEASURING PERFORMANCE: DEVELOPING GOOD ACQUISITION METRICS
----------
House of Representatives,
Committee on Armed Services,
Defense Acquisition Reform Panel,
Washington, DC, Tuesday, May 19, 2009.
The panel met, pursuant to call, at 8:00 a.m., in room
2212, Rayburn House Office Building, Hon. Robert Andrews
(chairman of the panel) presiding.
OPENING STATEMENT OF HON. ROBERT ANDREWS, A REPRESENTATIVE FROM
NEW JERSEY, CHAIRMAN, PANEL ON DEFENSE ACQUISITION REFORM
Mr. Andrews. Good morning, ladies and gentlemen. The panel
will come to order. Appreciate your attendance. I especially
want to thank the witnesses for their very diligent
preparation. We have had the opportunity to review the
testimony ahead of time. It looks like we are going to have a
very engaging and meaningful discussion this morning.
This is the last of our series of hearings looking at the
first question that we are going to be looking at in our
inquiry. The members will recall that we are proceeding on a
series of questions, the first of which is, ``Can we design a
series of metrics that accurately measure the difference, if
any, between the price paid by the taxpayers and the value
received by the taxpayers and the warfighters for the systems
and services that we are buying?''
We are going to proceed after today's hearing with our
second category of questions, which is really hypotheses as to
what has gone wrong. We are going to have a series of panels
talk about their theories and analyses of why we have a
difference between the price paid and the value received.
The first of our two panels on this metrics question dealt
with the sort of measuring the orthodox algorithms that are
used. In the major weapons system side, we had a panel on that,
and on the services side, we had a panel on that.
The purpose of today's panel is to bring in some people who
we think perhaps look at the whole question through a different
prism. They are willing to give us some new perspectives
through which we can analyze the difference between the price
we pay and the value that we receive.
We have three witnesses with a wealth and breadth of
experience in the acquisition field, but also who I think
benefit from what I would call a healthy distance from the
daily responsibilities for that function, so they can give us a
perspective that marries experience with a fresh perspective.
And hopefully, we will be able to use the testimony of these
witnesses to go forward, draw some conclusions about the best
metrics to use in our work, and then proceed with our work in
analyzing the various hypotheses given for why we have suffered
these cost overruns.
I also want to take a moment and thank Mr. Conaway, who
can't be with us this morning but who is ably represented by
Mr. Coffman, and all the members of the panel for their
excellent contribution to the work on the Weapons Acquisition
System Reform Through Enhancing Technical Knowledge and
Oversight (WASTE TKO) bill, which has passed the House. We will
be meeting with the Senate in formal conference at 4:30 this
afternoon to, I believe, put the finishing touches on that
conference report so we will be able to proceed to the floor in
short order on that, get the bill on the President's desk. Each
member of the panel has made a very significant and welcome
contribution to that effort.
Having said that, as the chairman and Mr. McHugh have said,
we think that that effort covers about 20 percent of the
landscape. It looks at major weapons acquisition, which
certainly needed to be examined. But there are so many other
areas that fall outside of that that the panel has to pursue,
as well as, frankly, reviewing the early stage implementation
of the law that we believe the President will sign this week.
So we are by no means at the conclusion of our work. We are
really at the outset of it. And one thing we would ask the
witnesses to do this morning is to think about the fact that,
although the Congress is about to pass major legislation
dealing with major weapons systems, we have not yet addressed
hardware that doesn't fall into the major weapons system
category and the 55 to 60 percent of acquisition that is
services that we have to look at on behalf of those who wear
the uniform and on behalf of the taxpayers.
We are glad that you are here. And at this time, I want to
turn to Mr. Coffman for his opening statement, and we will
proceed with the witnesses after that.
[The prepared statement of Mr. Andrews can be found in the
Appendix on page 27.]
STATEMENT OF HON. MIKE COFFMAN, A REPRESENTATIVE FROM COLORADO,
PANEL ON DEFENSE ACQUISITION REFORM
Mr. Coffman. Good morning. Mr. Chairman, ladies and
gentlemen, I would like to extend a welcome on behalf of
Ranking Member Conaway, as well. He is sorry he could not be
with us today. I would like to thank the chairman for allowing
me to take a few introductory remarks in Representative
Conaway's place.
Today's hearing is an appropriate follow on to the panel's
two hearings which focus on how the Department of Defense (DOD)
and the Government Accountability Office (GAO) currently assess
performance on major weapons systems and service contracts. The
purpose of this hearing is to think outside the box on how we
should measure performance and about what we should be
measuring and less about the how we are doing it today.
Our previous hearings revealed that current measures of
performance tend to break down if the program baseline is
unrealistic. We want to know if the program can be corrected--
if the problem can be corrected. Furthermore, are there metrics
beyond cost and schedule performance that are of value, such as
how closely does the delivery system meet actual warfighter
needs. Does the time of delivery of operational capabilities
satisfy warfighter needs? How do we determine whether optional
or tradable capabilities requested by the warfighter are
affordable?
Mr. Chairman, we have a distinguished panel of witnesses in
front of us today. I have looked at their written testimony,
and I look forward to their testimony. There are a couple of
points I would like to highlight. Although Mr. Dillard is not
testifying today, he did submit a written statement that I
would like to briefly comment on.
[The prepared statement of Mr. Dillard can be found in the
Appendix on page 77.]
Mr. Coffman. He states that, ``The proliferation of
autonomous fiefdoms within the department continues to
increase, with each being a stovepipe of oversight expertise,
imposing unique reporting requirements, assessments and
reviews.''
He goes on to state that, in regards to adding additional
acquisition workforce professionals, that, ``These new people
should not be housed in the Pentagon, but instead where the
execution of programs occur.'' Mr. Chairman, I think this is an
important point that I believe we need to follow closely.
Another point of observation I would like to make is in
regards to requirements in joint programs. We have many
programs out there that have the word ``joint'' in front of
them, but they are joint in name only. I understand that the
current requirements generation process is called the Joint
Capabilities Integration and Development System, or JCIDS.
It seems to me that, in order to have a truly joint program
that the requirements for that program must be borne jointly.
Yet, I do not believe our current system, or as Mr. Dillard
describes, current fiefdoms, foster such an environment. So I
would be interested in hearing from our witnesses in this
regard.
I also encourage our witnesses to share their views on
existing laws and regulations that are particularly helpful or
not helpful to the Department's efforts to obtain the best
value and capability for our warfighters. We have heard from
the Department in two separate hearings about how they
currently measure performance and value on contracts. What we
haven't heard enough about is how they should be measuring
value. Consequently, your input will be greatly appreciated.
With that, I conclude. And again, thank you, fellow
members, and thank you, Mr. Chairman. I look forward to the
witnesses' testimony.
Mr. Andrews. Mr. Coffman, thank you.
Without objection, the statement of any other member of the
panel will be included in the record. And without objection,
the statement of each of the three witnesses, plus the
supplemental floor statement, will be included in the record of
the hearing.
[The prepared statement of Mr. Coffman can be found in the
Appendix on page 28.]
Mr. Andrews. I think each of you are veterans at this
process. You know that we generally have a five-minute rule
where we ask for succinct summaries of your written testimony.
We are going to be a little bit more liberal with that this
morning, but then we are going to turn as quickly as we can to
the questions from the panel and get into the give-and-take.
I am going to read a brief biography of each witness, and
then, Mr. Patterson, we are going to begin with you.
Mr. Dave Patterson is the Executive Director of the
National Defense Business Institute, which he is establishing
at the University of Tennessee in the College of Business
Administration. It is an institution inspiring business
innovation for both government and industry by providing
practical, sound assistance and creating economically efficient
and effective defense business and acquisition programs.
Prior to taking his current duties, he was the Principal
Deputy Under Secretary of Defense Comptroller. As the Principal
Deputy, he was directly responsible for advising and assisting
the Under Secretary of Defense with development, execution and
oversight of the DOD budget exceeding $515 billion, with annual
supplemental requests of more than $160 billion.
From August 2003 to June 2005, Mr. Patterson held duties as
the Special Assistant to the Deputy Secretary of Defense. In
that capacity, he was responsible for managing the Deputy
Secretary of Defense's personal staff, as well as providing
direction and advice to the Office of the Secretary of Defense
staff on a wide range of national security operations and
policy subjects.
He served in the Air Force from 1970 to 1993, retiring in
the rank of colonel. During that time, he held responsible
leadership and management positions, with assignments at the
air wing level as the C5-A aircraft Commander and Deputy
Operations Group Commander at Major Command Headquarters, U.S.
Air Force, the Office of the Chairman, Joint Chiefs of Staff,
the Office of the Secretary of Defense and Inspector General.
In 1986, he was the Air Force Fellow at the American Enterprise
Institute and served in Vietnam, flying 02-As as forward air
controller.
Thank you for your service to our country, Mr. Patterson.
Glad to have you with us.
Mr. Fitch, Mr. David P. Fitch enlisted in the Navy in 1966.
He was commissioned in 1968 after graduation from San Jose
State College and Aviation Officer Candidate School in
Pensacola, Florida.
In 1969, after fixed and rotary wing flight training, he
was designated a naval aviator and a distinguished naval
graduate. He retired from the Navy in 1998 as a captain,
following a career that included three operational and major
acquisition command assignments.
Mr. Fitch culminated his tour in the Navy as the major
program manager for the international and joint multi-
functional information distribution system. He was a 1997
recipient of both the David Packard Excellence in Acquisition
Award and the Department of Defense Value Engineering Award.
In 2001, after nearly three years in the defense industry,
Mr. Fitch joined the faculty of the Defense Acquisition
University, teaching and consulting acquisition executives and
program management and other acquisition disciplines.
In 2006, he led a major independent study of the Coast
Guard Deep Water program. In July of 2008, he assumed the
position of Director, Acquisition, Technology and Logistics
(AT&L) Leadership Learning Center of Excellence after nearly
seven years as the Dean of the Defense Systems Management
College.
Mr. Fitch holds degrees in business and industrial
management from San Jose State college, an M.S. in education
from the University of Southern California, and he is a
graduate with highest distinction from the Naval War College in
Newport, Rhode Island. Thank you, Mr. Fitch, for your service
to our country, and we are glad you are with us this morning.
Dr. Daniel Nussbaum, from 2004 to the present, has been a
visiting professor at the Naval Postgraduate School operations
research department. From 1999 to 2004, he was a principal at
Booz Allen Hamilton, responsible for a broad range of costs,
financial and economic analyses with clients across the
government and commercial spectrum.
From 1996 to 1999, he was the director of the Naval Center
for Cost Analysis at the Office of the Assistant Secretary of
the Navy for Financial Management and Comptroller here in D.C.
From 1987 to 1996, he was division head of the Naval Center for
Cost Analysis. From 1982 to 1986, he was Deputy Director and
Acting Director for operations research and cost analysis
divisions of the Naval Air Systems Command in Washington, D.C.
Again, a very distinguished servant of our country, three
excellent people with experience and fresh insight. We are
happy to have you with us.
Mr. Patterson, we would like you to begin with your
testimony.
STATEMENT OF J. DAVID PATTERSON, USAF (RETIRED), EXECUTIVE
DIRECTOR, NATIONAL DEFENSE BUSINESS INSTITUTE, UNIVERSITY OF
TENNESSEE
Mr. Patterson. Thank you, Mr. Chairman, members of the
Defense Acquisition Reform Panel. I am very pleased to be here
this morning to participate in a discussion of a question that
has clearly captured the attention of the current
Administration and Congress: How should Congress assist the
Department of Defense in improving its acquisition of weapons
and services so that it can meet the needs of the warfighter in
the field while still being a good steward of the taxpayer's
dollars?
The first consideration for judging the success of an
acquisition program is whether it fielded a weapons system or
information system or service in time to make a positive impact
for the warfighter. A system or service fielded too late to
need may as well not have been bought at all. The phrase, ``too
little, too late,'' can mean lost lives.
But before we look at the measures of the acquisition
system merit, there is another consideration central to this
discussion. When Secretary Gates made his budget announcement
on April 6th, 2009, I believe he was speaking from frustration
that was as much about what has been the persistent problem
with the acquisition system that we depend on that is simply
not responsive to the immediate warfighter needs, as it was
about winnowing bloated, failed and unnecessary programs.
Implicit in that expression of frustration is a clear lack
of confidence in a system that actually produces programs with
uncertainty and instability. The most dramatic improvement
metric will be when the senior leadership in the
Administration, Congress and the Department of Defense have
seen such improvements, results, not words, that they can say
they have renewed confidence in the stability, predictability
and effectiveness of the defense acquisition system.
The Defense Acquisition Performance Assessment's Report
contended that program stability and predictability were
singularly and uniquely crucial to managing programs that were
on cost, on schedule and performing. To that end, in the time I
have, allow me to describe two areas of improvement for
measuring program effectiveness worthy of attention.
First, Major Defense Acquisition Programs, or MDAPs, often
start at milestone B, the beginning of engineering and
manufacturing development, with critical staff positions
vacant. Percentage of critical staff positions filled at
milestone B is an easy and important metric to be observed. It
makes little difference to implement programs to raise the
level of skills of the program staff if they are missing in
action.
Second, the acquisition strategy document, that is to lay
out how the weapon system is to be acquired, the initial
roadmap, if you will, is often flawed in that it focuses more
on presenting a case for required capabilities and quantities
than on laying out the reasoning for acquisition competition
methodologies.
For example, how the prime contractor participants in an
MDAP competition will select subcontractors and how the winner
of the competition will manage the subcontractors to gain
improved efficiencies and effectiveness are generally given
little consideration. Creation of the acquisition strategy
document is one of, if not the most, important tasks the
government acquisition program management can undertake.
The strategy should establish the template for all the
activities that will take place throughout the source selection
process, engineering manufacturing development, and follow-on
production and fielding. More important, it establishes how the
program management team is thinking about the numerous events
and activities that a program will encounter.
The defense acquisition executive should establish a common
set of strategy elements that all the military department
service acquisition executives must include in MDAP acquisition
strategy documents. Additionally, a set of standards or metrics
by which the strategy elements can be evaluated as effective
must be part of that process.
In closing, I would be remiss if I didn't acknowledge the
progress that has been made by the Department in improving the
acquisition system over the last four years. Though it is the
General Accountability Office's headline that the 96 major
acquisition programs have grown in cost by $296 billion that
gets attention, those numbers belie an equally worthy but
overlooked statistic published in the very same report.
The average increase in unit cost of the 28 MDAP programs
with less than 5 years since development started is only 1
percent. Compared with an average of 55 percent increase in
acquisition unit cost of 25 programs in the group with 5 to 9
years since program development start, there has been important
improvement that should be recognized.
With that, it is my privilege to be here. Thank you very
much, and I would be happy to take any questions that you have.
[The prepared statement of Mr. Patterson can be found in
the Appendix on page 32.]
Mr. Andrews. Thank you, Mr. Patterson.
Mr. Fitch.
STATEMENT OF DAVID P. FITCH, USN (RETIRED), DIRECTOR, AT&L
LEADERSHIP LEARNING CENTER OF EXCELLENCE, DEFENSE ACQUISITION
UNIVERSITY
Mr. Fitch. Chairman Andrews, Congressman Coffman, members
of the panel, thank you for the opportunity to appear here
today. I will address the subject of acquisition performance
metrics and your questions about how to increase the realism of
program baselines, making trades between affordability and
performance and how to assess the value of systems that do not
necessarily--please recognize that my opinions do not
necessarily reflect the views of the Defense Acquisition
University (DAU), the Department of Defense or the
Administration.
Mr. Andrews. That is why we are interested.
Mr. Fitch. I suspected as much.
Measurement of acquisition performance must encompass both
strategic and tactical elements. As emphasized in a recent
Defense Science Board report titled, ``Creating a DOD Strategic
Acquisition Platform,'' the management, execution and oversight
of acquisition programs is moot if we aren't spending taxpayer
dollars to buy the right capabilities if we aren't
demonstrating strategic choice. It is as important to decide
what capabilities we won't buy as well as what we will buy.
I believe one of the root causes of funding instability is
what I described as too many programs chasing too few dollars.
Too many programs chasing too few dollars is one of the root
causes, I believe, of overly optimistic cost estimates. The
recently implemented Materiel Development Decision (MDD)
process provides a framework for making strategic investment
decisions.
The MDD is the formal point of inquiry into the acquisition
process. MDD will increase integration of the three major
acquisition support systems, requirements, resources, and
acquisition.
Improving requirements management, an initiative supported
by the Congress, includes providing training to requirements
writers and managers to ensure that they have a sufficient
understanding of the critical elements of acquisition, such as
systems engineering and testing. The goal is to improve
collaboration between acquisition and requirements communities
to exploit cost and performance trades and improve acquisition
outcomes.
Having a formal requirements and acquisition process is not
unique to the DOD. We can learn important lessons from
commercial industry. If you compare the DOD acquisition system
with the process to develop electronic games, there are marked
similarities and differences.
Notably, the process to get games on the shelves for the
December holiday season starts with a precise clarity of what
will be developed by when, and includes a corporate commitment
to the resources required for the project. Precise clarity is a
result of intense interaction between the people that divine
the capabilities of the game and the people that will design
and test the software.
Turning to the subject of tactical acquisition metrics, the
most effective tools and templates incorporate metrics, both
quantitative and qualitative. The question was raised, ``Are
there metrics beyond cost and schedule performance that are of
value?'' The answer is yes, and an ongoing example is a
probability of program success metrics, POPS, that are
currently being developed and deployed across the services and
other federal agencies, such as the Department of Homeland
Security.
Starting with a blank sheet of paper, a group of DAU
faculty comprised of experienced program managers and other
functional experts ask themselves questions, including, ``What
conditions facilitate the success of programs? What metrics are
leading indicators of derailment?''
The resulting tool, POPS, uses a structured process to
continually assess and display key elements of planning,
resourcing, execution and external influences that promote or
negatively impact program success. Still evolving, POPS is
being used in the Army, Air Force, Navy, Marine Corps and Coast
Guard. Timely, accurate and transparent metrics integrated in a
management and oversight process, will produce better program
outcomes.
Another question that has been asked, ``Can we ensure
improved realistic baselines.'' Again, I believe the answer is
yes, and there are ongoing initiatives that will yield more
realistic baselines. These include increased emphasis on
technology readiness before starting major development,
emphasis on improved cost estimating, which I am sure that we
will talk about today, and competitive prototyping, which is
included in the new DOD 5000.2 and pending legislation.
Prototyping increases the opportunity to identify and
assess affordability and capability traits. Competitive
prototyping also allows the government to observe the
performance of competing industry teams before making a down-
select for engineering and manufacturing development. No matter
how thoughtfully we plan and discipline source selection, a
paper-only source selection process is never as good as
demonstrated performance.
The ultimate assessment of whether we have delivered value
and needed capability to the warfighter is feedback from the
field from the warfighter. At various times, I have seen photos
of messages written with felt-tip pens on trucks and personnel
carriers that have been returned to the government or industry
depots. One of those signed messages--a picture is in my
written testimony--reads, ``This truck saved my life, as well
as five others, 2-April-2008 at 2300 Lima, Basra, Iraq.'' Those
kinds of testimonies, those kinds of results, are what those of
us that are in the acquisition aim to achieve on a daily basis
by our efforts and commitment.
Before equipment is fielded, it undergoes rigorous levels
of development and operational testing, and the new DOD 5000
has increased emphasis on earlier testing.
Mr. Chairman, thank you very much for the opportunity to
participate in today's discussion. As the Secretary of Defense
has said, there is no silver bullet. It will be a combination
of initiatives and collaboration amongst all the stakeholders
to create an acquisition system that consistently produces
successful acquisition programs.
I look forward to your questions.
[The prepared statement of Mr. Fitch can be found in the
Appendix on page 57.]
Mr. Andrews. Mr. Fitch, thank you very much for your
testimony.
Dr. Nussbaum.
STATEMENT OF DR. DANIEL A. NUSSBAUM, VISITING PROFESSOR,
DEPARTMENT OF OPERATIONS RESEARCH, NAVAL POSTGRADUATE SCHOOL
Mr. Nussbaum. Mr. Chairman, distinguished members of the
panel, I would like to thank you for this opportunity to
discuss my thoughts on how to improve acquisition and cost
estimating processes in the Department of Defense. These ideas
are mine alone.
As the chairman said, I am a member of the faculty at the
Naval Postgraduate School in Monterey, California. And I have
spent the last 30 years mainly doing, and more recently,
teaching and researching in the defense acquisition management
system with a focus on cost estimating. I was a previous
director of the Naval Center for Cost Analysis and past
president of the Society of Cost Estimating and Analysis.
All my experiences in cost estimating confirm that three
things are necessary for sound cost estimating: Acceptance of
the underlying uncertainties in predicting the future, accurate
and plentiful historical data, and professionally trained and
certified personnel.
On uncertainty, there is intrinsic uncertainty in all
estimates. It derives from several sources, mainly that we are
usually designing or building or operating something that is
substantively different from what we did before. The difference
could be in the product or in the economic conditions or the
programmatic conditions.
An estimate reflects our knowledge at a point in time when
we freeze the problem and base the cost on the configuration
and programmatics as they are understood at that time. From
that baseline, many things can change that can also change the
cost estimate, including labor rates, overhead rates,
schedules, enhancements of the capabilities or quantities,
changes when a particular technical solution to a problem
doesn't work as planned. And we need an alternative technical
solution.
On data, a hallmark, a necessary characteristic of a sound
cost estimate, is that it is based on historical program
performance from similar or related ongoing or past programs.
Historical data is variable. Not every aircraft costs the same
as every other aircraft.
And the measurement of this variability is accomplished
through statistical constructs, things like standard error of
the estimate, things like confidence intervals. We assume in
our community of cost estimating that the patterns of the past
will repeat in the future. But these patterns are almost always
statistically grounded patterns modeled with the powerful and
subtle techniques known collectively as regression. Further, we
know of no alternative approach to using the past as a guide to
the future if we want a scientific that is a reproducible and
auditable approach.
Not all estimating is done by government employees. There
really are three sub-communities. One is the government in-
house estimators. One is employees of the large vendors who
also design, develop and build what we buy--Boeing, Northrop,
Lockheed Martin, for example. And thirdly, there are support
contractors or consultants to the government. Those are the
three communities.
And surely, we need to increase the capacity and the
quality, the numbers and the training of the government
estimators, but so do we need to enhance the professionalism of
the other two communities.
There are currently no undergraduate curricula in cost
estimating, and there are only four educational institutions
that I am aware of that teach at least one course in cost
estimating, and those are the Naval Postgraduate School in
Monterey, where I am; the Air Force Institute of Technology at
Wright-Patterson in Ohio; Defense Acquisition University,
diverse locations with a capital campus at Fort Belvoir; and
the Massachusetts Institute of Technology, which offers one
elective course in cost estimating within its engineering
curriculum.
The recent separation of the business cost estimating and
financial management career field into two separate cost
estimating and financial management tracks is a very welcome
development and should be supported, but note that DAU support
is largely limited to military and DOD, not the other two
communities.
The Society of Cost Estimating and Analysis--we say SCEA--
whose membership includes approximately one-third of all cost
estimators supporting DOD, is a central and indispensable
player in the training, initial certification and periodic
recertification of cost estimators. I note with pleasure that
the executive director of SCEA, Mr. Elmer Clegg, is in the
room.
SCEA has collected a body of cost estimating knowledge, and
it provides the members of the cost estimating community,
provides training in cost estimating, has developed and offers
an examination and experience-based certification program.
Lockheed Martin, Boeing, Northrop Grumman, other vendors use
SCEA's training and certification as their standard.
I appreciate very much what this committee seeks to
accomplish, Mr. Chairman. This concludes my prepared statement,
and I would be pleased to answer your questions.
[The prepared statement of Dr. Nussbaum can be found in the
Appendix on page 66.]
Mr. Andrews. Well, thank you, Dr. Nussbaum, and I thank
each of the three witnesses for their testimony. As I say, we
have had the chance to review the written testimony. We are now
going to get to the questioning phase.
Mr. Patterson, I was intrigued by your reference in your
written testimony to two kinds of requirements costs, which you
express as customer requirements and derived requirements. What
is the difference between the two?
Mr. Patterson. Customer requirements are established in key
performance parameters, which are requirements that are
established for a particular weapons system that the weapons
system then must perform against. Derived requirements are
those requirements that the customer did not ask for but that,
as the name would suggest, derive as a consequence of the
design process. ``Oh, look, we could do this better if only
we''--and of course, that is taken to the program manager.
The program manager will say, ``Okay, we can do that. Just
bring money.'' And they do.
Mr. Andrews. Now, I notice on page seven of your testimony
that you say that a recent study prepared by the Monitor
Company Group was based on selected acquisition report data,
estimates that approximately 33 percent of the cost growth from
2000, 2007--I think I read this--is attributable to this second
category of requirements that you are talking about, right, the
derived?
Mr. Patterson. That is correct. Yes, sir.
Mr. Andrews. Now, how is that number reached? Where did
that 33 percent come from?
Mr. Patterson. That comes from the selected acquisition
reports (SARs). In fact, they lay that out in the SARs and
explain in each of the SARs where the----
Mr. Andrews. Well, one of the things that we want to do is
to make sure that we discriminate between derived changes that
are beneficial and those that may be superfluous. How would you
suggest that we might do that? In other words, I don't want to
leave the impression that we are saying, or the report is
saying, that ``Oh, that 33 percent was waste.''
Mr. Patterson. Oh, absolutely not.
Mr. Andrews. And how do we draw the line between
beneficial, cost-effective derived requirements and not-so-
beneficial derived requirements?
Mr. Patterson. Well, let me give you a real-life example.
In 1993, I came to the C-17 program. It was a troubled,
problem-plagued program.
The two program managers decided that much of the problem
was that requirements were growing in an airplane. They
established a set of rules. The set of rules was very simple:
If you have a requirement or an engineering change, which is
effectively a derived requirement, then it must go to an
engineering change board.
The engineering change board will evaluate the change for
its intrinsic merit. But if it doesn't meet the safety of
flight or other driven requirements, then it has to have a
three-to-one payback in savings and not perturbate the
schedule.
Mr. Andrews. Interesting.
Mr. Patterson. Real simple.
Mr. Andrews. Now, in terms of the scope of that C-17
program, how much were the derived requirements overrun? In
other words, how much was attributable to derived requirements?
Mr. Patterson. I would only be guessing. I can certainly
get that for the record.
[The information referred to can be found in the Appendix
on page 83.]
Mr. Andrews. And with this method that you just described,
in your judgment, what percent of the derived requirements were
beneficial, and which failed to make that three-to-one cut and
didn't happen?
Mr. Patterson. Again, let me give you an example.
At Edwards Air Force Base, they were testing the airplane.
One of the requirements was that one airplane needed to start
another airplane, but it started it with two hoses from the
pneumatic system. Somebody said, ``I wonder if it will start
with one hose.'' It didn't. ``Oh, my gosh. Well, now we have to
go and figure out why that is the case.''
And so, that took a considerable amount of test time and
money, and as it turned out, they really wanted to have it
start with one hose. I am sorry, that would have taken----
Mr. Andrews. Who was the someone who said that? Not the
person's name, but where were they in----
Mr. Patterson. That was the test community out on the ramp
at Edwards.
Mr. Andrews. Right.
I am going to go back for a second round, but Mr. Fitch
mentioned this probability of success metrics program. Has
there been built into that a litmus test for a probability, if
you would fall below it, things stop? In other words, does that
program have in it a built-in go or no-go line?
Mr. Fitch. It does not have a go or no-go line. It is
information that is updated and provided----
Mr. Andrews. Do you think it should?
Mr. Fitch. I think that there needs to be an informed
review and decisions that look at each of the things that occur
that is negative, because some of the things that are in the
probability of success metrics and reporting are positive.
So I think that it needs to have the program manager, when
that information is put together on a monthly basis, needs to
look at that. It needs to be reviewed by someone in his Program
Executive Office (PEO), okay, and potentially even at higher
authority when there are negative occurrences.
Could I add something about--reporting?
Mr. Andrews. Sure, and then we are going to go to Mr.
Coffman. Sure.
Mr. Fitch. Yes.
I just wanted to explain that the derived requirement
process is actually part of the systems engineering process so
that if, for instance, the warfighter says in an aircraft, ``I
need a display that has color, I need a display that has this
amount of resolution,'' et cetera, the first pass, even by the
warfighter or the cost estimators, may say, ``This amount of
processing may be sufficient for that.''
When you get into, ``Well, by the way, you are going to
have these other software capabilities.'' You put those
together, and they start to build on one another, you can find
all of a sudden that the processor that was planned may be a
commercial off-the-shelf item with X amount of memory,
throughput, et cetera, is insufficient.
At that point is the point where it would be very useful to
have the requirements community have a real dialogue with the
acquisition community to say do I now take away some of the
requirements so I can continue to use that commercial off-the-
shelf processor that is less expensive, or are those
requirements really important now that we have figured out what
they are. So the point is the derived requirements is a very
important process, and it is actually the process where you
allocate where features will go to hardware or software.
Mr. Andrews. Thank you very much.
Mr. Coffman.
Mr. Coffman. Thank you, Mr. Chairman.
This is to all of you. It seems like part of the problem is
sometimes we are dealing with immature technologies, and I
think we have had previous testimony to that effect, that
sometimes we are asking the contractors to develop something.
Should we bifurcate the process? In other words, that you
are contracting with one entity to develop the technology, to
develop the--I don't know if you would go as far as a
prototype, and then where you can go into fixed cost production
with another--maybe that entity would be allowed to bid on it
as well. But is it better to bifurcate the process?
Mr. Nussbaum. You know, I would say we would do that now.
If we have open competition, we certainly have a research and
development (R&D) contract followed by an acquisition contract
with--there is certainly no guarantee that the R&D contractor
will have a follow on.
So in some sense, we do that, but in another sense, the
Department has now mandated technology readiness levels, TRLs
of certain levels before a program can get beyond a milestone.
They have narrowed that cone of uncertainty by saying you have
to have a TRL level six. They would prefer seven, and GAO
prefers seven, but you have to pass a TRL level of six before
you go into the next milestone.
Mr. Fitch. As a part of the analysis of alternatives, that
process that immediately follows it, the technologies that
would be appropriate for that system, that capability, are
assessed for technology readiness, when you get to the point of
saying who should develop it, I prefer the concept of the
competitive prototyping, because you are going to get proposals
from industry.
They are not going to all have the same strengths. Some are
going to view that I want to do this more in a hardware
function. Others may be more software intensive. There are
different technologies involved.
Their proposal, and then watching them deliver on what they
promise and they say is possible in competitive prototyping, is
key to then being able to transition and make an award to that
contractor who has proven that they did what they say they
could do. So having a different contractor, if I understood
your premise correctly, to develop technology and then award it
to somebody differently to develop the system, I think that is
not the intent of the competitive prototyping system.
Mr. Patterson. I think, too, there is a problem that arises
when we talk about spiral development. Originally, spiral
development, evolutionary development, were designed to have a
weapons system that effectively was fielded and then have block
upgrades to improve the weapons system.
In many cases, that is not how that works, because you do
have a parallel R&D program that is working on improvements.
And before the weapon is actually fielded, you have the attempt
to integrate improvements.
Well, the consequence of that often is that you have a
stretched-out program. Costs escalate. And in the end,
performance is degraded, and it takes a while for the--much
longer than had been anticipated for that weapons system to be
fielded. So I think we need to have very specific and clear
understood standards by which we will integrate or put in
upgrades or technology advances into weapons system. It
certainly can't be during engineering, manufacturing,
development.
Mr. Coffman. I think it was mentioned, and it has been
mentioned repeatedly, that the changes drive a lot of the cost
as they go forward. Where did the changes emanate from? I mean,
are some of the changes sort of broader in scope where the
affected branch who will receive and utilize the weapons
system, is it changes in evolving doctrine, or is it mostly at
a very technical level with this engineering requirements?
Mr. Nussbaum. You know, I think that changes come from all
sorts of places, including changes in labor, overhead rates,
schedules, requirements, quantity, absolutely everything, just
like building a house. Everything that changes has the
contractor saying, ``Cement has gone up. Brick has gone up. My
subs have gone up''--or down, but they are always changing.
And so there is a great churn. Some of it is part of life,
and some of it we try to control by saying, ``Tell us what you
are building, and we will cost that program.'' But I think it
is just intrinsic in the cost estimating process.
Mr. Coffman. Is there any way to just have more--I mean,
what would be some of the methods for having more discipline
over the--and I take it some of it would--having a change
board?
Mr. Nussbaum. There is something called a CARD, a Cost
Analysis Requirement Document, which is, to the chagrin of all
program managers, we say three or six months before we are
going to a milestone, you tell us what you are building, how
many, what the technologies are, and we will cost that program.
It may be that things change after that, but that is the
program we are going to cost, because otherwise we are chasing
our tail, and we don't know until the very last moment what we
are costing.
And in fact, it takes time to do a cost estimate. It is a
part of the discipline at the Office of the Secretary of
Defense (OSD) is a Cost Analysis Requirements Document will be
prepared six months before the milestone, and the lack of
availability of that CARD results in a day-to-day slip in the
milestone. So that is real discipline, but it has its obvious
down side, too.
Mr. Patterson. I think, too, that in dealing with the
discipline and structure, that I must tell you, I submit that
it requires rules. And the rules simply have to say that, after
milestone B, you don't have any more requirements. And, I mean,
it is somewhat draconian, but nonetheless, if it--unless of
course it is a safety of flight or it is an obvious design
failure that needs to be corrected.
But those kinds of things are few and far between, quite
frankly. And then, this idea that a requirement or a change
would give you a three to one or four to one payback in savings
while not perturbing the schedule is not a bad idea, either.
And as technology moves forward, that is entirely possible over
the course of a program.
Mr. Fitch. There are times with the requirements--I recall
when I was developing a black box that was going to go in 17
different platforms. I would have this platform coming to me
and say, ``It would really be easier if you changed it this
way. It will make me less expensive for my integration, save my
costs and schedule.''
What I came to understand is the program manager has to be
prepared to say no during the development phase. Now, the
configuration steering boards we have in place anticipates that
there may be compelling reasons to change a requirement,
moderate a requirement during development, but that is the
process of the configuration steering board, is to raise that,
if you would, visibility to the pressures of changing
requirements to have a senior level decision made about the
requirement once the milestone has been approved.
Mr. Andrews. Thank you.
Mr. Cooper.
Mr. Cooper. Thank you, Mr. Chairman.
And I would especially like to welcome a fellow Tennessean,
David Patterson, and congratulate him on starting Defense
Business Institute at the great University of Tennessee in
Knoxville. Thank you not only for your past service, but what
you are doing right now.
Can anyone provide me with enough historical perspective to
help me understand how, during World War II, men like Henry
Kaiser were able to produce destroyers and--I forget exactly
who produced airplanes, but it was an amazingly productive
period. And I don't know how--was there less bureaucracy then?
How was so much able to be accomplished so quickly? Anybody
know?
Mr. Nussbaum. My reading of history is that there were a
lot of failures, and then we tend to remember the successes,
which were terrific. But in fact, there were false starts.
The P-51 was wonderfully successful, but because it had a
bigger gas tank so it could keep up with the other aircraft, it
gave us greater range, but we didn't build it for that reason.
Just we sort of lucked into that, if you will.
I think that things were much simpler. We were able to turn
technology generations around much faster and, therefore,
absorb the failures. Today's systems are very complex. They
take a long time, and we just won't accept failure.
Mr. Fitch. I think that what we remember a lot is those
ships going down the ways, okay, daily, okay? That was in the
production phase.
Even the P-51 that was just mentioned by Mr. Nussbaum, when
it was first fielded, it didn't have the engine that ended up
being what everybody remembers about the performance of that
aircraft. There were various modifications made after it was
fielded. It was in the field and found that it was not exactly
was needed.
Mr. Patterson. I think, too, you had a tremendous
industrial base that was able to accommodate to the level of
technology that it was asked to accommodate to. Today, I am not
sure that we would be able to do that again in that amount of
time.
Dr. Ron Sega did a study not long ago when he was at DDR&E,
Director of Defense Research and Engineering, in which he
looked at the industrial base and what it could do, and found
that 62 percent of all of the Ph.D candidates that are enrolled
in disciplines critical to national security have temporary
visas. And that tends to put us at a disadvantage, because they
generally don't stay around and work for Lockheed or Northrop
or Boeing, and Skunk Works, particularly, or Phantom Works.
And if you will look at the Aerospace Industries
Association data, you will find that in the 1990, 1991, you had
1.3 million touch labor workers, highly skilled workers. Today,
we have something less than 700,000. Those are statistics that
should give us pause to consider what we are going to do in the
future.
Mr. Cooper. Help me understand. I know that we have had
failures in acquisition throughout our history, and Mr. Fitch
mentioned a success when he posted the note that was on the
vehicle that saved lives.
But on the front page of the Washington Post recently was a
statement by Secretary of Defense Gates when he attended the
return of the remains of some of our troops. He asked how they
died, and he was told they were in an inadequate vehicle. And
he cursed because that was symbolic of the fact that we have
had difficulty fielding relatively simple platforms like Mine
Resistant Ambush Protected vehicles (MRAPs) or up-armored
Humvees.
And at least for some period of time there was only one
manufacturer of up-armored Humvees in America, and yet, at the
same time, we have automobile companies going bankrupt and
looking for vehicles to build. So there seems to be a mismatch
somehow between pretty basic demands for troops and our ability
to field and source those even when we have able and willing
automobile companies who are looking for ways to keep their
plants busy. Why the mismatch?
Mr. Patterson. Oh, I think it is a very fundamental problem
in that you have a Department of Defense that understands fully
that we are at war with terrorists, and you have a country that
doesn't.
Mr. Cooper. So Chrysler or General Motors (GM) or Ford
didn't want to bid on the up-armored Humvee, or we couldn't----
Mr. Patterson. Well, actually, the subsidiaries of those
folks did, but they really were not in the business of putting
out those kinds of vehicles. The people who were, International
Harvester, obviously, and--but the design, fortunately or
unfortunately for us, that was available were foreign designs
of up-armored vehicles that were designed to sustain the kinds
of things that an MRAP would have to sustain.
Mr. Fitch. And the number of companies that have as a core
competency in the technologies for armor, okay, is not the same
as we have for the auto industry. In fact, we are doing a lot
of investment in the Department of Defense today to try to find
better armor, cheaper armor, especially lighter armor, whether
that be for the vehicles because, when you put the armor on it,
it puts a demand on the engine. It puts a demand on the
drivetrain. If you double the weight of the vehicle, whatever
the percentage is, okay, it has additional tolls upon the
reliability of the vehicle.
So we don't have the same industrial base. In other words,
the auto industry isn't the industrial base for our Army,
either.
Mr. Nussbaum. In a sense, the Humvee is the wrong vehicle
to up-armor, but it is the only vehicle to up-armor. But it was
built as a replacement for the Jeep, which didn't go in harm's
way.
So it was designed to optimize all the functions that it
was going to perform, and we missed the fact that it was going
to go in harm's way. So it wasn't suitable, wasn't optimal, for
up-armoring, but it is what we have, so we are going to up-
armor it.
Mr. Cooper. I see that my time has expired. Mr. Chairman,
look forward to another round.
Mr. Andrews. Thank you.
With the consent of my colleagues, we are going to do
another round, if that is okay with the panel as well, if it
fits your schedule. Thank you.
Mr. Patterson just made a suggestion that we might have a
rule that says no new requirements after milestone B, and that
prompted this question. Again, this piece of data that you
cite, about 33 percent of the program cost growth being
attributable to these requirements changes, how many of those
requirements changes happened after milestone B? Do you know?
Mr. Patterson. That data, as I recall, is after milestone
B.
Mr. Andrews. It is all after milestone B.
Mr. Patterson. That is where the majority of the growth
generally takes place.
Mr. Andrews. A related question: in your oral testimony,
you talk about the fact that, if I read this correctly, the
average increase in unit cost of the 28 MDAP programs of less
than 5 years since development is only 1 percent.
Mr. Patterson. That is correct. That is----
Mr. Andrews. But it is 55 percent from years 5 to 9. How
many of that in the 55 percent category, the five to nine, got
a waiver through milestone B, didn't meet the requirements to
get to milestone B but got waived past it? Do you know?
Mr. Patterson. No, I don't. I don't know exactly how many.
Mr. Andrews. But, I mean, would it be accurate to say it is
probably most of the 55 percent cost overrun comes from that?
Mr. Patterson. I would say a significant portion of it,
sure.
Mr. Andrews. One of the things that is in the conference
report that we will be looking at in the WASTE TKO bill today
is what we call intensive care, where if a program is permitted
to go forward, even though it didn't achieve the milestone B
criteria, if it is waived past it, there is a whole set of
intensive requirements that are imposed upon that program to
try to get it back under control.
I wanted to come back to, again, this bifurcation that you
create between derived and customer requirements. Describe for
us the process that you think ought to be instituted to
determine whether a derived requirement is added to the package
or not.
Let's say we are at a point--assuming for a moment that we
accept your proposition that there are none after milestone B,
which I assume you say there should be some exceptions, now as
you said, for true safety or emergency purposes, but let's
assume we are living in a world where, except for those narrow
situations, there are going to be no changes in requirements
after we hit milestone B.
We are now in pre-milestone B, and an ``Oh, by the way''
comes up, as you said earlier, ``Oh, by the way, this can do
this.'' Who should make the decision as to whether that gets
added to the package, and by what criteria?
Mr. Patterson. Well, I think that the program manager
should have the initial cut at whether or not they are going to
include that into the program. But program managers generally
are colonels or, in very large programs, brigadier generals who
have significant oversight within the Department.
I think that if they have a set of rules that say, first of
all, if it is not a safety of flight or if it is not some sort
of safety issue, or if it doesn't give me a return on my
investment, then I am going to have a thumbs-down, initially.
Mr. Andrews. How do you measure the concept of return?
Mr. Patterson. Well, let's take, for example, you have--
again, I will turn to this C-17.
There were parts of that airplane that were originally
designed for aluminum lithium, for example. Well, aluminum
lithium is strong, but it is brittle.
So an engineering change was made to change that to a
different alloy. We were breaking the aluminum lithium cargo
floor guides at a regular pace, so changing that eliminated the
problem of having to constantly replace it. That was a savings.
And those are the kinds of things that I would suggest are
three-to-one. And even the suppliers were given the opportunity
to do that.
Mr. Andrews. Got you.
It strikes me that this is really the essence of the 20
percent the secretary talks about in his 80 percent solution,
that what he really is aiming to get at here is to give us an
adjudicatory mechanism that draws the line between the 80 and
the 20 when you get to this point, and it is your suggestion
the program manager should be the first person to weigh in on
this.
Who should evaluate his or her recommendation?
Mr. Patterson. Well, then you have an engineering change
board that would provide a corporate view of it. And if it is a
particularly expensive change, then you are going to go to your
service acquisition executive, or if it is an ACAT 1-D, an
acquisition category 1-D, then you have the Defense Department
acquisition executive who would have a cut at that.
Mr. Andrews. And my final question, are you confident that
we can quantify this concept of value sufficiently to hit the
three or four? In other words, are all of the values that we
want to promote--you gave a great example of saving, replacing
a piece of a plane that is going to go wrong. But do all of the
value concepts lend themselves to that kind of quantification
that would let us say, ``Well, this fails to meet three-to-one,
so out?'' Pretty hard?
Mr. Patterson. No, it is much more difficult than that. And
that is why it takes a lot of research and study and to set up
standards and conditions whereby you can evaluate these.
Mr. Andrews. Frankly, and I will just conclude with this,
it is one of the reasons why we are glad we have the three of
you and the institutions that you represent, because we really
do turn to institutions like yours to assemble those data,
analyze them and give us a factual basis to draw the lines that
my questions imply. Thank you.
Mr. Coffman, your turn.
Mr. Nussbaum. Mr. Chairman, is it----
Mr. Andrews. Sure, Doctor.
Mr. Nussbaum. Is it appropriate for me to make a remark,
or----
Mr. Andrews. Sure. It is okay.
Mr. Nussbaum. I think it is not hard to measure the value
of ideas that replace current capabilities. It is always harder
to measure the value of things that represent new capabilities.
They don't represent a savings for the operating and support
detail that you were going to incur.
But if you are replacing a current capability, then it is
pretty easy to do an estimate, but it is still an estimate, of
what does it cost to invest to make this thing happen, and what
do you save over time in the operating----
Mr. Andrews. I agree. The much more difficult proposition
is where you have a new function that could be added by
something that you discover. How do you measure that, and that
requires trade-off analysis. It requires opportunity cost
analysis. It requires a lot of broader inquiries.
Mr. Nussbaum. And if you have that long tail, the question
then is do you do any discounting on it, the technical issue of
net present value and at what rates. The Office of Management
and Budget (OMB) helps us there.
Mr. Andrews. Not to be hyper-technical, but one of our
problems is then matching up the federal credit scoring and net
present value rules with the real-world ones, that very often a
decision--a classic example is in energy. The Department has
guidance to hit 25 percent alternative fuels by 2020. And in
order for them to do that, they need to do multi-year
contracts. But to do multi-year contracts, the Congressional
Budget Office (CBO) scores that as putting the whole net
present value into one year, which makes it almost impossible
to do, which means we don't do much of it, which means we are
falling backwards. So marrying the CBO criteria with the real-
world criteria is a bit of a challenge, too.
Mr. Nussbaum. Yes, sir. And so the devil really is in the
details on this, and----
Mr. Andrews. Well, the devil is in the CBO in this case.
Now, don't tell Mr. Elmendorf I said that, okay?
Mr. Nussbaum. And one other comment, and that is, when you
go to the Configuration Control Board or your Service
Acquisition Executive (SAE), you are proposing to spend
investment dollars to make this thing happen, and the promise
is that you will return operation and support (O&S) dollars
later on. That is a nice conversation, but it doesn't accord
with the budgeting realities.
Mr. Andrews. And it doesn't score in our--nor should it.
Mr. Fitch, do you want to add one thing, then I am going to
go to my friend from Colorado?
Mr. Fitch. If I could. I just wanted to say, again, that I
think it is useful to talk about the operation requirements,
the derived requirements, but to say again that, when we get
the requirement from the user, it is stated usually in
operational terms, okay, the results they wanted to see.
Mr. Andrews. Yes.
Mr. Fitch. To deal with industry, and for industry to build
something, those need to be translated into technical terms. We
also use the term ``derived requirements'' for that process. So
the derived requirements are really--there are a couple types
we are talking about here, that which are a part of the normal
system that you have to do.
The other one I would just say is your question is how do
we know we get value. As a program manager, I had an
acquisition program baseline. That was my contract. The way I
viewed it, it was my contract with my--decision authority and
with the American taxpayer to produce a capability at such a
cost with certain milestones.
And I think that is what most of us take and go back to.
And as we do these questions about derived requirements and
everything else, I think we keep in mind that framework.
Mr. Andrews. Yes. I think the panel clearly understands
that some subset of derived requirements are quite legitimate,
necessary and desirable. And I think the three of you have
given us some interesting tools to discriminate between
undesirable derived requirements and desirable ones, which is
what we are about.
Mr. Coffman.
Mr. Coffman. Thank you, Mr. Chairman.
We talked today about having discipline in terms of change
requirements. But do we also have to have discipline when
programs get so far out of line that they become questionable?
And I am thinking about--and I am working if you could
reflect on the President's--I can't remember the nomenclature
of the follow-on helicopter. Littoral Combat Ship (LCS) is in
question. I wonder if you all could reflect on when a program
gets so far out of line.
Mr. Patterson. Well, it would be helpful, I think, in the
total scheme of things, if we never let them get out of line.
But nonetheless, you are exactly right. They do happen. It does
happen.
And one of the things that I think is difficult when you
take, for example, the VH-71, the President's helicopter, and
that is a perfect example of where you came in with one set of
criteria as requirements and, over the course of the time, it
changed dramatically.
I think that we don't have a set of standards or conditions
that raise a budget flag that say, ``Wait a second, I am sorry,
you are red here, and you have been red three reviews in a row,
and we are canceling your program.'' And what we do is we put
ourselves in a position where it is the only game in town.
You have the President's helicopter that is arguably old,
and we don't have an alternative. You chose a manufacturer and
a helicopter, and there is no off-ramp. There is no plan B. And
we do this rather consistently.
And I am almost of the opinion that we do it by design. And
I would offer the Marines' Expeditionary Fighting Vehicle
(EFV), for example. We don't have a way to walk the dog back
down the path to get an amphibious vehicle that would replace
the EFV, so we need to make that work.
Mr. Fitch. I don't have the particulars on either of the
programs you asked about. I was the systems engineer for the
VH-3D, the current--well, it is one of the two current
presidential helicopters. It is a unique mission. I think there
is a desire usually if the White House says, or whatever or
whoever it is, says, ``I need a capability,'' that you find a
way to do it.
Going back to it, most of the changes that I saw, and the
pressures and the surprises that I got, occur in that first--
traditionally, the first 12 to 15 months of a program when you
go towards a Preliminary Design Review (PDR), because the
contractor is off doing a lot of things. There isn't a lot of
object deliverables to figure out does he get it, he doesn't
get it.
I think that the competitive prototyping system to focus on
doing PDRs earlier, to have effective communications with the
contractor teams and oversight of the contractor teams during
that period of time will do much more to get to a stable
baseline at milestone B, which is the actual program
initiation.
Mr. Nussbaum. I am going to sound like a professor on one
hand. On the other hand, because there are some historical
examples of things which failed, failed, failed and then were
terrific. Aegis was one, and Tomahawk was another. They just
took a long time to bring aboard. And for some reason, we had
the fortitude to stick with them and not say, ``This is an A-12
or a Gama Goat,'' and get rid of it.
So now we come to the category of LCS. Is it a Tomahawk or
is it a Gama Goat? We don't know. But I am taken with my
colleague's remark that, once you get past milestone B, if you
have three reds in a row, you have got some serious explaining
to do, with the presumptive answer, ``You are out.'' It is a
rule.
But the problem is knowing the future, and that is always
the problem. I don't know whether LCS is a Tomahawk which is
going to be absolutely terrific after a long incubation period,
and the same for V-22. You just don't know, so we make
decisions as people.
Mr. Coffman. Thank you.
Thank you, Mr. Chairman.
Mr. Andrews. Thank you.
Well, gentlemen, thank you very much for your expertise.
Your reward for doing such a good job is we will have to call
on you again. As the committee goes forward, our intention is
to try to make legislative proposals for the fiscal year 2011
authorization bill that will deal with the area of the problem
that the WASTE TKO legislation that we are dealing with today
does not deal with.
And I think you have given us some very intriguing ways to
measure the gap between what we pay and what we get. It is also
heartening to hear what I have heard this morning, a consensus
that this panel's contribution to the WASTE TKO bill was
essentially two concepts.
The first was to add a whole series of reviews and a lot of
scrutiny pre-milestone B, with particular emphasis on the
requirements process. And that did make it into the conference
report, and that will become the law this week, we think.
And second, the panel was very interested in much more
rigorous review, what we call intensive care, of systems that
pass milestone B by waiver, that have not met the requirements,
or that fail Nunn-McCurdy standards and get exempted from the
penalties there and go forward anyway. And I think if you look
at the cost overruns, a huge majority of them fall into one of
those two categories.
So what we wanted to do was to take the best practices that
you very ably described this morning and engage them as
intensely as we can in the systems that, again, never met the
criteria to get past milestone B but get past it anyway, and
those that fail Nunn-McCurdy but continue to live on.
And as Mr. Patterson said a few minutes ago--I think it was
Mr. Patterson--our ultimate goal is not to have any of those
cases in the future by unraveling the requirements process and
looking at it more intensely to intensify that pre-milestone B
analysis of what is going on.
And the other point that I would make that is more on our
side of the table than yours, I think that the principal reason
that we get these cost overruns is that, once something passes
milestone B, an enormous political constituency develops around
it. Now, there are tens of thousands of people deriving their
paycheck from a project, hundreds or thousands of
subcontractors, dozens or hundreds of congressional districts.
And as Secretary Gates I think can attest, making changes
in those programs is very politically difficult. If you get to
these flawed programs earlier when their political
constituencies are smaller and weaker, the chance to do the
right thing is a lot higher.
So not just for analytical reasons, but we think, given the
dynamic of the way these decisions are made in the political
world, the more precise we are in our measurements and the more
focused we are in our evaluation, in the requirements phase and
a little bit beyond that, we think the better job that we will
do.
So I would say to each of the three of you we welcome your
continued participation and input. We are certainly going to
call upon you for your feedback as we go forward in our
drafting process. And thank you very, very much for your time
and attention this morning.
Members will have a period of time by contacting either
majority or minority staff to supplement the record with
written questions, and we would invite the witnesses to do the
same thing.
And with that, the hearing is adjourned.
[Whereupon, at 9:09 a.m., the panel was adjourned.]
=======================================================================
A P P E N D I X
May 19, 2009
=======================================================================
PREPARED STATEMENTS SUBMITTED FOR THE RECORD
May 19, 2009
=======================================================================
[GRAPHIC(S) NOT AVAILABLE IN TIFF FORMAT]
=======================================================================
WITNESS RESPONSES TO QUESTIONS ASKED DURING
THE HEARING
May 19, 2009
=======================================================================
RESPONSE TO QUESTION SUBMITTED BY MR. ANDREWS
Colonel Patterson. To appreciate the context in which early C-17
cost overruns occurred, some background and explanation of a
methodology for arriving at an estimate of how much of the cost
overruns could be attributed to derived requirements is in order. In
this case engineering changes were the predominant example of what I
have described as derived requirements and that appear in the public
record.
The first manufactured part for the first C-17 was milled in
November of 1987 and the first squadron was declared to have Initial
Operating Capability in January 1995. The period between these events
was one of significant turbulence for the C-17 program. The contract
for the C-17 program was a fixed-price, incentive fee development
contract, because the aircraft was to be designed using off-the-shelf
technology and was determined to be ``low'' risk.
At the time of contract award, both the Government and MDC
[McDonnell Douglas Corporation] envisioned a program based on
commercial practices, minimum Government involvement and
concurrent development/production effort. Consequently, a fixed
price incentive contract was used to match the perceived low
risk of the program.\1\
---------------------------------------------------------------------------
\1\ Report of the Defense Science Board Task Force on C-17 Review,
December 1993, p 3
As the program became more complex and engineering changes more
frequent, MDC had no recourse to recoup investments in solving the
engineering problems and meeting the cost of addressing requirements
that were being added during full scale engineering and development
than to make claims to the Government against the contract.\2\ Though
it is not feasible to determine precisely how much of the total cost
overrun was attributable to derived requirements or engineering
changes, it is possible to understand the relationship between the
unanticipated cascade of engineering changes and increased requirements
and cost overruns by virtue of the amount of the claim MDC believed
justifiable.
---------------------------------------------------------------------------
\2\ Ibid.
---------------------------------------------------------------------------
COST OVERRUNS AND ENGINEERING CHANGES
As early as 1985 engineering changes were impacting the C-17
program costs. The Selected Acquisition Report for December 1985
described cost increase of $214.5M for engineering changes ``needed for
a four-pallet ramp, a combat offload rail system, and DoD standard
avionics racks.'' \3\ Though these modifications were not in 1985
referred to as ``derived requirements,'' they do meet the definition
used in the study included with the oral statement provided for the
record on May 19, 2009.\4\ These were the typical derived requirements
not in the original contract, but identified by the program management
to be necessary during development and presented as engineering
changes. These changes and the corresponding cost increase occurred in
the very early stages of the program, In December of 1985 the Full
Scale Development contract had been signed.\5\ The Government became
alarmed following a 1992 Department of Defense Inspector General Report
that identified a cost overrun of $700M on a $6.6B contract ceiling.\6\
McDonnell Douglas was paying for costs overruns on the fixed-price
contract and submitted claims for approximately $300M for what MDC
asserted were changes in scope of its initial contract agreement which
included ``costs for engineering, development, testing and production
of six C-17's and three test planes.'' \7\ In the end, after
significant negotiations with the Department of Defense, McDonnell
Douglas agreed to spend $456M in process improvements and the Defense
Department agreed to ``provide an additional $438M for the program.''
---------------------------------------------------------------------------
\3\ Selected Acquisition Report As of December 1985, p 10
\4\ Derived requirements, on the other hand, are requirements that
the customer has not specified directly as a requirement but that
emerge or derive from the design decisions that are made. Derived
requirements are not capabilities that the customer specifically has
identified. Particularly troublesome is a subset of derived
requirements that fall into the category of engineering changes--those
changes that improve on ``good enough'' and that have a combined effect
of driving up costs and missing schedule milestones.
\5\ Aeroflight, Aircraft of the World, Boeing C-17 Globemaster III,
retrieved May 27, 2009, from http://www.aeroflight.co.uk/types/usa/
boeing/c-17/c-17.htm
\6\ Battershell, A. L., The DoD C-17 versus the Boeing 777, A
Comparison of Acquisition and Development, National Defense University,
Washington, D.C. 1999, p 87-88
\7\ Adelson, A., Company News; McDonnell May Submit Big C-17 Bill,
New York Times, January 14, 1993, retrieved May 27, 2009, from http://
www.nytimes.com/1993/01/14/business/company-news-mcdonnell-may-submit-
big-c-17-bill.html
---------------------------------------------------------------------------
A significant portion of the cost problems centered on a cascade of
engineering changes that typified the years and immediate several
months leading up to the negotiated settlement between MDC and the
Department of Defense. The magnitude of the engineering changes
occurring during this period was described clearly in the December 1993
Defense Science Board's report as it stated, ``In May 1993 alone, over
1,000 changes were issued directly impacting production. The backlog of
engineering changes at the end of May 1993 reveals 5,800 open work
authorizations.'' \8\ Additionally, costly changes to the program were
government-imposed. A typical example is the C-17 test program which
was increased from an 80 aircraft month test program to a ``rebaselined
test program of approximately 152 aircraft months'' at the
recommendation of the Defense Science Board study, with cost split
evenly between the Government and MDC.\9\
---------------------------------------------------------------------------
\8\ Report of the Defense Science Board Task Force on C-17 Review,
December 1993, p 3
\9\ Ibid. p 12
---------------------------------------------------------------------------
SOME EDUCATED CONCLUSIONS
With the data that is immediately available from the historical
record it is reasonable to conclude that the engineering changes were
significant and made an impact on cost, schedule and performance on the
C-17 program. Referring again to Battershell, of the $7.3B for the
development costs and the first six aircraft in production Lots I and
II, MDC invested approximately $1.7B of its own funds.\10\ This amounts
to about 26 percent of the original contract amount of $6.6B. MDC
proposed a claim against the government of $1.2B in addition to the
$438M it had received in the settlement \11\ intended to recover what
it believed to be legitimate costs associated with program changes a
significant number of which were engineering changes. Though the term
``significant'' used above is not a precise accounting regarding ``how
much'' of the program was ``attributable'' to derived requirements, it
does provide a qualitative data point that is important in evaluating
opportunities to improve acquisition programs in the future. [See page
11.]
---------------------------------------------------------------------------
\10\ Battershell, p 90
\11\ Government Accountability Office Report, (GAO/NDIAD-94-141),
Military Airlift C-17 Settlement Is Not a Good Deal, April, 1994, p 6
---------------------------------------------------------------------------
|
NEWSLETTER
|
|
Join the GlobalSecurity.org mailing list
|
|
|
