Military

In this, my first annual report on the DoD test and evaluation (T&E) resources, I have two objectives: First, report on the "health" of T&E resources (people, facilities, and funding that support our test and evaluation programs), and second, describe my plans for measures to address the needs in these areas. I will also address my promise to the Secretary of Defense to work diligently to implement recent recommendations of the Defense Science Board (DSB) on T&E.¹ I am working to ensure that a test infrastructure is in place which is capable of adequate testing to meet the schedules of acquisition programs.

The majority of the developmental tests and some operational tests are conducted at the Major Range and Test Facility Base (MRTFB) locations shown in Figure 1. These sites contain the major, high-value test facilities within DoD and include significant air, land, and sea operating areas necessary to test modern weapon systems. The sites comprising the MRTFB are operated by 30,000 military, government civilian, and contractor personnel and consist of test facilities ranging from large wind tunnels and electronics integration test facilities to extensive open-air ranges. The book value of the MRTFB is over $25 billion and it includes over half of all land owned or used by DoD.

The policy governing the MRTFB has remained unchanged since its founding in 1974: "The MRTFB is a national asset that shall be sized, operated, and maintained primarily for DoD T&E support missions."² The intent of this policy is to ensure the MRTFB provides efficient, effective T&E capability with sufficient capacity margin to accommodate the fluctuations in test workload that are an inescapable feature of weapon system acquisition.

There are questions about the test infrastructure's ability to satisfactorily meet current and future T&E requirements in a manner that contributes to efficient weapon system acquisition. I am concerned that test resource issues may be one of the main reasons why adequate testing is not conducted in time to support deliberations associated with program decisions. Of particular concern are the following issues.

During the past decade while T&E infrastructure resources were being reduced, we witnessed an alarming trend of too many programs entering dedicated operational T&E (OT&E) without having completed sufficient developmental T&E (DT&E). As a result, the Services have conducted OT&E on immature systems and the results reflect the consequences. In recent years, 66 percent of Air Force programs have stopped operational testing due to a major system or safety shortcoming. Since 1996, approximately 80 percent of Army systems tested failed to achieve reliability requirements during operational testing.

In its December 2000 report, the DSB Task Force on Test and Evaluation Capabilities found that weapon systems are not adequately tested to assure their effectiveness and utility to operating units. The acquisition process fails to deliver systems to the warfighter that meet reliability and effectiveness requirements. And, unlike the commercial marketplace where managers place value on product testing, DoD program managers view it as an impediment.

The T&E process is not adequately funded. Limited budgets and acquisition reform have forced program managers to curtail test programs. Underfunded test centers are shifting more and more of their testing costs to acquisition programs. This practice leads to further limits on testing and encourages waivers of test requirements. The increasing number of test waivers undermines test adequacy and contributes to a false sense of system developmental maturity. T&E personnel ordinarily are not involved in early acquisition program budget deliberations. Consequently, program test budgets are too often firmly established before test programs are defined.

Infrastructure deficiencies contribute to inadequate testing by limiting the scope and depth of the testing that is conducted. An important step in carrying out my plan for improving our test infrastructure will be to identify such limiting deficiencies and prioritize measures to resolve them. My objectives are:

• Reestablish a modern, effective test infrastructure. Many facilities are aged and test capabilities for new technologies cannot be acquired due to insufficient investment. I will define and advocate a program to recapitalize the test infrastructure so that future technologies can be adequately tested.
• Redefine policy and funding to improve test effectiveness. Under existing policy, a growing proportion of test costs are passed to the users (in most instances, the acquisition programs). Too often this leads to less testing. I will work to redefine charge policy and provide necessary funding to encourage users to plan and conduct adequate testing.

The remainder of this section discusses these issues and my plan for addressing them.

Testing must be rigorous, robust, and focused on military missions, mission accomplishment, and total life-cycle suitability. To ensure thorough testing, DoD's T&E infrastructure must be restored and transformed along with our weapon systems. DoD must take management policy and resource actions to ensure rigorous and robust testing as well as timely, objective, continuous assessments of weapon systems. Appropriate actions taken now can assure adequate testing of future DoD systems by renewing aging infrastructure, investing for new technologies, and improving management processes.

The entire defense infrastructure has been underfunded for years and the T&E infrastructure is no exception. The recapitalization rate is based upon the value of the infrastructure and the annual resources provided to replace, upgrade, and maintain it. The facilities replacement rate for all DoD is approximately 192 years while the private sector replaces or modernizes its similar facilities at an average rate of once every 57 years.³ Secretary Rumsfeld's goal is to reduce the DoD infrastructure recapitalization rate to 67 years as shown in Figure 2.

While the recapitalization rate for the entire T&E infrastructure is 400 years, the key elements of that infrastructure are the technical facilities and equipment that are actually used to support testing. This T&E technical infrastructure has a recapitalization rate of approximately 70 years. Emerging weapon system technology demands a higher rate of test capability recapitalization to ensure adequate weapon system testing. The current recapitalization period is more than seven times that of comparable private sector industrial facilities and does not provide effective T&E capabilities. To reestablish an adequate technical test infrastructure, we must strive to reduce its recapitalization rate to half the existing one.

The aging T&E infrastructure increases the probability of failures in test support capabilities that could cause significant and costly schedule slippages in acquisition programs. For example, in December 2000 equipment failures at Arnold Engineering Development Center (AEDC) required major repair actions and delayed test support to acquisition programs. An AEDC dryer cooler leaked 23,000 gallons of water and ethylene glycol that delayed two planned tests. A motor failure occurred in one of the 35,000 horsepower drive motors in AEDC's Propulsion Wind Tunnel causing the wind tunnel to operate at reduced capability for seven months.

Investment funding for the T&E infrastructure provided over the past 10 to 15 years has not kept pace with the identified T&E needs, severely restricting our ability to adequately evaluate new technologies such as stealth, command and control systems, hypersonic weapons, and missile defense systems. Figure 3 shows the decline in the funding for modernizing the T&E infrastructure since FY90. In recent years, funding for targets and threat simulators has also been sharply reduced. The level of funding for improvement and modernization (I&M), while above the FY90 level, remains a fraction of that private industry invests in comparable high-technology infrastructure.

Investment for T&E does not approach that appropriate for a high-technology enterprise. From FY02 on, significant funding increases are required to adequately test future weapon systems incorporating emerging technologies. DoD began the 1990s with a validated backlog of $12 billion in T&E infrastructure modernization requirements. With new requirements emerging at a growing rate, the backlog is continuing to grow.

The principal programs for investment in the Department's T&E capabilities are: Service and Defense Agency Investment and Modernization (I&M) Programs and the Central Test and Evaluation Investment Program (CTEIP).

Service and Defense Agency Investment and Modernization (I&M) Programs. These programs provide funds for modernization of existing test capabilities and acquisition of new capabilities to meet the needs of the individual Service or Defense Agency. Each Military Service pursues an I&M program focused on their test facilities in areas that usually have limited tri-Service use or interest. The funding for Service I&M projects has increased modestly at a time when significant investments are needed for new technologies. Unfortunately, Service I&M requirements fail to compete effectively with other critical needs in the Service budget cycle. During the twelve-year period between the FY90 and the FY02 President's Budgets, total Service I&M funding increased less than $35 million in constant dollars for an infrastructure worth $25 billion.

Central Test and Evaluation Investment Program (CTEIP). The Central Test and Evaluation Investment Program (CTEIP) is an OSD-managed program established to improve T&E capabilities. CTEIP is the Department's corporate-level investment program for T&E capabilities that would be considered beyond a single Service's normal area of responsibility. Individual CTEIP investment projects are executed and implemented by the Services and Defense Agencies. In the past, CTEIP has focused not only on improving and standardizing range instrumentation but also on improving interconnectivity and interoperability among the ranges and acquiring advanced telemetry to meet demands for increasing data and data rates. Recent CTEIP investments focus on developing new capabilities that can be used by a variety of weapon systems with reduced need for user-interface development. Some example projects are:

CTEIP has proved to be a critical investment mechanism for the Department to ensure critical T&E capabilities are developed and in place to support multi-Service and joint weapon systems. It provides the Department-level view, thus enhancing the joint warfighting concept, minimizing duplication in T&E infrastructure, and reducing total acquisition cost to the taxpayer. An annual report describing on-going CTEIP projects is provided to interested DoD and congressional staff in the February timeframe.

When the capabilities of the test ranges are compared with requirements for testing current and future systems, significant deficiencies are evident. They limit the ability to conduct adequate testing of weapons and support systems. Some of the more significant deficiencies are:

• Range infrastructure. Inadequate compatibility exists between ranges (e.g., data acquisition and reduction systems, communications and data transmission systems, command destruct systems). Open-air test time on ranges is restricted as is ability to perform open-air jamming. Loss of radio frequency spectrum may inhibit test operations. Miss distance and attitude measurement systems lack adequate fidelity. Instrumentation shortfalls include limited radar, telemetry, and optical equipment assets to support multiple simultaneous engagements and insufficient instrumentation to track multiple vehicles. There are no chemical-biological test chambers large enough to accommodate complete systems. A replacement for the self- defense test ship is needed to retain the capability to demonstrate surface ship cruise missile defense systems.
• Targets and threat representations. Generally, realistic targets are not available in sufficient numbers to support the various weapon systems under development. Representative targets for certain anti-ship cruise missile threats are not available. Deficiencies exist in the quantity and types of ballistic missile defense targets. Threat representation shortfalls have also been identified. Needs include a vector-scoring capability on full-scale targets and improved capability for testing infrared missile engagements.
• Realistic test environments. New-generation systems have much more extensive operating footprints than their predecessors and, therefore, need much larger test ranges to support full-scale operational scenarios. Space test capabilities are not sufficient to meet space mission area testing requirements. Shallow water ranges for undersea warfare testing are inadequate. Chemical and biological simulators and simulants are not representative of the threat. Generally, there is a lack of priority and funding for testing of weapon systems in the extremes of their natural operating environments.
• Interoperability. Interfaces with other systems are not included in many test plans. Many systems are tested only on an individual basis. The failure to test systems with complementary ones in combined scenarios precludes effective assessment of their compatibility and ability to operate together.

Advances in weapon technologies and DoD's transformation initiatives will soon result in high-technology systems that cannot be adequately tested with the current test infrastructure. A T&E science and technology (S&T) program was approved for FY02 to expedite the transition of new technologies from the laboratory environment into T&E capabilities and to provide foundation technologies for a transformed test infrastructure. Such a program is essential to the continued viability of the test infrastructure. Initial funding of $16 million is planned for the FY02 program. Efforts are underway to provide continuing funding for FY03 and beyond, eventually ramping to a steady-state funding level of approximately $30 million per year.

T&E and S&T organizations have defined the critical technologies for the initial investments within this program. Focus areas for FY02 are spectrum efficiency, embedded instrumentation, information systems technology, space test, and hypersonics. In addition to initiating these projects, a long-range T&E technology roadmap will be prepared during FY02. The roadmap and successive updates will guide future investments and ensure that the program is focused and time-phased to meet the most critical T&E research and development shortfalls.

Testing modern military systems, and training with these systems, relies heavily on the use of the radio frequency (RF) spectrum. Recent studies show that the DoD transformation initiative and the next generation of technology being incorporated into weapon designs will generate proportionately greater data rates that will exceed the capability of our current test infrastructure. Not only will these systems require more spectrum to operate but the telemetry data rate requirements will be more demanding. In addition, the growth in the demand for consumer communication services has resulted in pressure from the commercial telecommunications industry for the reallocation of RF spectrum from government to non-government use. The reallocation of telemetry spectrum coupled with the increased data requirements have raised concerns regarding the availability of adequate spectrum to support test and training.

We have made major strides in addressing these concerns by investing in CTEIP initiatives that support our goals for spectrum efficiency. Current CTEIP projects address improvements in the capabilities to use existing allocations of spectrum through such techniques as efficient modulation, data compression, real-time system management, and networking. However, it will be difficult to support the exponentially increasing data transfer requirements using only these techniques. In order to meet the increasing demand for more test data, we need to investigate the potential of using higher frequency bands to supplement our current limited capabilities and resources. An agenda item for the 2006 World Radiocommunications Conference is the expansion of the spectrum available for telemetry by adding frequency allocations in the 3-30 GHz range. In anticipation of these allocations, techniques must be developed to overcome significant technical difficulties in operating in these higher bands. CTEIP has requested additional funding to determine the technical requirements and limitations of these bands.

The current approach to managing the DoD T&E infrastructure is through centralized oversight by DOT&E and decentralized funding and management by the Military Departments and Defense Agencies. Funding and manpower levels for the individual ranges and centers are programmed by the owning Service, even though the ranges may possess unique T&E capabilities which are used primarily by the other Services and Defense Agencies. This approach has led to a reluctance by the owning Service to fully fund and sustain some of these unique capabilities. The December 2000 Defense Science Board report recognized this management issue. Its most significant recommendation, from a management standpoint, is that ownership and resource management of portions of the T&E infrastructure should be consolidated in a unified DoD T&E Resource Enterprise.

Last year, the Service Vice Chiefs and the DOT&E began the development of a T&E Strategic Plan. The plan includes a vision, goals, and objectives that will provide a path to modernizing the T&E infrastructure. The plan will institutionalize a strategic review as part of the T&E investment process to bridge the gap between today's capabilities and tomorrow's technology. In the near future, I intend to focus on completing the plan and developing the necessary actions to implement it. I expect to reenergize the strategic planning effort in March 2002 and complete the plan by the end of the calendar year. Detailed implementations planning should begin in September 2002. Of course, decisions on the Efficient Facilities Initiative and actions taken to implement the recent DSB report could affect these timelines.

Over the last two decades, increasing challenges and limitations to the Department's use of its test and training ranges have arisen. Such range encroachment results from external factors, including urbanization, increasing environmental restrictions, and competition with civilian demands for airspace, land, ocean areas, and radio frequencies. The cumulative weight of these factors is increasingly recognized as a substantial threat to maintaining military readiness. To address this issue, the Defense Test and Training Steering Group (DTTSG), chaired by DOT&E, is leading a sustainable ranges initiative. Its purpose is to develop and implement a comprehensive strategy for access to range, frequency spectrum, and airspace essential to test and training needs. During 2002, we will develop a comprehensive set of legislative and regulatory proposals and formulate a multi-tier outreach effort with a goal of obtaining encroachment relief.

Sustainable range action plans for the following nine areas have been developed: Endangered Species Act and Critical Habitat, Unexploded Ordnance and Munitions, Frequency Spectrum, Maritime Sustainability, Airspace Restrictions, Air Quality, Airborne Noise, Urban Growth, and Outreach. Each action plan provides an overview and analysis of its respective encroachment issue area together with potential strategies and notional actions.

Reduced operating funding for those major test ranges included in the Major Range and Test Facility Base (MRTFB) has resulted in a significant shift of test costs to test range users. This increase in users' cost to test has encouraged acquisition program managers to accept additional risk by curtailing developmental testing. Consequently, weapons systems have often performed less effectively than expected in OT&E.

Another resource concern is the workforce available to conduct T&E. Workforce reductions have adversely affected developmental testing at the major test ranges.

The mission scope of the operational test agencies (OTAs) has expanded, notwithstanding a decrease in workforce. This is further compounded by reductions in military personnel with operational backgrounds allocated to developmental testing.

The charge and funding policy used within the MRTFB was established to provide a uniform approach to charging costs across the major ranges and test centers. By policy, DoD test customers reimburse MRTFB activities for direct costs readily identifiable with a particular customer order (i.e., an individual test program). Costs not funded by customers are funded from direct appropriations of the Military Service or Defense Agency responsible for the particular test range or center. These are referred to as institutional funds. This charge and funding policy was intended to ensure that adequate test capability and capacity is available to DoD test customers when needed and at reasonable and predictable rates.

Figure 4 shows the trends in test funding. The middle band of Figure 4 represents T&E institutional operating funds, which have been fairly constant since FY98 after falling nearly 25 percent during the prior eight years. The top band represents funding for military personnel. The chart shows that the funding represented by these two bands has decreased significantly.

The bottom band of Figure 4 is test customer funding. These funds are paid by customers for testing services provided by the test ranges and centers. User funding has remained relatively constant since FY90 but has increased in proportion to the total amount.

The cost for testing at the MRTFB is increasingly borne by the users as institutional funding has declined. A recent analysis shows that about $2.4 billion in test costs (previously funded in the MRTFB institutional budgets) have been shifted to the users since FY90. Eighty-five percent of the shift occurred during the last five years.

As institutional funds have fallen, the test ranges and centers have sought to recover more costs from users. The users, in turn, have reduced testing and accepted additional risk to remain within their budgets. Test adequacy has suffered as a consequence. In FY01, the MRTFB charged an estimated $250 million per year more to users than was charged to them prior to FY90. Effectively, this means that, although users in FY01 collectively paid the same amount as in FY90, they are doing less testing.

The shift of costs could be corrected in either of two ways: (1) allocate additional funds to the programs and designate them for testing or (2) allocate additional funds to the Service institutional accounts. The first approach would give the appearance of cost growth and could require significant monitoring by the Office of the Secretary of Defense to enforce the policy. The second approach would increase institutional funding to a level that restores the MRTFB to the traditional "institutional share" (actual amounts would vary by Service). This would provide an opportunity for more testing within a program's allotted test budget and reduce the negative incentive for users to conduct less testing.

The T&E workforce has declined by approximately one-third during the last decade despite a relatively constant workload. With few exceptions, MRTFB manpower has fallen steadily since FY90. Figure 5 displays the developmental test workforce. As is evident, all elements of this workforce are substantially smaller than they were a decade ago. Of particular concern is the dramatic decrease in the number of military personnel. Not having military personnel involved in early developmental testing results in missed opportunities to identify issues of military suitability and effectiveness.

In addition, there is anecdotal evidence that test centers are scheduling workload in a fashion that stabilizes the workforce at current levels. Such a practice adversely affects responsiveness to the needs of customers. This is a natural consequence of a smaller workforce. Also, because of limited resources for testing, emphasis is usually placed on minimizing test costs rather than minimizing the cost and schedule impacts on the weapon system acquisition program. This results in avoidable schedule delays and unnecessary program costs.

Last year's report stated that Operational Test Agency (OTA) workload was increasing. That trend has continued and is expected to do so for the foreseeable future. Factors increasing the workload include:

• New acquisition policy directing an evolutionary acquisition process emphasizing both earlier involvement of operational testers and evaluators in system development and more continuous operational evaluations,
• Increased complexity of the T&E process resulting from introduction of advanced technologies at an increasing rate, and
• Need to test systems in more complex operational environments.

Early and continuous involvement is recognized as an effective means to gain operational insights into a system under development when flexibility in the design still exists. It also allows the Operational T&E Agencies to build a knowledge base from which to develop an effective operational test program. Early involvement expands workload in ways not captured by traditional metrics used to account for the OTA workload (e.g., number of tests and number of OT events). Such metrics capture the end result of the test process but do not provide visibility into individual test activities like integrated product team meetings, document reviews, and preparation of T&E master plans. Moreover, existing metrics do not reflect workload generated by testing more complex systems.

Figure 6, depicting OTA workload projections, does not fully reflect the trend of growing workload. Nevertheless, each OTA has experienced rising workload during the past eight years while both funding and personnel have decreased. Civilian personnel fell 45 percent and military personnel declined 35 percent. Some losses were offset by increases in the contractor workforce. However, the total workforce is 25 percent smaller than in FY93.

Figure 7 depicts the downward workforce trend as well as the changing mix in military personnel, government civilians, and contractors.

Another concern is the aging of the government civilian component of the workforce. Presently, the average age of OTA civil servants from GS-7 to GS-15 is over 50. An analysis of retirement eligibility indicates approximately 38 percent of the civil servants will be retirement eligible by FY05. Exacerbating this situation is that younger professionals are not being hired in adequate numbers to compensate for future losses. This is reflected in the fact that only 11 percent of the OTA workforce is less than 40 years of age.

Finally, I am concerned about the proportionately smaller military presence in the OTA workforce. Coupled with an aging civilian population, the small military presence represents the potential for a significant loss of experience and capability in the core government workforce. It also suggests that DoD dependence on contractor support may grow.

Adequate resourcing of the OTAs is imperative if we expect to determine system operational effectiveness and suitability and contribute to formulating operational employment doctrine. Consequently, I will pursue remedying the staffing and funding deficiencies at each OTA and ensure that operational testers are involved continuously through the development process.

Secretary Rumsfeld has called for giving "increased priority to maintaining a robust test and evaluation program, which will require test centers and ranges.¼ This need for testing - and particularly for testing capabilities conducted over very long distances - requires the Department to maintain and modernize highly instrumented ranges and to manage the challenges of range encroachment." I will strive to meet these challenges. DOT&E will continue to champion the need for additional resources for T&E as well as develop proposals to increase the ability of T&E to contribute to DoD's acquisition programs. We must upgrade essential capabilities to meet the challenges presented by the increasing technological sophistication of our weapon systems and new operational concepts associated with DoD transformation efforts. Adequate investments in the T&E infrastructure will greatly enhance the ability of the acquisition process to deliver weapons systems that have been adequately tested to assure their effectiveness and utility to our warfighting forces.