Aviation Security: Development of New Security Technology
Has Not Met
(Chapter Report, 05/19/94, GAO/RCED-94-142).
Although the Aviation Security Improvement Act of 1990, passed in the wake of the terrorist bombing of Pan Am Flight 103, set goals for deploying new technology to detect explosives, the Federal Aviation Administration (FAA) has made little progress toward introducing new detection systems into everyday use. Several devices show promise, but technical problems are hampering their development and approval. FAA estimates that it could take as long as five years to approve new devices for airline use. Similarly, FAA's efforts to enhance airline survivability are promising but years from completion. GAO identified several weaknesses in FAA's security research program. For example, FAA does not plan to test new explosive detection systems at airports during the certification process. Further, FAA does not (1) do software reviews to evaluate system designs, (2) emphasize integrating different technologies into total systems, and (3) give enough attention to human factors. Purchasing the new security equipment will also place demands on airlines throughout the next decade. Yet FAA lacks a strategy to guide its and the airlines' efforts to introduce this equipment. If FAA expeditiously develops a strategy, the airlines will be in a better position to plan and budget for future security acquisitions. In addition, Congress is considering clarifying the availability of Airport Improvement Program grants to buy explosive detection systems. Several issues need to be resolved before such funds can be used for that purpose. --------------------------- Indexing Terms ----------------------------- REPORTNUM: RCED-94-142 TITLE: Aviation Security: Development of New Security Technology Has Not Met Expectations DATE: 05/19/94 SUBJECT: Air transportation operations Transportation safety Testing Aircraft research Research and development Safety standards Airports Facility security Terrorism Airline industry IDENTIFIER: FAA Airport Improvement Program ************************************************************************** * This file contains an ASCII representation of the text of a GAO * * report. Delineations within the text indicating chapter titles, * * headings, and bullets are preserved. Major divisions and subdivisions * * of the text, such as Chapters, Sections, and Appendixes, are * * identified by double and single lines. The numbers on the right end * * of these lines indicate the position of each of the subsections in the * * document outline. These numbers do NOT correspond with the page * * numbers of the printed product. * * * * No attempt has been made to display graphic images, although figure * * captions are reproduced. Tables are included, but may not resemble * * those in the printed version. * * * * A printed copy of this report may be obtained from the GAO Document * * Distribution Facility by calling (202) 512-6000, by faxing your * * request to (301) 258-4066, or by writing to P.O. Box 6015, * * Gaithersburg, MD 20884-6015. We are unable to accept electronic orders * * for printed documents at this time. * ************************************************************************** Cover ================================================================ COVER Report to Congressional Requesters May 1994 AVIATION SECURITY - DEVELOPMENT OF NEW SECURITY TECHNOLOGY HAS NOT MET EXPECTATIONS GAO/RCED-94-142 Explosive Detection Technology Abbreviations =============================================================== ABBREV AIP - Airport Improvement Program ATA - Air Transport Association DOT - Department of Transportation FAA - Federal Aviation Administration GAO - General Accounting Office OIG - Office of Inspector General OTA - Office of Technology Assessment RE&D - research, engineering, and development Letter =============================================================== LETTER B-256584 May 19, 1994 Congressional Requesters This report, prepared at your request, examines the Federal Aviation Administration's (FAA) efforts to develop new equipment for detecting explosives and methods to improve aircraft survivability as mandated by the Aviation Security Improvement Act of 1990. We are making recommendations to ensure that (1) FAA's process for approving new explosive detection equipment can provide the aviation community and the flying public with effective and reliable technology and (2) FAA's research efforts are properly managed to meet the threats to aviation. We are sending copies of this report to the Secretary of Transportation; the Administrator, FAA; the Director, Office of Management and Budget; and other interested parties. We will also send copies to others upon request. This work was performed under the direction of Kenneth M. Mead, Director, Transportation Issues, who can be reached at (202) 512-2834. Major contributors to this report are listed in appendix II. Sincerely yours, Keith O. Fultz Assistant Comptroller General List of Requesters The Honorable Frank R. Lautenberg Chairman The Honorable Alfonse M. D'Amato Ranking Minority Member Subcommittee on Transportation and and Related Agencies Committee on Appropriations United States Senate The Honorable Bob Carr Chairman The Honorable Frank R. Wolf Ranking Minority Member Subcommittee on Transportation and Related Agencies Committee on Appropriations House of Representatives The Honorable Tim Valentine Chairman The Honorable Tom Lewis Ranking Minority Member Subcommittee on Technology, Environment, and Aviation Committee on Science, Space, and Technology House of Representatives ============================================================ Chapter 0 EXECUTIVE SUMMARY ============================================================ Chapter 1 PURPOSE ---------------------------------------------------------- Chapter 1:1 The terrorist bombing of Pan Am Flight 103, which killed 270 people, clearly demonstrated the need for new technology to detect explosives. The Congress subsequently passed the Aviation Security Improvement Act of 1990, requiring, among other things, that the Federal Aviation Administration (FAA) support efforts to accelerate the research and development of new technologies to protect civil aviation from terrorists. Concerned about the safety of the traveling public in today's uncertain world, the Chairman and Ranking Minority Member, Subcommittee on Transportation and Related Agencies, Senate Committee on Appropriations, the Chairman and Ranking Minority Member, Subcommittee on Transportation and Related Agencies, House Committee on Appropriations, and the Chairman and Ranking Minority Member, Subcommittee on Technology, Environment, and Aviation, House Committee on Science, Space, and Technology, asked GAO to examine FAA's progress in developing new security technology and to specify the actions that FAA could take to improve its security research program. GAO is also providing information on several issues concerning the eventual implementation of new security technology. BACKGROUND ---------------------------------------------------------- Chapter 1:2 The Aviation Security Improvement Act set a goal for FAA to have new explosive detection equipment in place by November 1993. The Congress took this action to ensure that FAA's involvement would expedite the development of this technology. FAA's responsibilities include developing performance standards, assisting the private sector in developing systems, and approving (certifying) systems for airlines' use. To implement the act, FAA is supporting the development of new explosive detection devices and methods to improve the survivability of aircraft, including blast-resistant (hardened) luggage containers. RESULTS IN BRIEF ---------------------------------------------------------- Chapter 1:3 FAA has made little progress toward meeting the act's goal for deploying new explosive detection systems. Although several devices show promise, technical problems are slowing the development and approval of the devices. FAA's Aviation Security Research and Development Scientific Advisory Panel estimates it could take FAA 2 to 5 years to approve new devices for airlines' use. Similarly, FAA's efforts to enhance aircraft survivability are promising but are several years from completion. In addition, despite recommendations from the National Academy of Sciences and others, FAA does not plan to test new explosive detection systems at airports during the certification process. GAO identified several other weaknesses in FAA's security research program. For example, FAA does not (1) conduct software reviews to evaluate system designs, (2) emphasize integrating different technologies into total systems, and (3) focus sufficient attention on human factors issues. Developing new technology is only part of the challenge; purchasing the new security equipment will also place demands on the airlines throughout the next decade. However, FAA lacks a strategy to guide its and the airlines' efforts to implement this equipment. If FAA expeditiously develops a strategy, the airlines will be in a better position to plan and budget for future security acquisitions. In addition, the Congress is considering legislation that would clarify the availability of Airport Improvement Program grant funds to purchase explosive detection systems. GAO has identified several issues that need to be resolved before such funds can be used for that purpose. PRINCIPAL FINDINGS ---------------------------------------------------------- Chapter 1:4 APPROVED TECHNOLOGY IS NOT AVAILABLE FOR INDUSTRY'S USE -------------------------------------------------------- Chapter 1:4.1 Technical problems are slowing the development of new technology, and it may be several years before new security devices are in use that can meet FAA's requirements for screening checked baggage. FAA has 40 detection projects but has conducted laboratory tests on only 7; none fully meets FAA's performance requirements. In the interim, FAA is considering purchasing commercially available devices, but such devices have limitations. Although FAA's research on hardened luggage containers shows that they can help prevent explosions from damaging aircraft, FAA needs to resolve such issues as the cost, weight, and durability of the containers. FAA officials are optimistic that they can resolve these issues. FAA is continuing research on other aircraft survivability techniques, but officials could not estimate when they would be incorporated into commercial aircraft. FAA CAN TAKE STEPS TO IMPROVE TECHNOLOGY DEVELOPMENT -------------------------------------------------------- Chapter 1:4.2 Since explosive detection technology is evolving, FAA will be conducting security research well into the foreseeable future. GAO identified several weaknesses whose resolution would enhance FAA's current and future efforts. For example, FAA's process for certifying new explosive detection devices does not ensure that the technology can perform reliably in day-to-day use. FAA plans to rely on tests conducted at its own laboratory--not at a major domestic airport--before approving new technology for airlines' use. FAA officials believe that conducting such tests would, among other things, add time and cost to the certification process. However, the airline industry and others disagree with FAA's approach and believe that operational testing should be part of the certification process. In addition, FAA does not evaluate the effectiveness and/or performance of the software for the new devices even though the devices rely heavily on automation to reduce dependence on human operators. Also, despite recommendations from the Office of Technology Assessment and the Aviation Security Research and Development Scientific Advisory Panel, FAA has made little progress in integrating (linking) various technologies to maximize the strengths of each. FAA, the National Academy of Sciences, and others agree that no single device can meet all of FAA's requirements for screening checked baggage; therefore, devices will have to be used in combination. However, FAA plans to rely largely on the airlines to combine various devices into explosive detection systems. GAO believes a more prudent approach would be to address systems integration early in the development process to reduce development costs and delays and ensure that devices can work together effectively. Furthermore, although the devices rely heavily on automation, they are unlikely, in the near term, to eliminate the need for screeners. Yet FAA does not pay sufficient attention to human factors associated with using the new devices, such as how screeners understand alarms and make decisions about suspicious objects. In January 1994, FAA undertook a new initiative to accelerate the near-term development of new technology for industry's use. Through simulation modeling, FAA's initiative should provide some information about the impact of new devices on the flow of passengers. However, the initiative does not address other program weaknesses that GAO identified--software, systems integration, and human factors issues associated with current and future security technology. FAA NEEDS AN IMPLEMENTATION STRATEGY -------------------------------------------------------- Chapter 1:4.3 FAA does not have a strategy that articulates important milestones, sets realistic expectations, and identifies resources to guide efforts for implementing new explosive detection technology. The airline industry is particularly concerned about the acquisition and life-cycle costs for the new devices. The sooner FAA disseminates resource and other information, the sooner the airlines will be able to plan and budget for future security acquisitions. Legislation has been introduced that would make it clear that airports could purchase explosive detection equipment with Airport Improvement Program funds. However, the Federal Aviation Act of 1958, as amended, makes airlines--not airports--responsible for screening passengers and their luggage for domestic and international flights. In addition, the proposed legislation did not specify whether the devices must receive FAA's approval to be eligible for such funding. RECOMMENDATIONS ---------------------------------------------------------- Chapter 1:5 GAO recommends that the Secretary of Transportation direct the FAA Administrator to (1) require airport tests of the performance and reliability of new explosive detection devices before certifying new technology for industry's use, (2) evaluate software when reviewing system design, and (3) place greater emphasis on integrating devices during development. GAO is also making other recommendations. (See chs. 2 and 3.) MATTER FOR CONGRESSIONAL CONSIDERATION ---------------------------------------------------------- Chapter 1:6 The Congress may wish to consider requiring FAA to certify explosive detection equipment as a condition of eligibility for Airport Improvement Program grant funds. AGENCY COMMENTS ---------------------------------------------------------- Chapter 1:7 GAO discussed the facts and recommendations in this report with the Department of Transportation's Director of Intelligence and Security; FAA's Assistant Administrator and Deputy Administrator for Civil Aviation Security; Director, Policy and Plans for Civil Aviation Security; Director, Aviation Security Research and Development Service; and other FAA officials. These officials agreed with some of the recommendations. They stated, however, that FAA will test candidate explosive detection devices at airports "as necessary" after approving the devices but before directing widespread implementation. They noted that, once approved, a device that marginally met some of FAA's performance requirements could still be useful at lower-activity airports. FAA's statements about testing equipment represent the agency's first commitment to such tests. Throughout this effort, GAO noted that FAA was reluctant to test new devices at airports before mandating their use. In addition, FAA has not specified the criteria it will use to determine when the devices need to be tested. In GAO's view, FAA should test all candidate devices at airports during the certification process because testing "as necessary" will not be sufficient to gain the confidence of an industry skeptical of FAA's ability to develop and test new security technology. INTRODUCTION ============================================================ Chapter 1 Protecting civil aviation against terrorist attacks is a major challenge for security personnel throughout the world. Terrorists have continually increased their knowledge and sophistication in the use of explosives. Civil aviation has been and will continue to be a primary target for terrorists. The 1988 terrorist bombing of Pan Am Flight 103, which killed 270 people, clearly illustrated the need for new explosive detection technology. In May 1990, the President's Commission on Aviation Security and Terrorism reported that the aviation system was seriously flawed and was failing to adequately protect the traveling public. In October 1990, the Congress passed the Aviation Security Improvement Act, requiring the Federal Aviation Administration (FAA) to promote and strengthen aviation security through research and development. Among other things, the act directed FAA to support the acceleration of technologies and procedures to counteract terrorist acts against civil aviation and set a goal to deploy new explosive detection equipment at airports by November 1993. The Congress took this action to ensure that FAA moved forward expeditiously to support the development of technology that would help prevent a repetition of the Pan Am 103 incident. FAA PROMOTES RESEARCH AND DEVELOPMENT TO ENHANCE AVIATION SECURITY ---------------------------------------------------------- Chapter 1:1 FAA is responsible for the safety and security of civil aviation in the United States and plays an important role in the development of new security technology. Today, FAA is promoting the development of explosives, weapons, and trace detection systems and technologies to enhance aircraft survivability and is trying to increase airport screeners' proficiency and address other human factors issues. FAA also prescribes a regulatory process to certify systems for airlines' use.\1 Within FAA, the Assistant Administrator for Civil Aviation Security establishes security requirements, policies, and strategic plans; the Aviation Security Research, Engineering, and Development Service initiates technology development programs. To develop new security technology, FAA (1) establishes performance standards for equipment, (2) selects the mix of technologies for development, (3) provides oversight and technical assistance to contractors, (4) tests equipment to ensure that it meets the performance standards, and (5) certifies (approves) the equipment as suitable for airlines' use. Under the agency's regulations, the FAA Administrator can require airlines to deploy certified devices and systems. FAA can also allow airlines to purchase or lease unapproved equipment for testing at airports. FAA is also helping foreign countries develop new technology because officials believe that equipment deployed at foreign airports benefits Americans traveling abroad. FAA has cooperative research agreements with five foreign governments to exchange information on counterterrorism technology. FAA officials also participate in international conferences and assist in testing equipment at foreign airports. In conducting its security research, engineering, and development (RE&D) program, FAA has enlisted the help of universities and other such government agencies as the Departments of Defense and Energy to participate in research projects and provide experts for advisory panels. Since the passage of the Aviation Security Improvement Act, the Congress has provided FAA with about $130 million for security research. Specifically, FAA's security RE&D funding has grown from $9.9 million in fiscal year 1989 (before the act's enactment) to $35.9 million in fiscal year 1994--a 262-percent increase. FAA's Technical Center in Atlantic City, New Jersey, is responsible for managing the security RE&D program and has 35 staff working on various projects. Table 1.1 compares funding in fiscal years 1989 and 1994 and shows how funds are allocated among different research areas. Table 1.1 Funding for Security RE&D, Fiscal Years 1989 and 1994 (Dollars in millions) 1989 1994 -------------------------------------------- ------ ------ Explosive, weapons, and trace detection $9.9 $22.8 Airport security 0 2.5 Aircraft hardening program 0 7.8 Human factors 0 2.8 ============================================================ Total $9.9 $35.9 ------------------------------------------------------------ As indicated in table 1.1, FAA's security RE&D program has expanded to cover a wide range of research efforts. Before the Pan Am 103 incident, FAA focused primarily on developing weapons detection systems and the thermal neutron analysis device.\2 Currently, FAA is helping to develop new explosive, weapons, and trace detection systems and methods to increase aircraft survivability. It is also conducting research on human factors and on the security of FAA and airport facilities. FAA contracts with industry for the majority of the research; other federal laboratories and universities also participate in this effort. As of December 1993, FAA had 40 projects on detecting explosives; FAA has operating prototypes for 14. (App. I shows the status of these 40 projects.) -------------------- \1 In the context of this report, an explosive detection system is an automated device, or combination of devices, that can detect different types of explosives. \2 This device uses neutron radiation to detect explosives in checked baggage. It represents the first FAA-supported effort to develop an explosive detection device. FAA HAS ESTABLISHED REQUIREMENTS FOR EXPLOSIVE DETECTION SYSTEMS -------------------------------------------------------- Chapter 1:1.1 FAA has developed performance standards that new explosive detection systems for checked baggage must meet to be certified for airlines' use. In September 1993, FAA published its standards after a review of threats to aviation security and inputs from the U.S. intelligence community, numerous federal agencies, and the scientific and academic communities.\3 In the standards, FAA states that new detection systems must be more capable than current airport systems in detecting different material and smaller quantities of explosives. Current airport X-ray systems can detect only metal objects--not sophisticated plastic explosives. The bombing of Pan Am 103 illustrated the need for security devices to detect small plastic explosives. The plastic explosive suspected in that incident is virtually odorless; difficult to detect; can be molded into a shape that appears as a common, harmless item on X-ray screens; and can be rolled into thin sheets and placed in baggage lining. According to FAA's standards, new detection equipment will be required to rapidly screen baggage for explosives with a high rate of detection and a low rate of false alarms.\4 Moreover, FAA intends to automate explosive detection systems to make them faster and less dependent on human operators (screeners). The equipment will initially detect the explosives; screeners will not get involved until after the initial detection alarm has sounded. For example, screeners will not search luggage until an explosive detection device has identified a suspicious object and triggered an alarm. FAA, the National Academy of Sciences, airline industry representatives, and others agree that no single explosive detection device can currently meet all of FAA's new requirements for screening checked baggage. In the aftermath of Pan Am 103, many in the aviation community hoped that one device--coined a "silver bullet"--could rapidly and efficiently detect a wide range of explosives. This hope, however, has given way to the more pragmatic view that several devices will have to used in combination. Therefore, FAA's strategy is to develop a comprehensive mix of technologies that can be used in combination at airports. FAA is focusing on the development of two types of devices: (1) bulk detection devices, including X-ray and nuclear projects that screen baggage for explosives and weapons and (2) trace detection devices that "sniff" baggage, people, and electrical items for chemical particles used in explosives. According to FAA officials, in August 1994 they will review the trace technologies and determine whether such equipment could provide secondary screening for checked baggage or should be used to screen carry-on baggage and electronic items only. -------------------- \3 Some of the information about the performance requirements is classified; therefore, we are precluded from discussing issues such as performance goals and quantities of explosives that must be detected. \4 A false alarm occurs when a detection device sounds an alert but no explosive is present in checked baggage. WORK ON AIRCRAFT SURVIVABILITY COMPLEMENTS RESEARCH ON EXPLOSIVE DETECTION SYSTEMS -------------------------------------------------------- Chapter 1:1.2 Commercial aircraft have limited ability to resist the effects of an internal explosion. An explosion in a pressurized cargo or baggage hold, generally located beneath the passenger deck, could lead the aircraft to fail catastrophically. Therefore, FAA is examining methods to protect aircraft from damage caused by an internal explosion. FAA's security RE&D program includes research on blast-resistant luggage containers and techniques to harden aircraft structures. Because FAA does not fully understand the specific amounts, types, and locations of explosives that may cause catastrophic damage to commercial aircraft, the agency has started to gather empirical data on the vulnerability of aircraft to explosives. FAA expects to complete these efforts in 1996 and then plans to develop techniques to mitigate the effects of blasts. The relationship between aircraft survivability and explosive detection is important and will have a significant impact on FAA's efforts. If FAA finds that an aircraft cannot be made to withstand an explosion, then detection devices will have to be as or more sensitive than FAA now requires. In 1992, the House Committee on Appropriations directed FAA to analyze the trade-offs between survivability and detection. We testified in April 1993 that FAA does not expect to complete this analysis until fiscal year 1996.\5 -------------------- \5 FAA Budget: Important Challenges Affecting Aviation Safety, Capacity, and Efficiency (GAO/T-RCED-93-33, Apr. 26, 1993). THE AIRLINE INDUSTRY AND OTHERS VIEW FAA'S SECURITY RE&D PROGRAM WITH SKEPTICISM ---------------------------------------------------------- Chapter 1:2 The airline industry is skeptical about FAA's ability to develop effective explosive detection systems because FAA was previously unable to develop an effective thermal neutron analysis device. The airline industry criticized FAA for not rigorously testing this equipment before mandating its use. Thermal neutron analysis had dominated FAA's RE&D expenditures in the mid-1980s. In 1985, FAA awarded a design contract and in 1988 awarded a production contract for this equipment. FAA purchased six machines for airport testing and intended to require U.S. airlines to deploy the systems at domestic and international airports over a 5 year period, at an estimated cost of about $897 million. In its May 1990 report, the President's Commission on Aviation Security and Terrorism objected to the deployment of thermal neutron analysis devices because the equipment could not, without an unacceptably high rate of false alarms, detect the amount of material widely believed by investigators to have destroyed the Pan Am 103 aircraft. Furthermore, the Commission criticized FAA's specifications as inadequate because these specifications were based on estimates by FAA personnel. The Commission also noted that no computer modeling was performed to arrive at the specifications and that no testing was conducted on pressurized hulls to determine the minimum amount of explosives that could destroy a commercial aircraft. Airline officials expressed concern over the thermal neutron device's excessive size, high cost, slow speed in processing baggage, and high rate of false alarms. FAA continues to test the thermal neutron analysis device because, according to officials, it is the only device that can detect all types of explosives as specified in FAA's requirements and is "still as good" as any other automated device when processing over 500 bags per hour. OUTSIDE EXPERTS HAVE RECOMMENDED SIGNIFICANT CHANGES TO FAA'S PROGRAM ---------------------------------------------------------- Chapter 1:3 Since the Pan Am 103 incident and the report of the President's Commission, outside experts have also criticized FAA's security RE&D program and recommended significant changes. The following discussion briefly summarizes these assessments and their recommendations. The Office of Technology Assessment (OTA) reviewed FAA's program in 1991. OTA concluded that the program needed a more comprehensive technology focus, realistic technical requirements, independent operational testing of new equipment, and proper procedures for conducting test programs.\6 FAA's Aviation Security Research and Development Scientific Advisory Panel reviewed the status of FAA's technology projects in June 1992.\7 The panel concluded that FAA had not made sufficient progress and suggested that FAA purchase commercially available explosive detection equipment to improve aviation security until FAA resolved the technical problems with its program. In addition, in response to the Aviation Security Improvement Act, FAA contracted with the National Academy of Sciences to review its ongoing projects and provide guidance on the program's future direction.\8 In March 1993, the Academy recommended specific technologies for FAA to pursue, assessed the progress of ongoing projects, recommended testing approaches, and suggested that FAA place more emphasis on integrating explosive detection devices. -------------------- \6 Technology Against Terrorism--Structuring Security, OTA, Jan. 1992. \7 This panel advises FAA on a wide range of security research issues and comprises scientific and technical experts from the Department of Defense, a major airline, a law enforcement agency, an aircraft manufacturer, and academia. \8 Under the auspices of the National Academy of Sciences, the National Materials Advisory Board established a Committee on Commercial Aviation Security to fulfill the FAA contract. The committee included experts from government, private industry, and academia. OBJECTIVES, SCOPE, AND METHODOLOGY ---------------------------------------------------------- Chapter 1:4 At the request of the Chairman and Ranking Minority Member, Subcommittee on Transportation and Related Agencies, Senate Committee on Appropriations; the Chairman and Ranking Minority Member, Subcommittee on Transportation and Related Agencies, House Committee on Appropriations; and the Chairman and Ranking Minority Member, Subcommittee on Technology, Environment, and Aviation, House Committee on Science, Space, and Technology, we examined FAA's efforts to respond to the Aviation Security Improvement Act of 1990. Specifically, we were asked to examine FAA's progress in developing new technology and to specify the actions that FAA could take to improve its security RE&D program.\9 In addition, we have provided information on cost and funding issues facing FAA and the aviation community in implementing new explosive detection technology. To obtain the information in this report, we reviewed the Aviation Security Improvement Act of 1990 and its legislative history, as well as FAA's policies and procedures to implement the act. We also discussed program requirements, policies, and plans with DOT's current and former Director, Security and Intelligence, as well as other officials within that office; FAA's current and former Assistant Administrator for Civil Aviation Security; officials of the National Academy of Sciences and the Air Transport Association (ATA); and several airline officials responsible for security. In addition, we determined the status of all 40 explosive detection technology projects and examined 14 in detail by reviewing FAA project and contract files and interviewing officials responsible for monitoring the status of projects. We selected the 14 projects because, as of September 1992, FAA believed that they showed promise and had funded the development of prototype models for testing. We also determined the status of FAA's efforts to develop blast-resistant luggage containers and aircraft survivability techniques. In addition, we reviewed reports from OTA, the National Academy of Sciences, and FAA's Aviation Security Research and Development Scientific Advisory Panel on FAA's security RE&D program. We also reviewed FAA's plans for deploying "off-the-shelf" (commercially available) technology and discussed these plans with FAA officials. We did not examine efforts by foreign governments to test and deploy explosive detection devices. We also identified the challenges that FAA faces in certifying new security equipment by evaluating FAA's plans to test and approve new technology for airlines' use and FAA's proposed strategy to use commercially available explosive detection systems at category X airports.\10 We also reviewed proposed legislation that would clarify airports' eligibility to purchase explosive detection equipment under the Airport Improvement Program (AIP).\11 In addition, we visited Miami International Airport to observe FAA's tests of various new explosive detection equipment. To determine FAA's basis for investing in new security technology, we reviewed project documentation to determine the technical and economic factors that FAA considered when deciding whether to continue projects. We accepted FAA's analysis about the amount and type of explosives that equipment should detect and did not validate the threat levels that formed the basis for these requirements. We also reviewed development contracts and project files for explosive detection systems to determine whether FAA had defined technical requirements and conducted software evaluations. We performed our work primarily at FAA headquarters in Washington, D.C., and at the FAA Technical Center, Atlantic City International Airport, New Jersey. We conducted our work between November 1992 and March 1994 in accordance with generally accepted government auditing standards. As requested, we did not obtain written agency comments on a draft of this report. However, we discussed the facts and recommendations with DOT's Director of Intelligence and Security, FAA's Assistant Administrator and Deputy Assistant Administrator for Civil Aviation Security; Director, Office of Civil Aviation Security Policy and Planning; Director, Aviation Security Research and Development Service; and Manager, Requirements Analysis and Integration, as well as other FAA and DOT officials. DOT's and FAA's detailed comments and our evaluation are provided at the end of chapters 2, 3, and 4. -------------------- \9 See Aviation Security: Additional Actions Needed to Meet Domestic and International Challenges (GAO/RCED-94-38, Jan. 27, 1994) for our views on FAA's response to other provisions of the act. \10 FAA categorizes airports on the basis of passenger volume and the complexity of security programs in place. Category X airports are those that have high traffic levels and complex security programs and serve as international gateways. FAA has designated 19 airports as category X airports. \11 AIP provides grants to airports to sustain or increase their safety, security, and capacity by expanding and improving their facilities. TECHNICAL PROBLEMS ARE HINDERING FAA'S EFFORTS TO DEVELOP NEW EQUIPMENT ============================================================ Chapter 2 FAA has made little progress toward meeting the act's goal to deploy new explosive detection technology, and officials could not estimate when new devices would be certified for the airline industry's use. Although several devices show promise, technical problems have slowed their development. Similarly, FAA's efforts to improve aircraft survivability are promising but are several years from completion. For example, FAA tests indicate that new blast-resistant luggage containers are feasible; however, FAA must still address such issues as the cost, weight, and durability of the containers. Because new technology has not been developed as rapidly as the Congress expected, FAA is considering whether to purchase commercially available explosive detection devices as an interim measure--a step some foreign governments have already taken. However, commercially available technology has limitations. NEW DETECTION EQUIPMENT IS NOT AVAILABLE FOR AIRLINES' USE ---------------------------------------------------------- Chapter 2:1 It is uncertain when new equipment that can meet FAA's requirements for detecting sophisticated explosives in checked baggage will be available for the airline industry's use. New explosive detection technology is still evolving. Some technologies, such as automated X-ray devices, show promise for detecting explosives, but technical problems have slowed their development. Trace and nuclear technologies show little possibility of meeting FAA's requirements for checked baggage at this time. FAA's Aviation Security Research and Development Scientific Advisory Panel estimates that FAA could take 2 to 5 years to certify a device that can meet its standards for screening checked baggage. FAA officials cautioned that many technical challenges remain and that estimating development time is difficult. ADVANCED X-RAY TECHNOLOGY SHOWS PROMISE, BUT TECHNICAL PROBLEMS ARE SLOWING DEVELOPMENT -------------------------------------------------------- Chapter 2:1.1 As of December 1993, FAA had 40 research explosive detection projects, including 14 prototype units, 4 of which are suitable for screening checked baggage. Our review of the development status of the 14 prototypes showed that 9 had been delayed--by 1 to 18 months--because of technical problems. Furthermore, FAA has conducted laboratory tests on only seven devices; none fully meets FAA's performance standards. FAA officials said that they expect to have five additional advanced prototypes available for testing in fiscal year 1994 but could not estimate when the new devices would be certified for the industry's use. Airline security experts who are familiar with FAA's program are concerned about the agency's lack of progress, and one official noted that FAA has not approved a single device for screening checked baggage that differs from the equipment in use before the Pan Am 103 incident. Figure 2.1: Examples of Advanced X-Ray Systems (See figure in printed edition.) Source: FAA. (See figure in printed edition.) FAA's research has shown that each device has its own unique advantages and disadvantages. According to FAA technical officials, advanced X-ray technologies currently show the most promise for detecting sophisticated explosives. Some of the X-ray devices for screening baggage borrow heavily from advances made in the medical field. Although some advanced X-ray devices can detect more sophisticated explosives, these devices are either too slow or they have high false alarm rates (i.e., they indicate that explosives are present when they are not). According to Air Transport Association (ATA) officials, these deficiencies could cause delays in processing baggage that would have a devastating effect on airlines' ability to dispatch aircraft on time. The following examples illustrate the types of technical problems and development delays that FAA has experienced with the new X-ray technologies. In March 1993, FAA tested a computerized X-ray system that cost about $4 million to develop. This project was delayed about 1 year because the equipment was too slow in processing baggage. FAA is continuing to refine this system to increase its speed and efficiency. According to FAA officials, this is the most promising technology to date. FAA spent about $4 million to develop a coherent X-ray scatter system.\1 This project was delayed about 1 year because the equipment did not meet FAA's criteria for detecting specific amounts of explosive materials. FAA has decided to stop work on this contract, but officials believe the technology shows promise and will continue to pursue it at a later date. FAA has invested about $2.1 million in a multiview, dual energy X-ray system, but the system has a high false alarm rate. As of July 1993, this project was on schedule. FAA officials told us that recent upgrades in the device's hardware and software have improved performance. FAA tested a high-resolution X-ray system in its laboratory and at two airports. A contractor provided the equipment at no cost to FAA to conduct the tests. Although FAA found that this equipment demonstrated better detection capability than that currently used at airports, it had a high false alarm rate. -------------------- \1 This is an X-ray system that uses artificial intelligence to identify crystalline features common to explosives. TRACE AND NUCLEAR TECHNOLOGIES ARE NOT PROMISING -------------------------------------------------------- Chapter 2:1.2 In addition to X-ray technology, FAA has projects that utilize nuclear and trace technologies. As of the end of fiscal year 1993, FAA had spent over $20 million on nuclear technologies but, aside from some research data, had little to show for the investment. Furthermore, key components of a nuclear device--particle accelerators--exist only in concept or have met with only brief success in the laboratory. Also, FAA's research shows that current nuclear technologies are too expensive, too large, and much too heavy for use in airports. After spending about $11 million over 5 years on a nuclear resonance absorption project, the biggest investment in any one technology, FAA canceled the project in July 1993. However, FAA officials point out that nuclear technologies have certain advantages and that the thermal neutron analysis device's capabilities remain in the same range as other devices'. FAA officials also point out that nuclear technologies may prove useful for screening cargo. The potential for applying trace technology as a primary, or stand-alone, screening device appears doubtful. The National Academy of Sciences concluded that trace devices may have high false alarm rates and are not suitable as a primary method for detecting explosives in checked baggage. Therefore, trace devices would have to be used in conjunction with or to supplement other detection equipment to examine unopened baggage or cargo. FAA officials believe that trace technology may ultimately prove useful in detecting explosives on individuals or in small objects. FAA is working to develop protocols for a trace detection system for carry-on baggage as well as a trace portal system for screening passengers. In April 1993, FAA conducted a detailed review of eight trace systems. FAA concluded that four systems may have future potential, but it is considering canceling the other four projects (whose total costs exceed $5.6 million). For example, FAA plans to terminate its efforts on a trace device that uses spectroscopy technology. FAA found that the equipment could not differentiate between explosives and background materials. This system, which so far cost more than $485,000, is 1 year behind schedule because of technical problems. In August 1994, FAA plans to review the trace technologies to determine whether they should be used to screen carry-on baggage and electrical items rather than checked baggage. Meanwhile, FAA plans to award a $1.6 million grant to a university to continue research on the use of dogs to detect explosives. FAA plans to focus on training requirements and standardized testing of dogs and their handlers. THE OUTCOME OF FAA'S EFFORTS TO IMPROVE AIRCRAFT SURVIVABILITY IS UNCERTAIN ---------------------------------------------------------- Chapter 2:2 FAA's research on aircraft survivability techniques may offer the potential to significantly reduce the effects of in-flight explosions. However, it is uncertain when such techniques will be in widespread use. FAA plans to spend over $27 million on aircraft survivability research through fiscal year 1998 to (1) refine blast-resistant luggage containers; (2) assess the vulnerability of aircraft to different types and quantities of explosives; and (3) identify techniques to harden aircraft structures to withstand explosions. Although FAA may complete its efforts to develop more blast-resistant luggage containers in fiscal year 1994, it will probably not demonstrate its efforts to harden structures until the next generation of aircraft enter service. BLAST-RESISTANT LUGGAGE CONTAINERS PRESENT SOME UNRESOLVED ISSUES -------------------------------------------------------- Chapter 2:2.1 FAA's efforts on blast-resistant luggage containers may bridge the gap between the capability of existing detection technology and the types of blasts that aircraft can survive. In January 1992, FAA began testing prototype containers. Some new containers take advantage of technology and material developed several years ago by the military, such as Kevlar, which is now used in armored vehicles. Tests completed to date demonstrate that it is feasible to contain the effects--blast and fragments--of an internal explosion. The airline industry, however, has raised questions about the containers' cost, weight, and durability. Airline security officials point out that containers now used throughout the world (between 350,000 and 400,000) are generally made of aluminum, are frequently damaged by forklifts, and are exposed to a wide range of harsh weather conditions. Industry officials have similar concerns about the durability of the new containers. We testified in April 1993 that the containers that FAA is testing would add an average of 3,200 pounds to an aircraft's weight, thereby increasing fuel usage and operating costs. FAA is conducting research to reduce the cost and weight and ensure the durability of the containers. Figure 2.2: FAA Prototype of Blast-Resistant Luggage Container (See figure in printed edition.) Source: FAA. (See figure in printed edition.) As of September 1993, FAA had evaluated one manufacturer's design and was developing performance requirements for industry to use in building the containers. FAA is also working with industry to resolve operational concerns. For example, FAA plans to award a $1.2 million grant to an industry association, the Great Lakes Composite Consortium, to evaluate other manufacturers' designs. The consortium will also assess such factors as weight, operability, repair and maintenance needs, and associated costs. FAA officials are optimistic that the agency's research efforts can make the weight and life-cycle costs of the new containers competitive with the aluminum containers currently in use. Because of their size, hardened containers can be used only on wide-body aircraft that typically fly international routes. Wide-body aircraft in operation or on order comprise about 29 percent (4,435) of the aircraft worldwide (15,470). However, nearly 75 percent of the 57 bombings known to have taken place between 1971 and 1991 occurred on narrow-body aircraft that do not use containers to store checked baggage. Therefore, it is questionable whether hardened containers will have a major impact on increasing aircraft survivability until more wide-body aircraft are in service. FAA officials also pointed out that in about half of all successful bomb attempts, the device was not placed in the cargo hold. Nevertheless, FAA officials are optimistic about the prospects for the new containers because most aircraft flying from Europe--a high-threat region--are wide-body aircraft. FAA officials further noted that some narrow-body aircraft may be able to use the new containers. They also pointed out that the United Kingdom is conducting research on containers for narrow-body aircraft. FAA has not yet decided on the best approach for introducing the new containers or analyzed the financial impact on the industry of requiring their purchase. One official speculated that FAA could mandate the use of the new containers through a gradual phase-in that was consistent with airlines' schedules for replacing older containers. Plastic and aluminum containers currently in use last about 4 years. Some FAA officials believe that airlines will purchase the new containers without FAA's mandating their use if questions about the containers' cost, weight, and durability can be resolved. However, airline officials believe that FAA will have to mandate the containers' use and develop a reasonable timetable for their purchase. DOT officials noted that if FAA does not succeed in reducing the containers' weight and cost, it will likely have to mandate their use. DOT officials also noted that significant improvements might be obtained by using less then a full complement of containers--one or two--for suspicious luggage, cargo, or mail. FAA expects to approve design specifications for the containers by the end of fiscal year 1994. FAA IS ASSESSING BLAST-MANAGEMENT ISSUES -------------------------------------------------------- Chapter 2:2.2 In addition to developing hardened containers, FAA is pursuing other research to improve the survivability of commercial aircraft. Begun in late fiscal year 1991, this research is relatively new for FAA. Although the Department of Defense has considerable data on how explosions affect military aircraft, little information exists on how internal explosions affect the structural integrity of different commercial aircraft. As a first step, FAA is researching the effects of explosions on commercial aircraft and on such important components as flight controls. For example, FAA officials told us that an unknown concern is the effect of explosions on modern fly-by-wire aircraft.\2 The linchpin of FAA's aircraft-hardening efforts is an analysis of how explosions affect commercial aircraft structures. This analysis--known as a vulnerability assessment--will shape the direction of future efforts to improve aircraft survivability but will not be completed until fiscal year 1996. FAA expects to determine, among other things, how much damage is inflicted by internal blasts on different types of aircraft and how much explosive material is needed to destroy a commercial aircraft. According to FAA officials, the results of this effort could lead to important improvements in aircraft survivability. These improvements may include (1) placing special linings, or blankets, in baggage compartments; (2) using special composite materials to harden aircraft structures; (3) placing blow-out panels in the airframe;\3 and/or (4) protecting such critical subsystems as the flight controls. To help guide research on aircraft survivability, FAA is leveraging the Department of Defense's research on the survivability of military aircraft. According to FAA and Defense officials, techniques developed for military aircraft over the past 20 years may offer some promise. However, Air Force and FAA researchers also point out that significant additional research will be required because threats to commercial and military aircraft differ. For example, the military's experience is based on the explosion of a projectile outside an aircraft--not inside the fuselage from a terrorist device. In June 1991, FAA awarded a $3.8 million contract to the Air Force's Wright Patterson Laboratory to establish a technology base and a methodology for assessing the vulnerability of commercial aircraft and to provide technical assistance in developing a plan to implement the techniques that are selected. The Laboratory is also performing tests using explosives to blow up retired military aircraft to gain a better understanding of how explosions affect the structural integrity of aircraft. The Air Force expects to complete its analysis in June 1994. In addition, FAA is working with Boeing and McDonnell Douglas to take advantage of the aircraft manufacturers' skill, expertise, and intimate knowledge of commercial aircraft design. According to FAA and Defense officials, aircraft manufacturers' participation in aircraft survivability research will help speed the development and introduction of new techniques. Since 1990, FAA has been trying to obtain design data from aircraft manufacturers to assess the vulnerability of commercial aircraft to explosions. However, the manufacturers, who claimed that aircraft design data represent proprietary information, were initially reluctant to provide the data to FAA, thereby delaying the program about 1 year. To resolve the problem, FAA sponsored the formation of the National Institute for Aerospace Studies and Services that comprises three aircraft manufacturers. In February 1993, FAA awarded the group a $1.6 million grant to assess the vulnerability of a wide-body aircraft to an internal explosion. It is uncertain when aircraft-hardening techniques will be implemented. Depending on the technique chosen, protective measures could be retrofitted onto existing aircraft or designed for the next generation of aircraft. According to some industry officials, blast-resistance techniques will most likely be incorporated into the next, rather than the existing, generation of aircraft. Even if developed soon, these techniques will be available too late to be incorporated into such aircraft as the Boeing 777. FAA is currently ensuring that this aircraft meets minimum safety standards before certifying that it can be operated in the United States. FAA expects to certify the Boeing 777 in May 1995. -------------------- \2 Fly-by-wire aircraft rely on software-based systems to monitor and control functions traditionally performed by cockpit crews. In many cases, software-based systems have virtually replaced the hydraulic and mechanical systems used on earlier generations of aircraft. \3 Blow-out panels can divert the force of an explosive device outside the aircraft and away from passengers, crew, and critical aircraft components. FAA IS CONSIDERING COMMERCIALLY AVAILABLE EQUIPMENT AS AN INTERIM SOLUTION ---------------------------------------------------------- Chapter 2:3 FAA has reached a crossroads in its security RE&D program. Since FAA is several years away from approving new explosive detection equipment for checked baggage, FAA is considering whether to allow airlines to use commercially available equipment to provide improved capability--a step some foreign governments have taken. Some DOT, FAA, and industry officials as well as FAA's Aviation Security Research and Development Scientific Advisory Panel, believe that this action will provide not only an interim capability against terrorist attacks but also an opportunity to operationally test security equipment at airports. According to a June 1992 report and a November 1993 strategy paper presented to the FAA Administrator by the Aviation Security Research and Development Scientific Advisory Panel, FAA's RE&D program does not emphasize immediate and near-term technological solutions as needed to satisfy the intent of the Aviation Security Improvement Act. The Panel, concerned about FAA's progress, noted that "a terrorist attack could occur at any time and it is only a matter of time until a new terrorist act against civil aviation involves the significant loss of American lives. No new devices will be available in the foreseeable future that are both 100 percent effective and reliable. FAA could take an additional 2 to 5 years to approve equipment for airlines' use under its existing process. FAA could use commercially available equipment as an interim threat response measure." The Panel recognized that commercially available equipment would not fully meet FAA's performance requirements but believed that its use would increase detection capability, provide an opportunity to operationally test the equipment, and address the basic intent of the Aviation Security Improvement Act. Therefore, the Panel proposed that FAA use about $8.4 million to purchase systems for use by U.S. carriers at three or four foreign airports, where FAA believes the threat to aviation is greatest, as well 200 hardened luggage containers, costing about $2 million. Some airline security experts are frustrated by FAA's lack of progress and point out that several foreign governments and their aviation authorities have moved faster than FAA. According to these experts, these countries are testing commercially available equipment at selected airports and have incorporated the technology into airports' and carriers' operations. Although these devices have limitations and cannot meet FAA's performance standards, the foreign governments have decided that threat levels warrant their use. FAA officials told us that differing regulatory structures and less stringent standards for devices have allowed foreign governments to take these actions. However, some airline security directors still believe that FAA should follow the example of the foreign governments and test commercially available equipment. In a November 1993 letter to the FAA Administrator, the Aviation Security Research and Development Scientific Advisory Panel also expressed concern about FAA's lack of progress and leadership in light of foreign governments' initiatives to install and operate advanced U.S. technology at their airports. DOT's former Director of Security and Intelligence also believed that FAA should acquire improved explosive detection equipment. The former Director and other current officials within that office believed FAA cannot afford to wait several more years until new technology is available. These officials noted that foreign governments are, with FAA's assistance, testing "off-the-shelf" explosive detection equipment at high-threat foreign airports and believe that FAA should use the same approach at selected domestic airports that are gateways for international flights. DOT and FAA officials with whom we spoke believe that such testing should have a clearly defined end point to minimize the time needed to introduce the new technology. These officials could not, however, estimate the time needed to gain the necessary data on performance. Since November 1990, FAA has assessed eight commercially available systems to determine their effectiveness for screening such electrical items as radios in carry-on baggage. Although the detailed results of the tests are classified, FAA's test results indicate that the performance of commercially available equipment has limitations. In June 1992, FAA notified the eight companies that their systems had been approved for operational tests by airlines. However, only one airline volunteered to participate in this effort because of the shortcomings of the various devices. FAA officials believe that airlines' reluctance to participate was more a function of the cost to purchase the equipment (X-ray systems cost $35,000 to $300,000, and trace devices cost $21,000 to $1 million) rather than the shortcomings of the devices. Because the industry's participation was so limited, FAA initiated a program in 1993 to test commercially available trace detection equipment for screening electrical items--not checked baggage--at several domestic airports, such as La Guardia Airport in New York. FAA requested $1.5 million for fiscal year 1994 to continue this project. In its budget request, FAA noted that this project would provide valuable information on the performance of trace technologies in an airport environment while also providing more protection for passengers. FAA officials said they had developed a plan to install commercially available equipment at category X airports on an interim basis. One FAA study estimated that it would cost about $50 million to equip the 19 category X domestic airports with new devices, as well as with such other security equipment as hardened luggage containers. However, FAA postponed going ahead with this plan, deciding instead to place greater emphasis on developing and publishing the performance standards for new explosive detection devices for screening checked baggage. Although commercially available equipment has performance limitations, FAA technical officials believe that important information, particularly on how to integrate various devices, can be gained from testing such equipment at airports. CONCLUSIONS ---------------------------------------------------------- Chapter 2:4 FAA faces several difficult and important decisions about the development and implementation of new security technology. In view of the uncertainty surrounding the near-term introduction of advanced detection technology and methods to improve aircraft survivability, a dual-track strategy may be the most prudent course of action that FAA could adopt at this time. Specifically, FAA could test commercially available equipment at airports while also continuing to develop, evaluate, and certify advanced explosive detection equipment and methods for improving aircraft survivability. Although using commercially available equipment may be a stopgap measure, it would allow FAA and the industry to gain valuable experience in using security equipment at airports and could help guide future decisions. However, FAA would have to monitor these systems carefully to determine what increased capability they provide. If the equipment enhances security, then FAA's expanding this effort to other airports may be warranted. RECOMMENDATION ---------------------------------------------------------- Chapter 2:5 We recommend that the Secretary of Transportation direct the FAA Administrator to assess the effectiveness of commercially available explosive detection equipment for screening checked baggage by acquiring and testing such equipment at a limited number of domestic airports. AGENCY COMMENTS AND OUR EVALUATION ---------------------------------------------------------- Chapter 2:6 FAA concurred with our recommendation concerning the testing of commercially available technology for screening checked baggage but noted that recent tests conducted at a foreign airport indicate that such equipment did not perform as well as expected. Nevertheless, FAA officials believe that valuable information can be gained and some improvements in security may be achieved by testing commercially available equipment. Although DOT and FAA officials generally agreed with the information in this chapter, they offered the following comments. FAA's approach for developing new explosive detection devices is based on requirements set out in the Aviation Security Improvement Act. Although FAA officials agreed that progress in developing new explosive detection technology has met several technical obstacles, they pointed out that the development of new detection equipment requires significant research and entails considerable risk. Technologies that appear to have promise in the early stages of development may eventually prove ineffective. Some technologies turn out to be cost- or size-prohibitive. In addition, although some technologies detect explosives, they may not be able to detect them in the amounts FAA requires; and some may meet the detection requirements but not be able to operate fast enough to be operationally practical. We agree with FAA that research, by its nature, entails risk, and we believe that our report appropriately recognizes such risk. In addition, FAA officials pointed out that the investment in nuclear technologies was driven by the fact that such devices remain the best (and in some cases the only) ones capable of detecting certain explosives as required by FAA's performance standards. These officials also noted that nuclear technologies provide significant advantages for screening cargo and are the "best hope" for developing cargo screening systems. Moreover, FAA officials indicated that the investment in trace technology was made consistent with the approach advocated by a number of FAA advisers and that emphasis on the program was reduced after the National Academy of Sciences concluded that it could not develop protocols for testing such devices as the law requires for certification. In addition, FAA recognizes that trace technology is not suitable as a primary screener for checked baggage and that considerable challenges remain in developing standards for such technologies. However, these officials point out that trace technologies show promise for detecting explosives in carry-on baggage and on people. FAA COULD IMPROVE ITS CERTIFICATION PROCESS AND SECURITY RE&D PROGRAM ============================================================ Chapter 3 Although FAA has made some progress in promoting the development of new explosive detection equipment, additional actions are needed to improve its certification process and its security RE&D program. FAA's certification process for new explosive detection equipment does not ensure the performance and reliability of new systems.\1 FAA does not plan to test devices at airports as part of the certification process but plans, instead, to rely on tests by contractors and the FAA laboratory to determine the performance of the new equipment. Both of these approaches have significant shortcomings. Furthermore, FAA's performance standards for new detection equipment do not include reliability criteria even though the reliability of equipment can have a significant impact on airlines' operations. In addition, FAA continues to invest in trace technologies without having defined performance standards for evaluating and certifying such equipment. We also identified several weakness in FAA's security RE&D program. Specifically, FAA does not (1) conduct software reviews, (2) pay sufficient attention to systems integration issues, and (3) place enough emphasis on human factors, such as how operators will work with new detection devices. Without adequate attention to these factors, FAA cannot make informed decisions about the direction of current and future efforts. -------------------- \1 Reliability is the length of time that explosive detection equipment should operate without failure. For example, the Department of Defense commonly uses "mean time between failure" as a measure of reliability for military equipment. FAA'S PROCESS FOR CERTIFYING NEW TECHNOLOGY HAS WEAKNESSES ---------------------------------------------------------- Chapter 3:1 FAA's process for certifying new explosive detection equipment for checked baggage is the key to ensuring that the new technology can meet the terrorist threat. However, FAA's process does not include testing the new systems at airports as a condition of certification, FAA's performance standards do not set reliability criteria for new devices, and FAA has not developed performance standards for trace technology. Under FAA's planned approach, the agency runs the risk of approving devices that cannot reliably detect sophisticated explosives under actual airport conditions. CERTIFICATION PROCESS DOES NOT INCLUDE OPERATIONAL TESTS -------------------------------------------------------- Chapter 3:1.1 In September 1993, FAA issued its plan for certifying bulk detection systems (nuclear and advanced X-ray technologies). FAA's certification plan defines the process, performance requirements, and testing standards for vendors to obtain approval for explosive detection devices and systems. The plan, however, does not include airport testing--a key step to ensure that new equipment works and to boost the airline industry's confidence in the equipment. FAA's Director, Office of Civil Aviation Security Policy and Planning, believes that major obstacles preclude FAA's performing such tests during the certification process. These obstacles include the additional time and cost required to conduct the tests, airport operators' concerns about using real explosives for tests, and unique airport operating environments that make it difficult to select representative test locations. However, the airline industry, the National Academy of Sciences, and others believe that airport testing must play an important role in FAA's certifying new detection technology for the industry's use. The Aviation Security Improvement Act states that the FAA Administrator cannot require airlines to purchase any explosive detection equipment unless test data demonstrate that the devices can perform effectively under realistic operating conditions. Furthermore, the National Academy of Sciences reported in March 1993 that testing explosive detection devices against FAA's performance standards under realistic operating conditions must be the keystone of FAA's certification process. The Academy's report stated that FAA's certification process must ensure that each explosive detection device used at an airport will perform at least as well as the one that passed the certification test. The Academy noted that performance at an airport could differ significantly from performance in a laboratory. FAA officials believe that realistic operating conditions can be simulated at the FAA Technical Center. However, this approach is questionable because tests at the FAA Technical Center's laboratory are conducted under sterile and controlled conditions without the distractions found in an airport. For example, a single Boeing 747 can carry between 370 and 400 passengers and their luggage. FAA cannot duplicate the stress and activity presented by such numbers of passengers in its laboratory. Also, FAA can control the environmental conditions (i.e., temperature and humidity) in the laboratory that can affect a device's performance. Airports do not have this luxury. During our observation of FAA's tests on four devices at Miami International Airport, which were conducted in the baggage area underneath the terminal, we noted heat, humidity, and dirt--factors that can and did affect performance and reliability. As an alternative to conducting its own tests, FAA, in its certification plan, requires contractors to test equipment at airports and submit data on the results to FAA. However, this approach has several weaknesses that may affect the outcome of the tests. First, FAA expects the contractors to screen passengers' baggage. The contractors will not test baggage with either real or simulated explosives. Therefore, the contractors cannot provide FAA with data on the equipment's detection performance. Second, FAA does not require contractors to use a prototype model representative of a production unit or to gather data on the equipment's reliability as part of the test. An FAA testing official told us that the performance and reliability of a laboratory model could differ significantly from those of an advanced prototype. Third, FAA does not plan to witness any of the tests to verify the results. The airline industry and others believe that it is important for FAA to conduct thorough and consistent testing to ensure that each explosive detection device approved by FAA meets a minimum standard of performance. As discussed earlier, the airline industry is skeptical about FAA's ability to test equipment because it did not rigorously test the thermal neutron analysis device before deciding to use it at airports. According to airline and ATA officials, exaggerated and confusing claims made by competing equipment vendors make it difficult for the industry to choose the best equipment. ATA officials also note that an extensive, realistic operational evaluation of explosive detection systems should be an indispensable condition for certifying equipment. ATA believes that a 1-year evaluation at all of the domestic category X airports would be appropriate. Furthermore, the Office of Technology Assessment's January 1992 report criticized FAA because it did not plan to conduct adequate operational tests before approving equipment. Moreover, officials from DOT's Office of Security and Intelligence told us that it is absolutely essential for FAA to operationally test new equipment. According to these officials, FAA can overcome the obstacles to operational tests at airports. For example, FAA could use real explosives in laboratory tests and simulants (fake explosives) in airport tests. According to FAA officials, simulants can provide adequate and reliable results. DOT officials also believe that FAA could select representative airports for the tests and work closely with airport and airline officials to overcome other obstacles. Also, they believe that airport testing would give the airline industry experience in using the equipment. They believe that close cooperation with the industry is essential to make FAA's program effective. According to FAA officials, the certification testing and evaluation of an explosive detection device in FAA's laboratory will take about 3 months, during which time officials will evaluate the vendor's data, conduct the test, and prepare a report. The actual testing of a device at FAA's laboratory will take only about 1 week. Testing the devices at airports, according to FAA Technical Center officials, would add 6 months to the certification process. Neither FAA headquarters nor Technical Center officials could estimate the costs of conducting tests at airports during the certification process. According to DOT officials, valuable information and experience about the performance of new equipment would be gained from operational testing; these benefits would counterbalance the costs of conducting the tests. CERTIFICATION STANDARDS DO NOT INCLUDE RELIABILITY REQUIREMENTS -------------------------------------------------------- Chapter 3:1.2 FAA does not plan to test the detection equipment's reliability during the certification process and did not include specific criteria for reliability in its certification standard. Therefore, FAA cannot assure airlines that the equipment will operate without failure for a reasonable period of time and will not disrupt airport operations. Conceivably, FAA could approve a system without knowing how often it would break down. According to an official in the Office of Civil Aviation Security Policy and Planning, FAA's certification standard does not include criteria for reliability because FAA is developing technology that does not have a performance history. Therefore, FAA believes it is difficult to develop absolute numbers to use as criteria. This official also noted that the certification process would take too long if FAA tested the equipment's reliability. Furthermore, the official noted that the Aviation Security Improvement Act requires FAA to establish detection requirements, not reliability criteria. The reliability of new explosive detection equipment is important because it can significantly affect airlines' operations. According to ATA officials, the airlines have learned from years of experience that unreliable security equipment will disrupt their operations. ATA believes that all equipment must meet clearly defined operational and maintenance standards and recommended that FAA require contractors to show that a system can be economically operated and maintained before it is certified. Other federal agencies that develop new technology routinely, such as the Department of Defense, establish requirements for equipment's reliability on the basis of operational needs. These requirements provide developers with criteria to ensure that new equipment will not fail frequently when it is placed in service. FAA officials, however, believe that they should not establish requirements for reliability before knowing the equipment's capabilities. Our observations of new technology and the conditions under which it must operate confirm the need for reliability standards and testing. We observed that equipment failed during FAA's tests at Miami International Airport.\2 Specifically, FAA could not operate two of the four test devices for 2 days. Moreover, FAA had to suspend the tests until the equipment failures were resolved. These brief tests raise serious questions about the durability and reliability of the new security technology and its impact on airlines' operations. -------------------- \2 The purpose of these tests was to examine the characteristics of baggage, not the ability of the devices to detect explosives. FAA LACKS PERFORMANCE STANDARDS FOR TRACE DETECTION TECHNOLOGY -------------------------------------------------------- Chapter 3:1.3 Problems persist in defining performance standards to evaluate the ability of trace technology to detect explosives in an airport environment. The Aviation Security Improvement Act requires FAA to develop performance standards that include conducting tests of the equipment in accordance with protocols developed in consultation with outside experts. To meet this requirement, FAA contracted with the National Academy of Sciences in May 1992. After grappling with technical issues for almost a year, the Academy advised FAA in its March 1993 report that it could not define performance standards for trace detection systems because of the difficulty in discriminating between very small traces of explosive material and much larger quantities of other materials in an airport terminal. The Academy also noted that the equipment needed to test trace detection devices is not available. FAA has recently taken over the task of defining performance requirements for trace technology and is consulting with industry and academia to formulate an acceptable standard. FAA officials could not estimate when they would complete this effort. Although FAA has not defined performance standards for trace detection equipment, it plans to invest about $5 million in fiscal year 1994 on such devices. FAA officials noted that although concerns exist about the feasibility of using trace detection technologies to screen checked baggage, such devices show promise for screening passengers, carry-on baggage, and electrical items. DOT officials are concerned that FAA is attempting to develop this technology without providing vendors with specific performance requirements for its development. FAA'S SECURITY RE&D PROGRAM HAS WEAKNESSES ---------------------------------------------------------- Chapter 3:2 Because of the changing nature of the terrorist threat and the technical challenges facing FAA, the agency will be conducting security research well into the foreseeable future. However, FAA's security RE&D program has several weaknesses that will hinder its ability to guide investment decisions and speed the development of new technology. Specifically, FAA does not (1) conduct software reviews to evaluate automated functions that control the performance of equipment, (2) give sufficient attention to integrating different technologies into a synergistic system, and (3) place adequate emphasis on human factors when developing new detection devices. FAA IS NOT EVALUATING THE PERFORMANCE OF CRITICAL SOFTWARE IN NEW SYSTEMS -------------------------------------------------------- Chapter 3:2.1 Currently, FAA technical staff do not evaluate software that performs explosive detection system functions even though automation is a major element of the new technology. A major objective of FAA's security RE&D program is to automate systems, thereby improving airlines' ability to detect explosives and process baggage and minimizing reliance on human screeners to detect explosives. Therefore, FAA is developing devices that utilize sophisticated software to determine whether a suspicious object requires closer scrutiny. The effectiveness of the software's design can dramatically affect the performance of explosive detection equipment. For example, the speed with which the software analyzes baggage for explosives is critical for the dual energy computerized X-ray tomography system. If the analysis of the baggage is too slow, much greater computer power may be required to speed up this system, and the device's software may need to be modified. FAA has not reviewed the performance of the software to resolve this issue. In the past, we noted that the effectiveness of software is the Achilles' heel of FAA's technology programs and has caused considerable delays.\3 As FAA moves towards integrating explosive detection systems, software becomes even more important because it is a critical factor in making systems work together. Instead of evaluating software, FAA technical officials review features of the hardware's design and do not determine whether the system's performance can be optimized and development costs reduced by changing the software's design. FAA officials advised us that they rely on contractors' progress reports to monitor the development of software. These officials also noted that agency staff within the security RE&D program lack the necessary expertise to evaluate software. However, other organizations within FAA, such as the Systems Engineering and Configuration Management group, have such expertise. Therefore, the security RE&D staff could draw on this office for advice and assistance in evaluating software. -------------------- \3 Air Traffic Control: Continuing Delays Anticipated for the Advanced Automation System (GAO/IMTEC-90-63, July 18, 1990) and Air Traffic Control: Ineffective Management Plagues $1.7 Billion Radar Program (GAO/IMTEC-90-37, May 31, 1990). FAA IS NOT PLACING ENOUGH EMPHASIS ON SYSTEMS INTEGRATION -------------------------------------------------------- Chapter 3:2.2 Integration may offer FAA opportunities to develop a system that can reliably detect sophisticated explosive devices by overcoming the technical shortcomings of individual devices. However, FAA is not emphasizing integration because the agency believes this task should be left to the airlines to perform. We have previously reported that systems integration is a major factor in determining the success of FAA's research efforts--including the development of new security technology.\4 Since no single explosive detection system can currently meet all of FAA's requirements, FAA, the National Academy of Sciences, the Office of Technology Assessment, and others recognize that several technologies will have to be combined to achieve the agency's performance requirements. Combining devices can mitigate specific shortcomings. For example, a device that is slow in processing luggage but can reliably detect explosives can be combined with one or more devices that are faster but more prone to false alarms. If a bag sets off an alarm, then the slower but more effective device can be used to investigate the luggage. In its March 1993 report, the National Academy of Sciences noted that individual detection devices can be integrated into a system that takes advantage of the strengths of each method. The Academy's report stated that explosive detection technology is continuing to advance and that several devices show promise. Furthermore, the report noted that these devices will become essential building blocks for an explosive detection system that could reasonably be installed in airports. The Academy observed that a large range of possible performance and cost options exist, depending on the system chosen. Integrating systems is important for placing new devices in the overall framework for security at airports as well as for combining the devices' operation with the flow of passengers. New devices must operate in a dynamic environment where weight, size, maintenance, and volume of passenger traffic are important factors. In June 1992, the Aviation Security Research and Development Scientific Advisory Panel noted that additional research is needed on how new detection equipment would affect the flow of passengers and the procedures currently in place at airports. FAA plans to rely primarily on the airlines to integrate individual explosive detection devices into systems after the agency approves the devices. FAA believes that airlines are in the best position to decide which devices can meet their needs. However, in our discussions with FAA, DOT, and industry officials, we identified several concerns with this approach: First, this approach overlooks technical factors that FAA should address when designing new systems. A DOT technical official told us that automated devices should be designed so that their software is compatible with other devices to readily exchange information. For example, two different enhanced X-ray devices must be able to share information on the location of suspicious objects and determine whether the screener needs to closely examine the object. Second, DOT and industry officials indicated that explosive detection equipment may not have matured to the point that FAA can rely on others to refine the technology. These officials said that systems will need to be integrated throughout the next decade as equipment improves. Third, FAA's approach assumes that several devices will meet the agency's requirements for detecting explosives and that airlines will enjoy the luxury of selecting devices that meet their specific operational needs. However, it appears unlikely that several candidate devices will be available in the near future. Therefore, the choices may be quite limited, and the best approach may be to find systems that can work effectively together. Lastly, it is questionable whether the airline industry has the financial resources to conduct the analysis and research needed to craft an acceptable system. Although the airlines are willing to participate in tests, officials doubt that they can afford the research associated with systems integration. As discussed later, the cost of new explosive detection technology is significant. In 1991, the President's Commission on Aviation Security and Terrorism noted that FAA's reliance on industry and market forces to develop new technology was unfounded. DOT officials believe that this is still the case and note that FAA--not the airlines--must be the engine that drives systems integration. FAA's own experience indicates that integration features should be designed into equipment during development--not as an afterthought. For example, in November 1992, FAA tested a computerized X-ray tomography system and found that it was too slow in processing baggage and had a high false alarm rate. As a result, FAA modified the contract and provided an additional $979,500 for the contractor to identify other devices that could be integrated to screen baggage. This modification will likely involve changing software to integrate the system with another device and delay the project about 1 year. The FAA technical official who manages this project stated that, in this case, it might have been cheaper to initially develop compatible software rather than change systems after completing development. FAA, DOT, and airline officials with whom we spoke believe that FAA could have made significant progress if it had focused adequate attention on integrating devices. Moreover, FAA technical officials believe that FAA may have lost between 18 months and 2 years in developing new technology because it did not pay adequate attention to systems integration. -------------------- \4 Aviation Research: FAA Could Enhance Its Program to Meet Current and Future Challenges (GAO/RCED-92-180, June 3, 1992). FAA HAS NOT FOCUSED ADEQUATE ATTENTION ON HUMAN FACTORS -------------------------------------------------------- Chapter 3:2.3 The security of the traveling public rests on a careful blend of technology, procedures, and policies. Developing new explosive detection devices is only part of the solution--improving security also involves people. The introduction of new explosive detection equipment represents the next step in the evolution of aviation security after the introduction of metal detectors. In FAA's, DOT's, and other security experts' view, careful attention to human factors--such as the effectiveness of the people operating the new devices--is necessary to complement the technology. Moreover, the Aviation Security Improvement Act directed FAA to explore ways of enhancing human performance in aviation security. We recently reported on the importance of human factors in security and recommended that FAA pay greater attention to screeners' proficiency, airport employees' awareness of security concerns, and passenger profiling (interviewing).\5 As new explosive detection devices are installed, research on human factors will become critical to ensure that operators can effectively use the new equipment. Although FAA intends to automate new devices, it is doubtful that technology can, in the near term, completely replace screeners. Indeed, operators of new equipment will find it difficult to interpret alarms and detect artfully concealed explosives. DOT and airline officials as well as FAA's own Aviation Security Research and Development Scientific Advisory Panel believe that FAA should place greater emphasis on several human factor issues associated with the new explosive detection devices. First, operators of the new devices will have to be vigilant and able to make important decisions. The probability of a screener's finding a bomb in a piece of luggage is very low--characterized by airline experts as a "one in a billion" chance or a search for the proverbial needle in a haystack. Effective screening requires airline employees to be very alert and motivated over extended periods. Techniques for training, selecting, and motivating operators need to be explored and updated routinely. Second, how the operator interfaces--or works--with the new devices is critical. Screeners must be able to read the displays easily, work with the machines, and understand the alarms. Important actions are under way in this area. Manufacturers of new X-ray equipment have begun developing test programs to be built into devices for screeners. One manufacturer has developed a system that can insert the image of an explosive device in a piece of baggage on the screener's display. The screener is alerted by the computer that the image is a test object before any action can be taken. This type of test shows promise, but when it will be in widespread use is uncertain. Lastly, FAA, DOT, and airline security experts believe that the new devices must be used in conjunction with passenger profiling. Profiling, which is a method of separating potentially threatening individuals from other travelers through an interview, is credited with preventing a terrorist act against a foreign carrier in 1986. Currently, profiling is done only on some international flights and is based on several key questions. Officials from one airline with whom we met are pilot-testing an automated profiling system that works from a new perspective--it seeks to screen-out nonthreatening passengers. FAA recently began working with the airline to refine this system. As the President's Commission pointed out, FAA has not paid adequate attention to human factors and training. Although FAA is now more aware of human factors, airline and airport officials believe that top-level FAA management is still emphasizing technological solutions to security problems, such as developing new explosive detection systems. According to FAA officials, before the Pan Am 103 tragedy and the act's passage, most research centered on detecting weapons and explosives--not on human factors. FAA has had difficulty developing an effective human factors research program because of the high turnover rate in a key staff position at the FAA Technical Center. According to officials, FAA funded some human factors research in fiscal year 1993 with funds from other security projects. In fiscal year 1994, FAA plans to more than double its human factors effort to, among other things, examine and enhance screeners' proficiency. -------------------- \5 Aviation Security: Additional Actions Needed to Meet Domestic and International Challenges (GAO/RCED-93-38, Jan. 27, 1994). FAA FORMED A TASK FORCE TO EXAMINE SECURITY RE&D ISSUES ---------------------------------------------------------- Chapter 3:3 In response to concerns expressed by the new Assistant Administrator for Aviation Security and the National Academy of Sciences, FAA formed an in-house task force in January 1994 to accelerate its short-term efforts to approve new explosive detection equipment for the industry. This task force will (1) assess current explosive detection technology, (2) develop information for certification testing, and (3) simulate, through computer modeling, explosive detection systems and their impact on airport operations. The most important element of FAA's initiative is an emphasis on simulation modeling. FAA intends to rely heavily on computer modeling of airport operations, particularly of baggage processing, to develop information on systems integration, the operational impact of new devices on airline operations, and the cost of new technology. FAA also plans to use the results of research being conducted by the United Kingdom on commercially available technology. According to FAA officials, by using simulation modeling, the agency will be able to examine total life-cycle costs for individual devices, develop a range of cost estimates for combinations of devices, and enhance ongoing research on human factors. The task force's plans are ambitious. FAA expects to begin laboratory simulations in May 1994 and to determine in September 1994 whether an airport demonstration is necessary to validate the simulations. In January 1995, FAA plans to decide whether additional program changes are warranted. Recently, FAA officials publicly commented that they do not expect manufacturers of new devices to be able to meet the performance standards for screening checked baggage and that they will decide in January 1995 whether to hold to the current performance standards or adopt an interim standard. However, FAA officials would not comment on this issue during our review and said they planned to hold to the current performance standards. According to FAA officials, although some computer models of airline operations exist, additional work is needed to develop a model that can incorporate passenger-processing times and an individual device's performance. In its March 1993 report, the National Academy of Sciences cautioned that simulation modeling must be carefully thought out and cannot be substituted for rigorous testing at airports under "real world" conditions. The Academy noted that "Simulation modeling is not a panacea. The development of good simulation models is an expensive, time-consuming effort which requires the dedication of high-caliber experts. Simulation can give very good, or very bad, results, depending on how it is used and how faithfully the underlying simulation models represent the 'real world.' The results of simulation, by their nature, must be imprecise, but there may be a tendency to attribute greater precision to the numerical results than is warranted." FAA's initiative should provide some information about the feasibility of integrating the operation of new devices with the flow of passengers and about the potential costs of new systems. However, it will not address software, systems integration, and human factors issues in current and future security technology. FAA officials noted that it may take as long as 1-1/2 years to fully develop simulation models to explore systems integration issues. CONCLUSIONS ---------------------------------------------------------- Chapter 3:4 FAA has made some progress in developing new detection technology. However, improvements are needed in its certification process and other aspects of its security RE&D program to ensure the development of new technology in a timely manner. Specifically, FAA does not plan to test devices at airports during the certification process. Although adding airport tests to contractors' and the FAA laboratory's tests may increase the cost and time for FAA to certify new equipment, such tests are necessary to ensure that the equipment meets the agency's defined threat and will operate reliably when used at airports. Explosive detection equipment that cannot operate reliably will disrupt airlines' operations, increase airlines' costs to maintain and operate the devices, and jeopardize confidence in the new technology. In addition, such testing may identify problems that could ultimately forestall widespread implementation of the technology. Lastly, airport testing may engender confidence in the technology before the airline industry invests millions of dollars in new security devices. FAA faces significant technical challenges to define performance standards for trace technology. However, without such standards, FAA's investing in the development of trace technology while simultaneously attempting to define the technology's capabilities is overly ambitious. Explosive detection technology is evolving and will take time to mature. However, FAA cannot compare the performance and capabilities of new explosive detection systems and make informed decisions about future development efforts because the agency does not (1) evaluate the software that controls the equipment's operations and detection functions, (2) place sufficient emphasis on integrating systems early in the development and testing of new devices, and (3) focus adequate attention on human factors. FAA is taking an important, albeit long overdue, first step toward linking the new detection technology with airports' and airlines' operations by forming a task force to assess explosive detection technology. This effort focuses on how to accelerate the development of technology in the short term and will rely heavily on simulation modeling to gain a better understanding of the impact of new technology on airports' operations. Although this effort is a good starting point, it does not address all of our concerns. RECOMMENDATIONS ---------------------------------------------------------- Chapter 3:5 To improve FAA's certification process for new explosive detection technology, we recommend that the Secretary of Transportation direct the FAA Administrator to require operational tests of the performance and reliability of explosive detection systems at airports during certification, include reliability criteria in the certification standards for new equipment, and discontinue the development of trace technology for screening checked baggage until certification standards have been established. To further improve FAA's security RE&D program, we recommend that the Secretary of Transportation direct the FAA Administrator to evaluate software when reviewing systems' designs, place greater emphasis on integrating devices when initiating development projects, and focus on human factors associated with using new devices, especially on how operators will work with the new technology, throughout the development process. AGENCY COMMENTS AND OUR EVALUATION ---------------------------------------------------------- Chapter 3:6 FAA agreed with some of our recommendations but not with others. FAA did not concur with the need to test new explosive detection devices at airports as part of the certification process. FAA officials noted that passenger activity, distractions, and stress situations common to the airport operating environment are extraneous variables when testing fully automated equipment; indeed, their existence is the reason that FAA requires new systems to be automated. The officials also said that airport testing and demonstrations of new equipment will be conducted "as necessary" before FAA directs widespread deployment and that such tests will provide more information on the equipment's tolerance of environmental conditions and maintenance. According to these officials, once certified, even a device whose false alarm and/or baggage-processing rate was only marginally acceptable could be used at lower-activity airports. The Director of DOT's Office of Intelligence and Security and his staff believe that operational tests of candidate explosive detection systems are necessary; however, they said that these tests need not be conducted as part of the certification process. They also noted that questions of reliability and maintainability should be addressed after certification but before deployment. In our view, FAA cannot adequately portray airport conditions in its laboratory or by simulating the operation of new equipment. Furthermore, airport testing may be the key to gaining the confidence of an industry that is growing increasingly skeptical about the capabilities of the new equipment. Throughout our review, we noted a reluctance by FAA to test new explosive detection technology at airports before mandating its use. In responding to our report, FAA stated for the first time that it would test equipment at airports before deploying it. In addition, FAA has not specified the criteria it will use to determine when the devices need to be tested. In our view, FAA should conduct airport tests for all candidate systems during the certification process because testing "as necessary" will not be sufficient to gain the confidence of the airline industry. In addition, FAA disagrees that reliability criteria should be part of the certification standard. FAA officials said that the issues of equipment availability, reliability, maintainability and operating efficiency are not fundamental to their certifying the detection capabilities of the equipment. Although FAA can mandate the deployment of new detection equipment, it is not the purchaser or end user of the equipment. According to FAA officials, the economic trade-offs among purchase price, availability, reliability, and maintainability can be made only by the end user--the airlines. In our view, concerns about the availability, reliability, and maintainability of new explosive detection equipment may hinder FAA's efforts to deploy the new technology in the future. Precisely because they are concerned about these issues, airline officials emphasized the disruptive effect of unreliable equipment on their operations. Our observations of limited testing at the Miami airport confirm the validity of the airlines' concerns about the new equipment's reliability. Given the financial status of the industry and the cost of the new devices, we believe that FAA would do well to ensure that reliability is built into the new devices from the start. FAA concurs with our recommendation that efforts to develop trace detection technology should be discontinued until standards for that technology have been developed. However, FAA officials noted that trace technologies may prove useful for screening passengers and some carry-on items. Although FAA agrees that software plays a critical role in the new detection equipment, it disagrees with our recommendation that it should evaluate the software of new explosive detection devices. FAA believes that the industry should be responsible for evaluating the software systems that perform explosive detection system functions. According to officials, FAA's security RE&D program will take the technology through the testing of prototype models in its laboratory and will then transfer the technology to industry. Therefore, FAA should not be concerned with verifying computer code and/or optimizing hardware and software. In addition, for technology that the industry initially developed, FAA has had difficulty obtaining information about the hardware and software that the industry claims is proprietary. In our view, evaluating the software is a necessary complement to examining the hardware of a system. New explosive detection equipment relies heavily on software to analyze data and, ultimately, to determine whether an explosive device exists in checked baggage. Major improvements in detection may come from software refinements, and systems integration depends on linking devices, and their software, together. A closer examination of software might identify a problem that could forestall deployment of equipment in the future. Without examining software, FAA cannot ensure that the new technology is working as intended. Software problems that have occurred with other FAA-developed technology, such as the Advanced Automation System, have hindered the technology's development and delayed implementation. FAA fully concurs with our recommendation to place greater emphasis on systems integration. Officials noted that FAA had concentrated on developing a "silver bullet" but now recognizes that this solution is not feasible in the near term. According to FAA officials, without the technological developments of the past year or so, major investments of time and money in systems integration might, at best, have yielded only marginal progress, but today the technological prospects are substantially better. They also noted that the timely marriage of devices will be very difficult because of the proprietary nature of the hardware and software and other marketplace considerations. Ultimately, according to FAA officials, the federal government may have to direct efforts as it would during wartime to achieve systems integration. Notwithstanding these concerns, FAA must play a greater role in integration. Furthermore, DOT officials, FAA technical staff, and industry officials noted that significant progress could have been made if the proper focus had been placed on systems integration. In fact, two advanced X-ray technologies--if combined--show promise from a theoretical perspective for meeting FAA's requirements for screening checked baggage. Some FAA officials recognized this possiblity well over 1 year ago. DOT and airline officials are concerned about FAA's current approach and note that the airline industry may not have sufficient resources to integrate various technologies and that FAA, not the industry, must be the engine that drives systems integration. FAA concurs with our recommendation that it place greater emphasis on human factors in developing new detection technology. As evidence of its commitment, FAA provided almost $1 million for human factors efforts in 1993 and plans to spend about $2.3 million in 1994 on improving screeners' performance and training. FAA officials did not believe that our report had provided adequate information on their new initiatives to develop detection technology. They noted that the security RE&D program started during the past year to focus on the most promising technologies for the shorter term. Agency officials also noted that FAA's performance standards for screening devices for checked baggage exceed the capabilities of all but a limited number of systems and that other devices are many years away from deployment unless major technological breakthroughs occur. Thus, FAA reduced, deferred, or stopped funding for such security RE&D efforts as nuclear and trace technologies for screening checked baggage. We believe that our report appropriately captured this and other information about FAA's initiative. THE COST OF NEW SECURITY EQUIPMENT RAISES IMPORTANT ISSUES ============================================================ Chapter 4 Developing new explosive detection technology is only part of the challenge; the airline industry will also have to purchase and implement the new technology throughout the next decade. The cost and source of funds for purchasing new security equipment are important issues. The airline industry is concerned about the costs of purchasing and operating new detection devices, which, it estimates, could range from $250,000 to over $1 million per device. Because devices will probably be used in combination, the costs to acquire new security technology could skyrocket. However, FAA does not have a plan or strategy to guide the government's and the airline industry's efforts in this area. As a result, airlines cannot plan or budget for new security equipment. The Congress is considering legislation that would clarify airports' authority to purchase explosive detection equipment with Airport Improvement Program (AIP) grant funds. Several issues, such as who is responsible for the new equipment, need to be resolved before AIP grant funds are used for this purpose. THE COST OF NEW DETECTION EQUIPMENT IS A MAJOR CONCERN ---------------------------------------------------------- Chapter 4:1 Because of the precarious financial condition of the airline industry, ATA and airline officials believe that the estimated costs of the new systems alone dictate that FAA carefully evaluate their operational and economic implications as part of the certification process. FAA officials could not provide us with information on the cost of acquiring new technology but noted that the most promising device would cost about $800,000 per unit. Airlines are responsible for screening passengers and baggage and, therefore, have historically been responsible for purchasing detection devices. Airline officials with whom we spoke expressed several concerns about the cost of purchasing new equipment. First, industry estimates that the cost of a single device could range from $250,000 to $1 million. Because new devices probably will have to be used in combination, the costs of a single integrated system could be significant--in excess of $2 million at one location, at one airport. At one foreign airport, a contractor is testing a system that cost about $500,000, according to FAA officials. Because of concerns about the system's reliability and the need to monitor a large number of passengers, the foreign government bought three systems. Second, airline officials point out that costs for explosive detection equipment will have to be considered for each airport--both domestically and internationally--at which FAA requires the screening of checked baggage. Furthermore, a DOT official pointed out that problems with reliability may force airlines to acquire significant numbers of backup systems to ensure that equipment is available to screen baggage. Lastly, airline security officials recognize that explosive detection technology is evolving and that improvements will continually be made and, perhaps, mandated by FAA. Airline security experts are concerned that FAA may mandate the use of one system and 1 to 2 years later mandate the use of another. According to the Director, Office of Civil Aviation Security Policy and Planning, FAA cannot analyze costs until after it certifies systems for airlines' use because the agency does not have information on manufacturing, operating, or maintenance costs. In addition, since FAA intends to develop a shopping list of certified systems and allow the airlines to choose their own combinations for individual airports, the economic and operational implications for installing new systems may vary for each airline and airport. The Director and other FAA officials noted that FAA will develop cost estimates before mandating that airlines use the systems and said that it is not FAA's intention to require airlines to purchase equipment every time a new technology is certified. FAA NEEDS A STRATEGY TO IMPLEMENT NEW TECHNOLOGY ---------------------------------------------------------- Chapter 4:2 Introducing new explosive detection equipment will be difficult because the airlines are concerned about costs and the aviation community has little experience with the technology. Therefore, DOT and industry officials--and FAA's own Aviation Security Research and Development Scientific Advisory Panel--believe that FAA needs to develop, in close cooperation with the industry, a plan or vision that clearly outlines FAA's strategy for introducing new detection equipment. Although FAA has various planning efforts under way and is considering the use of commercially available technology, the agency has not developed an effective road map for guiding its and industry's efforts. At a minimum, such a plan should articulate FAA's role in developing and assisting the industry in implementing new technology, set milestones indicating when airlines should be prepared to purchase and implement new equipment, identify foreign and domestic airports that will be earmarked for priority implementation, list contingency equipment that the airlines could use if an urgent threat arises, outline anticipated procedures for using new equipment in the general framework for aviation security, and identify the government and industry resources (staffing and costs) needed to implement the new equipment. FAA would have to periodically update the plan to reflect progress on a number of issues and identify changes needed in the overall direction of the technology's development and the agency's philosophy. ISSUES NEED TO BE ADDRESSED BEFORE AIRPORT IMPROVEMENT PROGRAM GRANT FUNDS ARE USED TO PURCHASE DETECTION SYSTEMS ---------------------------------------------------------- Chapter 4:3 The Congress is considering legislation that would clarify airports' authority to purchase explosive detection systems with AIP grant funds. Historically, airlines have been responsible for purchasing new detection equipment. This proposal, if approved, would not represent the first use of federal funds to purchase security equipment. In response to the rash of hijackings in the 1970s, FAA purchased the first generation of metal detectors for the industry. After a short time, FAA turned this responsibility over to the industry. DOT, FAA, and industry officials believe that federal funding may well be needed to speed the introduction of new explosive detection technology. However, FAA officials caution that a broad range of financial and operational ramifications of any statutory changes might directly or indirectly affect the airlines' existing responsibilities for screening baggage. On the basis of our discussions with FAA, DOT, and industry officials, we identified three issues that we believe need to be resolved before FAA allows airports to use AIP grant funds to purchase explosive detection equipment. First, regulations promulgated under section 315 of the Federal Aviation Act of 1958, as amended, make airlines--not airports--responsible for screening passengers and their luggage for both domestic and international flights. FAA officials told us that airlines would have to enter into agreements with airports to use the equipment while maintaining the responsibility for screening passengers and baggage. Under one alternative being explored, airports would be allowed to lease explosive detection equipment to the airlines. FAA points out that other issues--including questions of responsibility and liability--need to be addressed. According to FAA officials, most airports would be reluctant to assume responsibility for screening passengers and baggage. Second, the proposed legislation does not include the eligibility requirements for airports to purchase explosive detection equipment with AIP grant funds. For example, the proposed legislation does not state whether the explosive detection system must be approved by FAA to be eligible for AIP funding. Under the Aviation Security Improvement Act, the FAA Administrator must certify that explosive detection equipment meets a minimum standard of performance before requiring it for airlines' use. According to FAA officials, it is unclear whether (1) the equipment falls under the Aviation Security Improvement Act and must be approved by FAA before it can be eligible for funding or (2) any commercially available explosive detection device can be eligible without being tested by FAA to ensure that it meets a minimum standard of performance. DOT officials are concerned that AIP funds will not be used effectively unless FAA testing and approval are required for an airport to obtain AIP funds for equipment purchases. Lastly, the impact on AIP grant funds could be significant. AIP provides airports with funds to enhance their capacity and safety, mitigate noise, and improve security. AIP has funded almost half of the $500 million in costs for airport computer access and control systems since 1989. Because FAA has not analyzed the costs associated with the new explosive detection equipment, the financial impact of acquiring this equipment is unknown. However, adding the cost of acquiring explosive detection equipment to the cost of refining computer access systems could place significant financial demands on AIP funds. In addition to the three concerns we identified, FAA officials offered another view of the proposed legislation and its intent. They viewed the legislation as a way for airports to obtain explosive detection devices and use them for purposes other than screening passengers and baggage. For example, FAA officials told us that a detection device could be used to investigate a potential bomb threat at an airport. CONCLUSIONS ---------------------------------------------------------- Chapter 4:4 The cost and source of funds for new security technology remain important issues and will continue to challenge the Congress, FAA, and the aviation community. The airline industry is concerned about the cost of acquiring new security technology in the near future and over the next decade. Although the cost of new devices and systems is uncertain, it appears to be significant. FAA can help the industry by developing a plan or general framework for implementing the new technology that identifies important milestones, resources, and roles for industry and government. If FAA expeditiously developed a plan, with industry input, the airlines would be in a better position to plan and budget for future security acquisitions. Legislation has been introduced that would clarify the use of federal grant funds for the purchase of new explosive detection devices by airports. However, several issues need to be resolved during congressional deliberations before such a proposal is feasible. RECOMMENDATION ---------------------------------------------------------- Chapter 4:5 To facilitate the introduction of new explosive detection equipment, we recommend that the Secretary of Transportation direct the FAA Administrator to develop a plan, with industry, that provides a strategy for implementing new detection technology during the next decade. This plan should include important milestones and identify roles; cost estimates for the purchase, operation, and maintenance of explosive detection systems; and FAA and industry resources. MATTER FOR CONGRESSIONAL CONSIDERATION ---------------------------------------------------------- Chapter 4:6 The Congress may wish to consider requiring FAA to certify explosive detection equipment as a condition of eligibility for AIP grant funds. AGENCY COMMENTS AND OUR EVALUATION ---------------------------------------------------------- Chapter 4:7 FAA concurs with our recommendation that it develop a plan to implement all new technology. During the past year, FAA has been working closely with its Technical Center to develop a computer- based project planning and tracking system to help assess the impact of security RE&D program changes on the rulemaking process. They also noted that the FAA Strategic Plan provides a "rough sketch" of both the short- and long-term efforts in security research, and they have recently completed a list of contingency equipment that airlines could use if an urgent threat arises. In our view, FAA's strategy or vision for implementing new explosive detection technology must also address FAA's role in developing explosive detection technology, anticipated government and industry resources, and procedures and a general framework for using the new equipment. Developing such a plan and working closely with industry would engender closer cooperation with the airlines, set expectations, and help lessen the financial impact of implementing the new technology. According to FAA officials, the airlines will find it difficult to accept the acquisition and life-cycle costs for new explosive detection systems, and airport authorities have already raised concerns about the difficulties they may encounter in installing new systems in existing terminal facilities. They acknowledged that the costs to the airlines for new devices will be significant. Therefore, FAA expects that any mandate to use new devices that are not funded by the government will meet stiff resistance from airlines. FAA officials noted that in the past, airlines and airports had resisted participating in demonstration projects fully funded by FAA. We believe these concerns further point to a need for FAA to develop a plan that includes identifying the resources needed to implement the new technology. The sooner FAA provides the industry with such information, the sooner airlines and airports can begin to plan, budget resources, and set aside the necessary space for new equipment. STATUS OF KEY SECURITY RESEARCH PROJECTS =========================================================== Appendix I FAA categorizes the development of new explosive detection technology into three activity phases. First, FAA evaluates the concept underlying the new technology to determine whether it would enhance security. During this phase, industry, a national laboratory, or academia performs a
|Join the GlobalSecurity.org mailing list|