In February 1996, OT&E was conducted by the Office of Health Systems Evaluation (the OTA at the time) on the Medical Electronic Customer Assistance system at the Defense Personnel Support Center in Philadelphia, and on the Forward Customer Support system at Lackland AFB, TX, and the National Naval Medical Center, Bethesda, MD. Although the test results were generally very favorable and most users enthusiastically supported the new system, some concerns were noted. The concerns were in the areas of data base accuracy and currency, user training, system availability, and response times. The DMLSS PM took immediate action to correct these deficiencies, and DOT&E concurred with the OTA that the use of both the Forward Customer Support system and Medical Electronic Customer Assistance system should continue, as should the planned DMLSS AIS development program.

In August 1996, OT was conducted on DMLSS AIS Release 1 at Camp Lejeune, NC; Fort Hood, TX; and Maxwell AFB, AL. OPTEVFOR, the new lead OTA, determined that the system was "potentially operationally effective" and "potentially operationally suitable." (The word "potentially" was used because this was an early OT&E that did not test all the DMLSS AIS required capabilities, many of which are still being developed.)

No OT&E was conducted between 1997 and 1998. The OT&E of DMLSS AIS Release 1 in 1996 was conducted in compliance with the TEMP approved by DOT&E on July 30, 1996. The updated Release 2 TEMP was approved by DOT&E in March 1998. FOT&E is scheduled for Release 2 in June 1999.

In its evaluation of the Release 1 OT, the OTA commented that the DMLSS AIS users were extremely satisfied with the new capability and were eagerly and aggressively using the system during their normal work routine. The OTA noted that the deficiencies from the previous OT&E of the Medical Electronic Customer Assistance system and the Forward Customer Support system had been corrected. However, many of the facility management functions of Release 1 could not be tested due to insufficient time to populate the data base and user inexperience with some of the new features. The OTA's evaluation included many recommendations for improvement. It noted major deficiencies in the areas of contractor-provided maintenance, data base building and updating, and interfaces with related systems. DOT&E concurred with the OTA findings. The DMLSS AIS PM quickly took action to address all of these concerns.

Subsequently, DOT&E concurred with the deployment of DMLSS AIS Release 1, but further OT of the system is clearly needed. FOT&E is scheduled for DMLSS AIS Release 2 in June 1999. Release 2 will contain upgrades to both the materiel management and facility management modules, and will replace two materiel management legacy systems. In addition to focusing on new functional capabilities (particularly those of the facility management module), FOT&E will address the operational effectiveness concerns (e.g., data base accuracy, security) and operational suitability concerns (e.g., maintainability, interoperability, training) noted in previous OT&E.

Much of the difficulty in testing the facility management module of Release 1 was experienced because OT took place only a few weeks after the system had been installed at test locations. There had been insufficient time for the users to become acquainted with the new system and learn how to perform all of the functions. Even more importantly, there had not been enough time to populate the data bases sufficiently with all of the unique data pertaining to the test locations. Determining the optimum amount of time to wait between the installation at test sites and the beginning of OT is a challenge that differs with each automated information system. Past experience has shown that at least one month (preferably two) is the minimum period. Some systems may require more time before they are ready to be tested. For these systems, either deployment must be delayed, or ways must be found to adequately prepare the OT locations prior to the start of the test.