The Defense Information Systems Agency began the DMS program in 1988. Since the 1997 IOT&E of release 1.0, DMS has continued to improve through OAs in 1998 and 1999, and an OT&E of release 2.1. The AUTODIN backbone has been downsized to three message-switching centers called DMS Transition Hubs (DTHs). An OA, conducted late October through early November 2000, found release 2.2 to be not effective and not suitable. The 2.2 OA revealed several system deficiencies, as well as numerous problems with site installations, configurations, and overall security posture of the system. DOT&E directed that DMS 2.2 be retested after the deficiencies were rectified.

A Limited Field Test (LFT) of DMS 2.2 was conducted from March 26 through April 3, 2001 to reassess system effectiveness problems identified during the earlier OA. The LFT did not address operational suitability. Subsequently, DMS 3.0 underwent an OA in November 2001, and is now scheduled for a full OT&E late spring 2002. This OT&E will determine operational effectiveness and suitability for sensitive compartmented information enclaves as well as unclassified and secret traffic. It will also have users performing more realistic mission essential tasks in a scenario-like environment.

DMS 2.2 performed significantly better in the LFT than in the earlier OA, and it was concluded that DMS 2.2 is operationally effective but not suitable. Administering DMS requires attention to detail and relies heavily on complex documentation and manual processes. Security tests revealed that system administrators had failed to protect all elements. A typical system administrator was poorly equipped to install, maintain, troubleshoot, and ensure security configuration of the system. In spite of these deficiencies, release 2.2 did not pose sufficient risk at this point in DMS maturation to preclude its operational use. However, the PMO must continue to streamline system operations, improve training, and enhance documentation. The system administrators must strictly follow all established security policies and procedures.

Although many measures of effectiveness were successfully met, the OA of DMS 3.0 showed that the system was not sufficiently mature for a full OT&E. Significant improvement was noted in the security posture, based on a limited security assessment. Interfacing to the legacy AUTODIN system was problematic, and especially so within the intelligence community. There were also problems with certificates and Fortezza cards within the Certificate Management Infrastructure.

The diversity of site configurations and operational needs poses design and testing challenges. Focus on security training and greater attention to detail during site installation and configuration is necessary and can lead to a relatively secure system. Installing and maintaining DMS is complex, and automated tools for assessing the system security configuration would be helpful. Operational tests have not focused on automated tools because they were under development. We recommend development focus on simplifying and supporting system administrator tasks followed by OT&E events exercising these critical support tasks, tools, and procedures. To achieve DTH closure by September 2003, the intelligence community will need a concerted effort to mature their implementation of DMS with particular focus on their Multi-Function Interpreter and Decision Agent products. Occasionally, DMS messages do not get delivered. In most of these situations, there is some type of system notification to either the sender or to a system administrator. Operational testing needs more focus on how message writers ultimately ensure the messages are received by the intended readers.