Space

WESTERN FIELD OFFICE

26722 PLAZA ST., SUITE 130

MISSION VIEJO, CA 92691-6300

Mr. Keith Englander,

In our last correspondence we provided what we feel is absolute irrefutable scientific

proof that TRW's discrimination technology does not cannot, and will not work, and falls

far-short of meeting the TRD/contract requirements. After reviewing the POET draft report

we feel even stronger about that statement. We have just received the final version of-the

POET report but have not had time to review it in detail.

I would like to start by highlighting a few areas of the four-page "Executive

Summary" from the POET draft report which are extremely supportive in exposing the

serious flaws in the TRW discrimination technology. These statements fully corroborate

our findings and reiterate what we have been saying for over two years.

Starting with the first paragraph of the "Executive Summary" (page iii), last

sentence, it states "Target discrimination and selection must be done with or without

handover information about the target from the rest of the NMD system " The second

paragraph, first sentence, states "TRW has developed an onboard autonomous

discrimination architecture and a set of algorithm for the Boeing EKV "

The first indented paragraph on page iii, #1, states "Overall, the Baseline

Algorithm(s) (BLA) are well designed and) work properly, with only sonic refinement or

redesign required to increased robustness of the overall discrimination functions. " We

feel this statement is misleading and incomplete, and contradictory of later material in the

report. The indented paragraph #4, page iv, states "The Performance of the discrimination

The target signatures are heavily influenced by threat type, target characteristics, sensor-

current discrimination architecture to make it more robust by including, for example, real-

time adaptation capability and certain threat-type-based system-level discrimination.

We definitely agree with the above statement which totally supports and

corroborates what we have been saying, and contradicts indented paragraph 91 which states

"Overall the BL4 are well designed and work properly, etc" along with the statement that

TRW's discrimination architecture is autonomous. The Technical Requirements Document

(TRD) requires that the discrimination technology be capable of operating with or without

knowledge of threat types. The POET clearly indicates that TRW's discrimination BLA is

not robust, and it is crucial that the training data sets closely match actual threats for the

fine discrimination function.

The POET's statement of "Overall, the BLA(s) are well designed and work

properly" would be accurate if stated that TRW's BLA's are well designed and work

algorithm has to be capable of adjusting to unpredictable and unknown target objects

which an object from an expected object class is missing. The TRD specifically states

that "Included in the scenarios are accidental and unauthorized launches by the

Commonwealth of Independent States (CIS, formerly the Soviet Union) and China, and

limited strikes by China and the Rest of World (ROW) countries" (page A-2). Also, the

BLA must perform when an object from an expected object class does not perform

nominally. TRW's discrimination algorithms have never been able to function in this

situation. Their algorithms are heavily dependent on prior knowledge, or they adjust

experimental results after the fact.

On pace vi of the Executive Summary, the last sentence on the page states "If targets

suite. " It also states in the body of the same paragraph "The most stressing target suite for

EKV discrimination contains the same ten targets as those deployed in IFT-IA. " This

contention seems questionable because TRW requested the removal of one of the objects

from the target suite for the EFT-3. This information surfaced at the July 21 & 22, 1998,

meeting with TRW at the Aerospace Corporation. The reason for the removal of the object

was that its off-nominal deployment caused TRW's BLA significant problems. This

confirms the POET's statement "The performance of the discrimination architecture may

be fragile".

On page vi of the Executive Summary, third paragraph, it states "One of the most

discrepancies appears to be correct." In TRW's report titled "(U) POET Review Of EKV

Discrimination Algorithms" dated July 21 & 22,1998, they state "The primary differences

The very high PAT provided in the 60-Day Report is a direct result of the

extremely close match between the OSC code trained with EFT-IA flight test data and the

IFT-IA flight test data results. It is certainly clear that the IFT-3 actual results will not be

available as a training set until the flight test has been completed.

In our March 25, 1998, report we provided our views regarding- the Gap Filling

Algorithm (GAP) and the serious adverse of using this consequence algorithms We related

that both NRC and GBI representatives informed us, during our December 1997 trip to

Huntsville, that they do not understand how TRW's GFA for the signatures works and that

TRW has never explained or validated the method. We explain that TRW's GFA attempts

to fill in missing data points in the EKV sensor signal output by using values from the

OPTISIG library. However, the intended program inadvertently used zero values for the

data points. This resulted in the "anomalistic" behavior of the signal outputs analyzed by

'INRC during their testing of TRW's discrimination technology, wherein every signature had

an unexplained dip. Additionally, TRW artificially chopped the amplitude of the sensor

signal output which results in arbitrarily forcing the signal mean to a predetermined value.

This resulted from the sensor output noise level being too high for the baseline

discrimination algorithm to process. It has been shown by TRW that the GFA manipulates

data in such a way to give artificial results. TRW showed this in their correction to the 60-

Day Status Report by changing the location of the center of the ellipses, changing the shape

of the ellipses, and by changing two of the features. This allowed an artificial improvement

in the Probability of assigned Target (PAT). The PAT should be solely dependent on the

calculated PAT. This GFA process is done outside of accepted statistical scientific

methods. Prior td the down-select TRW stated they were not going to use the GFA for the

IFT-3 which is also stated in the POET's report. This can only raise the question of how

would TRW handle any gaps in the actual IFT-3 signatures.

On page vi the POET states "Another important issue concerns the "improper"

misunderstanding of Bayes rule and misinterpretation of TRW's approach. " Our

understanding of TRW's ranking scheme is that it selects the object with the highest

probability of belonging to the RV class given the object's extracted feature vector. This is

the case even though it is possible that the particular object selected may have a higher

posterior probability of belonging to another class. It can also be the case, that another

object with lower PAT than the highest-ranking object could actually have the largest

probability of being the RV among the classes. The table below is a simple illustration of

this possibility:

OBJ-ECT OBJ-ECT CLASSES

RV MB LB

1 0.40 0.60 0.00

2 0.39 0.305 0.305

We do not misunderstand Bayes rule. We apply it differently than TRW does in its

MBQC. Rather than assuming that each tracked object surely belongs to one of five

different classes with a given prior hypothesis probability per object class, we assume the

equally likely prior hypotheses that one of the tracked objects is surely the RV. In addition

we view the set of extracted independent feature vectors as comprising the compound

posterior event.

Given our hypothesis and event structure, our Monte Carlo analysis (1000 samples)

yielded an estimated probability of correct RV selection that is within a few percentage

points of the POET'S. The Confusion Matrix, displayed on page 18 of the POET draft

report, shows their probability of correct RV selection. The POET results and ours are

consistent and mutually supportive. The difference between the two is well within the

Monte Carlo sampling error.

The last paragraph, page vi, of the Executive Summary the POET states "It is a

the threat types included in the TRD cannot be obtained. The POET study did not evaluate

to what extent the BLA may be used to meet the general requirements described in the YRD

beyond the ability to determine which object among a threat cloud is lethal (given that the

threat type is known). However, the POET team's concerns regarding the potential lack

of robustness of the current BLA have been indicated earlier. The POET team also

recognizes that the concept of threat typing can have significant impact on EKV

advantage of threat-typing information while at the same time reducing potential risk "

We agree with the sense of this paragraph even more strongly than as stated by the POET.

In reviewing the POET report we noted that part of the report focused on the Kalman

Filter (KF) performance. We reviewed the Statement of Work (SOW) generated by Dr.

Frank Handler for the POET but did not see anything about the KF. Our concern is when

did this become part of the SOW? Why weren't we notified? Why didn't we get a

copy of the new SOW? and Why didn't we get a chance to provide some input? In

reviewing the reference list for the POET report we noted that the "I-SETAC ANALYSIS

OF ROCKWELL KALMAN FILTER FEATURE EXTRACTION CONCEPT (LD" dated

November 14, 1996 was not identified. The Nichols Research Corporation (NRC), which is

the technical advisor to the Ground Base Interceptor (GBI) Office in Huntsville, Alabama

authored this report. We feel that before the POET made any comments regarding the KF

they should have had all immediately available reports concerning the matter.

From reviewing the POET draft report, the TRD requirements, and all the other

accumulated relevant documents we will summarize a number of points on why the TRW

BLA is not, nor has it ever been basically sound or capable of meeting the contract

requirements:

(1). Almost all features are only descriptive statistics which wash out any intrinsic

physics and spectral information contained in the intensity signatures. Moreover, the

features are not functionally independent.

(2). The features, for all object classes, are assumed to be constant over time; they

are not. Feature time variability is artificially compensated by lumping it into the apriori

feature error covariance matrices. Lumping feature time varying effects into these

covariance matrices creates large regions of overlap in the feature spaces of different object

classes. These overlap regions grow as mission elapsed time increases because of the

secular trends in the features. Such trends, reflected in the extracted features, are highly

likely to cause one detected object's extracted feature vector to wander close to the center of

the apriori error ellipse family for a different object class. Depending on the actual

engagements object-by-object materials, temperatures, aspect angle temporal variation, etc.

the RV's extracted feature vector will, with uncomfortably high probability, travel out of the

apriori likely RV class region into the likely region of another object class; conversely the

extracted feature vector for another object class is likely to "walk" close to the RV apriori

feature vector mean. We can predict with some certainty that such confusion will grow

overtime as more object-by-object data is accumulated. This is exactly what a sound

algorithm should not do. A sound algorithm should be able to select the RV with greater

and greater confidence as the object-by object data spans increase.

(3). TRW's BLA, while computing a feature extraction error covariance matrix

for each tracked object, does not use this matrix in its Baysean Likelihood calculations.

This omission is both statistically incorrect and computationally inefficient. We can only

speculate that the required use of this matrix as a summand was left out because to include it

would generate larger total error overlap and selection performance degradation.

(4). The strictly recursive feature extraction approach is questionable. Recursive

computations have appreciable startup transient times and preclude the exploitation of

spectral methods, which require a batch of data over an appropriate time span. Batch

estimation is more appropriate than recursive estimation for truly time invariant parameters.

(5). With one exception the total feature set has nothing to do with the underlying

physics, thermodynamics and mechanics.

(6). To paraphrase one of the POET's main finding's, apriori threat identification

data will not always be available. Even if accurate missile typing data is transmitted from

surveillance assets to the GBI via the BMC 3 element, there is no assurance whatsoever that

the corresponding MDL would faithfully represent the set of objects, RV and decoys,

carried by the missile's PBV, nor all likely variations in the actual object-by-object

composition, temperature distribution, deployment geometry, rotational dynamics

properties, etc.

(7). Given (1) to (6) one can expect that the BLA performance will be exquisitely

sensitive to feature extraction start times, data spans, non-nominal object deployments,

irradiance data gaps, dips, and spikes of whatever origin. Such dips and spikes can arise

from particle and gamma ray events, sensor electronic anomalies, or actual rapidly time

varying aspect angles as well as the individual objects projected area, shape, and body

element material composition changes.

(8). As for robustness, we assert that the BLA is, instead, extremely fragile, This

contention is borne out in both the POET's and our results as shown in the table on page

66 of the draft report titled "Independent Review of TRW Discrimination Techniques", 7-8

December 1998, by Dr. Ming Tsai et al (page 67 of the POET final report refers).

(9). It is clear from our reading and analysis of the documentation cited in the

POET's bibliography that TRW heavily censored the IFT-IA signature data, deleting

approximately the first 20 seconds and the last I I seconds to go from poor results in its 45-

Day Report, to excellent results in its 60-Day Report Addendum. Removal of the last I I

seconds is especially critical because we have evidence to believe that in this interval of

time the RV's PAT fell dramatically and dropped below another object's rising PAT.

Additionally, TRW generated the MML used for the 60-Day Report analysis by training the

OSC code with the detailed knowledge from EFT- IA flight data since that information was

used in constructing the MDL. The very high PAT achieved in the 60-Day Report is a

direct result of this close match and has nothing to do with the GFA. Such after-the-fact

(10). We also suspect that early data was censored to hide the difficulty the BLA

was having with real signature spikes, not Boeing/GBI sensor-induced spikes. These

signature spikes were actually induced by object attitude motion as verified by the_

Airborne Sensor Testbed (AST) data. In addition the use of impermissible after the fact

knowledge by TRW to switch features in mid-data stream was also noted, and previously

reported to the POET.

(11). TRW's BLA cannot be well designed if it is lacking, a critical component

Putting together all these instances of impermissible use of after-the-fact

knowledge, early data censoring to remove difficult data, late data censoring to make sure

that the RV was selected, and a missing key algorithm, can only cast grave suspicion about

the ineffectiveness and fragility of TRW's BLA's capability to meet contract requirements.

There is no crime in producing a failed algorithm during a Research and

Development project. That's all part of the acceptable risk in order to eliminate the non-

viable approaches. The crime is in producing a failed algorithm and knowingly covering up

its failure.

In conclusion, we are looking forward to meeting with you and the POET this week.

I request that this correspondence not be disseminated outside your office or members of the

POET. I want to thank for your time and interest in this matter.

Sincerely,