Measuring Performance: The Advanced Technology Program and Private-Sector Funding (Letter Report, 01/11/96, GAO/RCED-96-47)
Pursuant to a congressional request, GAO examined the impact of the
Advanced Technology Program (ATP), focusing on whether: (1) research
projects would have been funded by the private sector if they had not
received ATP funding; and (2) ATP aids in the formation of joint
ventures.
GAO found that, based on a survey of 123 ATP applicants that submitted
highly rated bids between 1990 and 1993: (1) 89 applicants received ATP
funding, while 34 did not; (2) about half of the applicants that did not
receive ATP funding continued their projects using other funding
sources, but generally met their project goals later than scheduled; (3)
applicants that received ATP funding were evenly divided on whether they
would have pursued their projects without ATP funding; (4) 63 percent of
the applicants did not request funding from other sources before
applying to ATP, but those that did were often told by prospective
funding sources that their projects were too risky or precompetitive;
and (5) ATP satisfied its goal of aiding in the formation of joint
ventures, as 26 of 34 joint venture applicants reported joining forces
solely to pursue ATP projects.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: RCED-96-47
TITLE: Measuring Performance: The Advanced Technology Program and
Private-Sector Funding
DATE: 01/11/96
SUBJECT: Research and development
Scientific research
Research program management
Technology transfer
Surveys
Federal funds
Cost sharing (finance)
Economic development
IDENTIFIER: NIST Advanced Technology Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** <info@www.gao.gov> **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to the Ranking Minority Member, Committee on Science, House of
Representatives
January 1996
MEASURING PERFORMANCE - THE
ADVANCED TECHNOLOGY PROGRAM AND
PRIVATE-SECTOR FUNDING
GAO/RCED-96-47
Advanced Technology Program
(307723)
Abbreviations
=============================================================== ABBREV
ATP - Advanced Technology Program
FTE - full-time equivalent
GAO - General Accounting Office
NIST - National Institute of Standards and Technology
R&D - research and development
Letter
=============================================================== LETTER
B-270551
January 11, 1996
The Honorable George E. Brown, Jr.
Ranking Minority Member
Committee on Science
House of Representatives
Dear Mr. Brown:
This report responds to your request that we assess the impact of the
Advanced Technology Program (ATP), which is administered by the
National Institute of Standards and Technology within the Department
of Commerce. ATP's purpose is to provide support on a cost-sharing
basis for industrial research and development projects--projects that
have a significant potential for stimulating economic growth and
improving the competitiveness of U.S. industry. Federal funding for
ATP has grown sharply, from $68 million in fiscal year 1993 to $341
million in fiscal year 1995. Recently, however, budget proposals
have suggested eliminating ATP's funding for fiscal year 1996.
As agreed with your office, our objective was to examine, as one way
to assess ATP's impact, whether research projects would have been
funded by the private sector if they had not received funds from ATP.
We also examined ATP's impact in terms of other goals of the program,
such as aiding the formation of joint ventures. We agreed on this
approach because of the difficulty of assessing the net impact of
ATP's investments in technology on the economy. For example, it is
difficult to establish a causal link between a successful project and
government funding earlier in the project. Moreover, the impact of
ATP should be measured not only by its effect on the firms that
receive funding but also by its effect on other firms--a difficult
undertaking that our approach avoids.
To meet our objective, we focused on two groups of ATP applicants,
which we called "winners" and "near winners." Both groups submitted
proposals that were rated highest during ATP's review, but the near
winners did not ultimately receive ATP funding. We surveyed by
telephone all applicants that qualified as winners or near winners
during ATP's first 4 years (1990-93). We achieved a 100-percent
response rate from the 123 respondents that we included in our
analysis (89 winners and 34 near winners). We asked the near winners
if they had continued their proposed projects using other funding
sources after ATP declined to fund them. Given the similarity in the
qualifications of the proposals submitted by the winners and near
winners as determined by ATP, another purpose of this question was to
determine whether the winners would have been likely to continue
their projects without ATP funding. We also asked both groups if
they had sought funding from other sources before applying to ATP.
This question provided information on whether private-sector sources
had the opportunity to fund proposed projects before an applicant
sought ATP funding.
In our survey, we also collected information that provides an
extensive profile of the respondents, which we used in some of our
analyses; additional information from this profile is provided in
appendix I. Appendix II contains our survey questions and an
aggregate list of all the responses, and appendix III provides
further detail about our objectives, scope, and methodology.
RESULTS IN BRIEF
------------------------------------------------------------ Letter :1
ATP has funded research projects that would have been funded by the
private sector as well as those that would not. Half of the near
winners continued their projects without relying on ATP funding,
while the other half discontinued their projects for various reasons.
The winners were nearly evenly divided when asked if they would have
pursued their projects even if they had not received ATP funding.
Almost all the near winners that continued their projects did so on a
modified schedule, meeting the projects' milestones later than
scheduled in their proposals to ATP.
In most cases, private-sector sources did not have the opportunity to
fund ATP projects. Of the 123 applicants we surveyed, 77,\1 or 63
percent, did not look for funding from other sources before
requesting it from ATP. Those applicants that did look for funding
looked for a long time and made many attempts to find funding, on
average. Seven applicants turned down offers from private sources
because they could not reach an acceptable funding arrangement.
Our survey also found that ATP had other effects. More than
three-fourths of the joint-venture applicants indicated that they had
come together solely to pursue an ATP project, thus satisfying ATP's
goal of serving as a catalyst for the formation of joint ventures.
Furthermore, of the 45 applicants that tried to find funding
elsewhere before turning to ATP, about half were told by prospective
funders that their projects were either too risky or
"precompetitive"\2 --characteristics that fulfill the aims of ATP
funding.
--------------------
\1 One applicant did not know.
\2 "Precompetitive" refers to the stage during research and
development at which a preliminary assessment of a technology's
commercial potential can be made but before commercial prototypes are
developed.
BACKGROUND
------------------------------------------------------------ Letter :2
ATP's mission is to stimulate economic growth in the United States
through technology development. The program seeks to accomplish that
mission by sharing the cost of industrial research and development
projects. The projects selected by ATP for funding are characterized
by "a potential broad-based economic impact but a relatively high
technical risk and a long time horizon," according to ATP.
ATP's guidance states that if the technical risk associated with a
project is very low, federal funding should not be necessary. In
addition, when submitting a research proposal, applicants must sign a
form stating that "this proposal is not requesting funding for
existing or planned research programs that would be conducted in the
same time period in the absence of financial assistance under the
ATP." This wording suggests that ATP should not fund projects that
other sources would have funded or, when ATP does fund such projects,
that ATP funds should enable applicants to complete their projects in
a shorter time.
APPLICANTS' ACTIONS AND
INTENTIONS TO FIND FUNDING
BEFORE APPLYING TO ATP
------------------------------------------------------------ Letter :3
Most ATP applicants did not look for funding from other sources
before requesting it from ATP. In addition, the applicants were
about evenly divided when asked if they intended to pursue their
projects whether or not they received ATP funding.
When asked if they had searched for funding from other sources before
applying to ATP, 63 percent of the applicants (77 of 123; one
applicant did not know) said that they had not. Considering the
winners and near winners separately, we found that 65 percent (58 of
89) of the winners had not looked for funding before applying to ATP,
along with 56 percent (19 of 34) of the near winners.
Of the 45 applicants that had looked for funding before applying to
ATP, about 53 percent (24 of 45) sought it from private sources only,
9 percent (4) from public sources only, and 38 percent (17) from some
combination of private and public sources. On average, 42 of these
45 respondents (3 did not respond) searched for funding for 18 months
before applying to ATP and made eight separate attempts to find
funding.
When asked what reasons the non-ATP sources gave for not providing
funding, 54 percent of these applicants (22 of 41; 4 did not respond)
said that their projects were viewed as either too risky or
"precompetitive"--both outlined in Commerce's regulations as reasons
for ATP to fund projects. Sixteen percent (7 of 45) said they had
turned down funding offers because they could not agree on terms with
the prospective funder; 3 of these 7 eventually received ATP funding.
We asked the winners and near winners if they intended to pursue
their projects whether or not they received ATP funding. When we
considered the two groups together, 42 percent (52 of 123) said "yes"
or "probably yes;" 41 percent (51) said "no" or "probably no;" and 16
percent (20) were uncertain. Of the respondents that said they
intended to pursue the project, 94 percent (49 of 52) indicated that
their projects' schedules would be modified and that the milestones
would be met later than scheduled in their proposals to ATP. When we
considered the ATP winners' answers alone, 40 percent (36 of 89) said
"yes" or "probably yes;" 16 percent (14) were undecided; and 44
percent (39) said "no" or "probably no."
Most of the joint-venture applicants came together to apply to ATP.
Seventy-six percent (26 of 34) said they had formed a new group to
pursue the projects described in their ATP proposals. The remaining
joint-venture applicants had worked together before applying to ATP,
pursuing either the projects that they proposed to ATP or other
projects.
NEAR WINNERS' ACTIONS AFTER ATP
DECLINED TO FUND THEIR
PROPOSALS
------------------------------------------------------------ Letter :4
After ATP declined to fund their proposals, half of the near winners
continued their projects using other funding sources. The near
winners with certain characteristics were more likely to continue
their projects than others.\3
--------------------
\3 In our findings for this section only, the total of near winners
drops to 28 because 6 of the near winners were granted funding by ATP
in a subsequent round of competition. Thus, we eliminated them from
consideration here, focusing only on those that found funding from
sources other than ATP.
HALF OF NEAR WINNERS
CONTINUED PROJECTS USING
OTHER FUNDING SOURCES
---------------------------------------------------------- Letter :4.1
Half of the near winners (14 of 28) continued their projects using
other funding sources after ATP declined to fund them. These other
funding sources included federal government programs other than ATP;
state government agencies; and private funders, such as industry
groups or trade associations, other private companies, venture
capitalists, and the company itself. All 14 near winners used some
private funding to continue their projects; 8 of these financed their
projects using private funds only.\4 Ninety-three percent (13 of 14)
of the projects that were continued were or are being carried out on
a modified schedule, meeting their milestones later than scheduled in
the proposals submitted to ATP.
--------------------
\4 Some of the near winners that continued their projects using other
funding sources likely benefited, during their search for other
funding, from having been rated highly by ATP. We did not evaluate
the extent to which this "halo effect" may have occurred; however, we
acknowledge that a high rating from ATP might have proved beneficial
to some near winners.
SOME GROUPS OF APPLICANTS
WERE MORE LIKELY THAN OTHERS
TO CONTINUE THEIR PROJECTS
USING OTHER FUNDING SOURCES
---------------------------------------------------------- Letter :4.2
Some near winners were more likely than others to continue their
projects after ATP declined to fund them. For example, 86 percent
(12 of 14) of the near winners whose projects were under way before
they applied to ATP continued them, compared with 14 percent (2 of
14) of those whose projects were not under way. Similarly, 77
percent (10 of 13) of the near winners that had looked for funding
from other sources before applying to ATP continued, compared with 27
percent (4 of 15) that had not looked for funding before applying.
Table 1 groups the near winners according to different
characteristics and shows odds ratios, which indicate the degree of
association between the characteristics of these groups and the
likelihood of their continuing their projects. Odds ratios measure
the association between two variables through a single value. The
closer the odds ratio is to 1.00, the weaker the association. For
more information on odds ratios, see appendix III.
Table 1
Odds Ratios Showing Association Between
Characteristics of Certain Near Winners
and Continuation of Their Projects
Odds ratio: How
many times more
likely was near
winner A to
continue the
Characteristics of near winner Characteristics of project than near
A near winner B winner B?
------------------------------ ------------------ ------------------
Project under way before near Project not under 36
winner applied to ATP way before near
winner applied to
ATP
Looked for funding from other Did not look for 9.17
sources before applying to funding from
ATP other sources
before applying
to ATP
Funds 50 percent or more of Funds less than 50 2.5\a
research and development percent of
internally research and
development
internally
Single applicant Joint venture 2.75
Company with more than 10 Company with 10 or 1.6\c
FTEs\b fewer FTEs
Company with more than 50 FTEs Company with 50 or 1.6
fewer FTEs
Company with more than 100 Company with 100 1.39\d
FTEs or fewer FTEs
Company with more than 500 Company with 500 1.05
FTEs or fewer FTEs
----------------------------------------------------------------------
\a Nine companies did not indicate what percentage of their research
and development they fund internally; therefore, they are not
included in this calculation.
\b FTE = full-time equivalent.
\c Nine companies did not provide their number of FTEs; therefore,
they are not included in this calculation.
\d Because this odds ratio is close to 1.00, we can say that the odds
of a company with 100 or more FTEs continuing its project are about
the same as the odds of one with fewer than 100 FTEs continuing its
project.
In addition, single applicants more often continued their projects
than joint-venture applicants: 58 percent (11 of 19) of the single
applicants continued their projects, while only 33 percent (3 of 9)
of the joint-venture applicants continued theirs. Seventy-one
percent (5 of 7) of the companies that generally fund 50 percent or
more of their research and development from their own internal
company funds continued their projects. In contrast, 50 percent (6
of 12) of the companies that generally fund less than 50 percent of
their research and development from their own internal company funds
continued their projects. (Nine companies did not indicate what
percentage of their research and development budgets they fund
internally.)
Finally, among the near winners, the smaller companies continued
their projects somewhat less frequently than the larger companies.\5
For example, 50 percent (3 of 6) of the near-winner companies with 50
or fewer full-time equivalents (FTE)\6 continued their projects,
while 62 percent (8 of 13) of the near-winner companies with more
than 50 FTEs continued theirs. (Nine companies did not provide the
number of FTEs for their companies.)
--------------------
\5 Joint-venture applicants were not included in this comparison
because they may include companies of different sizes.
\6 Measures of FTE indicate a company's size by estimating how many
full-time employees are represented by all part- time and full-time
employees considered together.
SOME NEAR WINNERS DID NOT
CONTINUE THEIR PROJECTS
---------------------------------------------------------- Letter :4.3
Fourteen near winners did not continue their projects after ATP
declined to fund the projects. The reason they most often gave for
not continuing was lack of funding (cited by 11 near winners). Two
near winners indicated that their projects were too long-term; one
cited market changes; one said that the project was too risky; and
one joint-venture near winner said that its newly formed partnership
had not worked out. (The near winners could provide more than one
reason.)
Of the near winners that did not continue their projects, 64 percent
(9 of 14) searched for funding but did not find it. Eight of these
nine reapplied for ATP funding during a subsequent round of
competition. Six of the nine are no longer looking for funding to
continue their projects.
STATUS OF PROJECTS FUNDED BY
ATP AND OTHER SOURCES
---------------------------------------------------------- Letter :4.4
ATP funded 89 projects from 1990 to 1993, and 14 near winners carried
out their projects using other funding sources during this time.
Sixty-four percent of these projects (66 of 103) were still under way
at the time of our survey. Twenty-seven percent of the projects (28
of 103) had been completed, while 5 had been discontinued. The
respondents to our survey listed the status of the four remaining
projects as either suspended or delayed. When we asked all the
applicants that had carried out their projects how satisfied they
were with either the projects' technical direction and progress (for
ongoing projects) or outcome (for completed projects), 94 percent (84
of 89) of the winners and 79 percent (11 of 14) of the near winners
whose projects were funded by other sources indicated that they were
at least generally satisfied.
CONCLUSIONS
------------------------------------------------------------ Letter :5
According to the results of our survey, ATP funds both projects that
would have been funded in its absence and projects that would not
have been funded. In addition, ATP achieves other goals, such as
aiding the formation of joint ventures and helping companies achieve
research milestones faster. These results should be considered
together when assessing ATP's impact.
AGENCY COMMENTS AND OUR
EVALUATION
------------------------------------------------------------ Letter :6
We provided a draft of this report to the Department of Commerce for
comment. Commerce found much of the report to be well done but
recommended certain changes. For example, Commerce felt that the
report needed to highlight our survey's results showing that those
research projects of near winners that were continued with
alternative funding continued at a slower pace than planned. We have
revised the report as appropriate. Commerce's written comments,
along with our detailed responses, are provided in appendix IV.
---------------------------------------------------------- Letter :6.1
We conducted our assessment from July 1994 through December 1995 in
accordance with generally accepted government auditing standards. As
arranged with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days
after the date of this letter. At that time, we will send copies to
the Secretary of Commerce; the Director, National Institute of
Standards and Technology; the Director, ATP; the Inspector General,
Department of Commerce; the Director, Office of Management and
Budget; and other interested parties. We will also make copies
available to others on request.
Please call me at (202) 512-3841 if you or your staff have any
questions. Major contributors to this report are listed in appendix
V.
Sincerely yours,
Victor S. Rezendes
Director, Energy and
Science Issues
PROFILE INFORMATION FOR
SINGLE-APPLICANT COMPANIES
=========================================================== Appendix I
In our survey, we asked individual applicants for general information
which, taken together, provides a profile of their companies. We
used some of this information for the analysis summarized in table 1
of this report. We include here supplemental profile information
that provides an overall picture of the coverage provided by the
Advanced Technology Program (ATP). This information includes the
size of the companies based on the number of employees as well as on
gross sales. We requested these figures for the fiscal year
completed before the company applied to ATP. We also asked for the
year the company made its first sale as an indicator of the age of
the applicant company. Included also are figures for the sources
that each company relies on for its overall research budget. We
again asked companies, in answering this question, to base their
responses on the fiscal year completed before they applied to ATP.
Figure I.1: Number of
Full-Time Equivalent Employees,
Including Both Outsourced and
Permanent Employees
(See figure in printed
edition.)
Figure I.2: Total Value of
Gross Sales in Fiscal Year
Before Company's Application to
ATP
(See figure in printed
edition.)
Figure I.3: Year of Company's
First Sales
(See figure in printed
edition.)
Figure I.4: Percentage of
Company's Direct Research and
Development (R&D) Budget
Provided by Federal Government
Agencies
(See figure in printed
edition.)
Figure I.5: Percentage of
Company's Direct R&D Budget
Provided by State Government
Agencies
(See figure in printed
edition.)
Figure I.6: Percentage of
Company's Direct R&D Budget
Provided by Venture Capitalists
(See figure in printed
edition.)
Figure I.7: Percentage of
Company's Direct R&D Budget
Provided Through R&D Contracts
With Other Companies
(See figure in printed
edition.)
Figure I.8: Percentage of
Company's Direct R&D Budget
Provided From Internal Funding
(See figure in printed
edition.)
Figure I.9: Percentage of
Company's Direct R&D Budget
Provided by Industry Groups or
Trade Associations
(See figure in printed
edition.)
Figure I.10: Percentage of
Company's Direct R&D Budget
Provided by Other Sources
(See figure in printed
edition.)
(See figure in printed edition.)Appendix II
SURVEY QUESTIONS AND RESPONSE
FREQUENCIES FOR WINNERS AND NEAR
WINNERS OF ATP AWARDS
=========================================================== Appendix I
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
OBJECTIVES, SCOPE, AND METHODOLOGY
========================================================= Appendix III
The objective of this report was to examine whether funds from ATP
are used to support research projects that would not have been funded
by the private sector, or if it replaces private funds that would
have otherwise been available. To meet this objective, we requested
from ATP a rank-order listing of all the applicants that received a
score from the Source Evaluation Board\1 for their ATP proposal
during the program's first four rounds of competition (1990-93).
In the first four rounds of competition, the Source Evaluation Board
gave scores only to proposals that had been determined to have "very
high" scientific and technical merit and that had passed a screening
stage in which it was determined that the proposals satisfied the
program's requirements. The Source Evaluation Board then assigned a
score to all of the proposals on the basis of a business review and
all the criteria contained in Commerce's regulations. According to
ATP officials, on the basis of this score, the proposals with strong
technical and business merit were ranked and recorded on a list
before the final oral review stage. ATP provided that list to us.
Using this list, we identified those that received ATP awards as
"winners" and those that did not receive ATP funding as "near
winners."
In our first primary research question, we asked the near winners if
they had continued their proposed projects using other funding
sources after ATP declined to fund them. We developed this question
to shed light on whether ATP winners (given their similarity to the
near winners) would have continued their projects using other funding
sources if ATP funding had not been provided. In our second primary
research question, we asked the winners and near winners if they had
sought funding from other sources before applying to ATP. This
question provided information on whether private-sector sources had
the opportunity to fund the proposed projects before the applicants
sought federal funding.
Our work was structured in three phases. First, we interviewed ATP
officials, winners, and near winners to increase our understanding of
ATP's review process and the applicants' experiences with it. We
spoke with representatives of other research and development (R&D)
funding sources, such as the National Venture Capital Association.
We also reviewed the relevant economic, policy, and evaluation
literature and consulted with outside experts on the overall design
of our assessment.
Next, we designed a computer-aided telephone interview in which we
requested several pieces of information from each applicant.\2
Specifically, we asked questions on (1) the applicant's general
characteristics, such as the size of the company and its sources of
R&D funding; (2) the history of the project put forth in the ATP
proposal, such as whether the project was under way before the
applicant requested funding from ATP; and (3) the project's status
(for winners and near winners that continued their projects using
other funding sources). We got expert review of a preliminary
version of the questions from knowledgeable consultants.
To test the validity of the questions, we pretested a draft survey
instrument with three ATP award winners and four near winners. We
selected them using the following factors: the round of competition
in which the application was submitted, geographic location, type of
applicant (single company applicant or joint venture), company's
size, proposal technology area, and award status (award winner or
near winner). We conducted the first three pretests in person in
Gaithersburg, Maryland and Somerset, New Jersey; we conducted the
remaining four by telephone with winners and near winners located in
Ann Arbor and Auburn Hills, Michigan, and San Jose and Menlo Park,
California. On the basis of the comments and reactions from the
experts' review and our pretests, we revised the telephone interview
questions so that they would be uniformly interpreted and understood.
In the final phase of our work, we conducted telephone interviews
with all the applicants that qualified as winners or near winners
during ATP's first four rounds of competition, a total of 128 (89
winners and 39 near winners). Our survey achieved a 100-percent
response rate. In our findings for both research questions, we
excluded 5 near winners, reducing our total number of respondents to
123. We did this because these five near winners indicated that ATP
had disqualified them late in the review process because new
information indicated that their proposals did not satisfy the
program's basic requirements. For example, in one instance ATP
decided that the applicant would do the project without ATP funding
and in another ATP decided that the project did not focus on
precompetitive or generic research. In our findings for the second
research question only, our total of near winners drops to 28 because
6 near winners eventually received ATP funding in a subsequent round
of competition, eliminating them from our consideration because we
focused only on near winners that found funding from sources other
than ATP.
To examine how certain characteristics affect whether the near
winners continued their projects, we calculated "odds ratios." Odds
ratios measure the association between two variables. The closer the
odds ratio to 1.00, the weaker the association; the further from
1.00, the stronger.\3 To illustrate, table III.1 reports the number
of joint-venture and single- applicant near winners that did and did
not continue their project after ATP declined to fund them.
Table III.1
Number of Joint-Venture and Single-
Applicant Near Winners That Continued
Their Projects After ATP Declined to
Fund
Joint ventures Single applicants
------------------------------ ------------------ ------------------
Continued 3 11
Did not continue 6 8
----------------------------------------------------------------------
The odds ratio is calculated through cross multiplication and
division--(11x6) divided by (3x8)--for a value of 2.75. Rounding to
3, we interpret this odds ratio to mean that single-applicant near
winners were about three times more likely than joint-venture near
winners to continue projects after ATP declined to fund them.
(See figure in printed edition.)Appendix IV
--------------------
\1 For each competition, ATP forms a Source Evaluation Board to rank
the proposals. A typical board consists of about a dozen
senior-level managers from the National Institute of Standards and
Technology (NIST), supplemented with additional technical consultants
from NIST and other federal laboratories. The board members'
backgrounds vary widely. The board may include, for example, an
electrical engineer, a chemist, a biotechnologist, a materials
scientist, a computer scientist, and others with business and
economics expertise.
\2 The computer was programmed to skip questions that were irrelevant
to the individual respondent. For example, those involved in joint
ventures were not asked for the year of their first sale. In these
cases, the computer tabulated the results as a missing response. See
app. II.
\3 For more detail on the theory underlying odds ratios and their
calculation, see William Page, "Interpretation of Goodman's
Log-Linear Model Effects: An Odds Ratio Approach," Sociological
Methods & Research, Vol. 5, No. 4, May 1977.
COMMENTS FROM THE DEPARTMENT OF
COMMERCE
========================================================= Appendix III
(See figure in printed edition.)
See comment 1.
(See figure in printed edition.)
See comment 2.
(See figure in printed edition.)
See comment 3.
See comment 4.
GAO'S COMMENTS
------------------------------------------------------- Appendix III:1
The following are GAO's comments on the Department of Commerce's
letter dated December 21, 1995.
1. While our draft Conclusions section referred to the pace of
research projects, we have added a sentence to our report's
Results-in-Brief on this point. In our opinion, however, Commerce's
overall conclusion based on our survey results overlooks a number of
significant points. First, although ATP appears to enable applicants
to complete their research projects faster, companies still find it
worthwhile to pursue the projects, although on a slower schedule,
without ATP funds. Second, when asked, "What reasons, if any, did
ATP cite for declining your ATP award?," only one applicant said that
ATP had decided the project would be done without ATP funding.
(Subsequently, that project did not find funding elsewhere.) Third,
while our survey results do indicate that a higher percentage of ATP
awardees indicated satisfaction with the technical direction and
progress/outcome of their projects than the near winners, nothing in
our survey supports Commerce's conclusion about what those results
indicate.
2. Our draft Conclusions section referred to the pace of continued
research projects, and we have added a sentence to our
Results-in-Brief section on this point. While Commerce is concerned
with multiple interpretations of one of our survey questions, we
reduced the potential for multiple interpretations by instructing our
interviewers to say specifically, "I'll be asking you several
questions about the history and status of 'the project described in
your ATP proposal,' which I'll sometimes refer to as 'the project.'
By this we mean a project that you consider to be essentially the
same as the one in the proposal." If further clarification was
needed, the interviewer would add, "To be 'essentially the same
project,' it should focus on the same technical work as the one in
the ATP proposal."
We used this wording to allow the respondent to rely on his or her
own judgment in determining if the work that had continued was still
the same project--despite changes in scope, schedule, and riskiness,
among other things--or if in the respondent's judgment, changes have
resulted in a different project altogether. At a minimum, the
project had to include the same technical work, even though, for
example, some intended commercial applications of the work had
changed.
3. As noted in comments 1 and 2, we feel that Commerce's conclusion
overlooks a number of significant points based on our survey results.
4. We have made the suggested change.
MAJOR CONTRIBUTORS TO THIS REPORT
=========================================================== Appendix V
RESOURCES, COMMUNITY, AND ECONOMIC
DEVELOPMENT DIVISION, WASHINGTON,
D.C.
Bernice Steinhardt, Associate Director
Robin M. Nazzaro, Assistant Director
Alice G. Feldesman, Supervisory Social Science Analyst
Andrew J. Vogelsang, Evaluator-in-Charge
George Warholic, Senior Evaluator
Patton Stephens, Staff Evaluator
Kelly Ervin, Social Science Analyst
Mitchell B. Karpman, Senior Operations Research Analyst
Judy Pagano, Senior Operations Research Analyst
OFFICE OF THE CHIEF ECONOMIST
Loren Yager, Assistant Director
GENERAL GOVERNMENT DIVISION,
WASHINGTON, D.C.
Kevin B. Dooley, Senior Evaluator (Computer Science)
*** End of document. ***
|
NEWSLETTER
|
|
Join the GlobalSecurity.org mailing list
|
|
|
