AIP FYI #65: Update on Evaluation of R&D Programs

Last fall the Administration announced its intention to
develop evaluation investment criteria for applied and basic
research. The Office of Management and Budget and the
Department of Energy made initial efforts to apply an
evaluation methodology to Solar and Renewable Energy, Nuclear
Energy, Clean Coal, Fossil Energy, and Energy Conservation
programs (see 2001 FYI #117). While the implementation of
these evaluation criteria for the FY 2003 budget request was
more difficult than first envisioned, the Administration and
Congress are moving ahead on the application of evaluation
criteria to federal R&D programs.

The most recent evidence of the Administration’s determination
is a May 29 announcement in the Federal Register of the
formation of a Performance Measurement Advisory Council. OMB
will establish a six-member panel of experts in performance
measurement “who are outstanding in their professional field
and who are objective.” “Council members will draw upon their
expertise in creating, implementing and evaluating performance
measurement standards and will make recommendations regarding
the types of measures and benchmarking systems that
departments and agencies can employ most effectively to track
program performance.” The council will exist for nine months.

Judging by a hearing of the House Appropriations Subcommittee
on Interior and Related Agencies earlier this spring,
appropriators, DOE officials, the OMB, and the National
Research Council appear to be on the same page about the
feasibility of program evaluation. Ranking Minority Member
Norman Dicks (D-WA) explained that the subcommittee had
contacted the National Academy of Sciences in 1999 to
determine if the $22 billion of taxpayers’ money invested in
energy research in the last two decades was “worth it.” The
resulting report identified significant economic benefits from
the research. Dicks explained that the subcommittee had
worked since then with NAS, OMB, and DOE to develop a benefit
measurement methodology that would be broader in scope.

Robert Fri, chairman of the committee that conducted the
study, testified about the application of a methodology that
was developed and applied to 39 case studies of energy
efficiency and fossil energy programs. Fri testified that the
committee found that “it is feasible to design and to apply
consistently a uniform methodology to a diversity of applied
research programs.” Fri explained that future evaluations
should be made of the research system, and not the progress of
the research programs.

Next to testify was Mark Everson, Comptroller of OMB. He told
the subcommittee that “Understanding the actual return on our
federal investment dollar is all the more essential in the
current environment, when we need to ensure that national
security needs are fully met. We cannot afford to increase
funding for all programs.” He added, “Measuring the
performance of R&D activities is one of the toughest
challenges we face in bringing accountability to our
investment decisions and linking funding to program
performance. . . . We cannot, however, simply abandon efforts
to assess the value of our $100 billion a year R&D
investment.” Everson stated that the Administration was
working with experts and stakeholders to refine the R&D
criteria that will be used in the FY 2004 budget process.
Everson informed the subcommittee that he had spoken with OSTP
Director John Marburger, who said that the pilot test had
given the Administration experience with the evaluation
process, and that it can move forward.

A joint statement by David Garman, DOE Assistant Secretary for
Energy Efficiency and Renewable Energy and Carl Michael Smith,
Assistant Secretary for Fossil Energy, was supportive of
evaluation. “This Administration is strongly committed to
improving DOE’s research and development performance
management system,” they told the subcommittee.

A later statement by Garman and Smith summarized the current
status of the evaluation of R&D programs: “The President and
Secretary Abraham have made it very clear: improving our
methodologies for estimating benefits and using these tools to
develop better performance-based program strategies is one of
the Department’s highest priorities. It will also be one of
the most complex and challenging analytical efforts the
Department has ever undertaken.”

###############

Richard M. Jones

Media and Government Relations Division

The American Institute of Physics


fyi@aip.org

(301) 209-3095

http://www.aip.org/gov

##END##########