NASA Contract Award to Resources for the Future: Measuring Research Performance in Space Station Research (redacted copy)
Full document (redacted copy provided by NASA Johnson Space Center)
EXCERPT
SECTION C – DESCRIPTION/SPECIFICATIONS/STATEMENT OF WORK
Measuring Research Performance in Space Station Research
III. A.1. Objectives
We value what we measure and measure what we value. The nation’s investment in the International Space Station (ISS) reflects expectations of scientific return in new discoveries and improvements in the quality of life throughout the world. As expressed in the President’s FY12 budget, the ISS is “humanity’s foothold in space, bringing nations together in a common pursuit of knowledge and excellence.”1 ISS research is taking place within and across several disciplines, including fluid physics, materials science, and other physical and natural sciences. Notably, the ISS provides the only place to study long-term physical effects of the absence of gravity. In the fields of biology and biotechnology, research conducted in this unique microgravity environment has implications for the directionality and geometry of cell and tissue growth. In earth and space science, the ISS serves as a platform in low earth orbit to provide unique observational capacity. And the ISS human research facility enables distinctive opportunities to advance diagnostic telemedicine as well as other important studies pertaining to human life science.
In all cases, the scientific benefits of this research include new knowledge – that is, new data, advances in theory, information, and findings. New knowledge changes what we know, expands how much we know, challenges what we thought we knew, and even plays a role in how we perceive the world. That our knowledge is changing in these ways is measurable, particularly because much of the research is reported in peer-reviewed and other literature. In addition, applied research findings are leading to patents, licenses, and wholly new, tangible advances in technologies, products, and services.2
Because of the unique environment on the ISS, the conduct of multidisciplinary research on board, its international inclusiveness, and the significance of the efforts that culminated in its achievement, the station presents a special challenge for those who wish to analyze its impact. A primary goal of this research is to create a framework to measure and communicates the dynamic scientific progress resulting from ISS science.
Our objective is to design an Atlas of ISS Science, a systematic, rigorous and peer-reviewed framework and analysis that is then visually displayed to compellingly characterize and demonstrate the new knowledge created from the nation’s “lab aloft”3 In essence, the Atlas, and the data and analysis on which it is built, will be the visual representation of the metaphor of maximization of “delta K over K – an expression used by the Chief Scientist of the National Aeronautics and Space Administration (NASA) to appeal to the science community to define, as a task of scientific leadership, how the agency’s research changes our knowledge relative to what we know.4 We will use state-of-the-art “science of science” quantitative techniques applied to bibliometric and other databases of scientific return to document advances in knowledge across a range of ISS science disciplines. The project team includes recognized experts in the blend of disciplines required for this interdisciplinary research: science of science methods, space policy, the economics of innovation, computer programming, and graphics design.
We structure our approach specifically to answer these questions:
– What is the nature of the science return from ISS research as indicated by one possible measure, scientific publications?
– What are the limits and advantages of scientific publications as a measure of science return?
– Are indications of ‘breakthrough’ results discernible in patterns of these publications and citations to them within and across disciplines and subdisciplines?
– Can we discern which disciplines appear to provide the greatest scientific return as indicated by relative trends in the number, rate, and citation influence in peer-reviewed publications?
– Can patterns in scientific publications in non-space based research serve as benchmarks for science return from human space-flight based research?
– How does the science return from ISS research compare to these benchmarks?
– What is the timeframe for realization of returns from ISS research as measured by scientific publications?
– Do scientific publications derived from ISS research appear to be differentially treated in mainstream journals (those not limited to space research) and is there discernible evidence about the cause? ISS results may receive differential treatment in the scientific community/or many reasons, including difficulty in replicability of results, the problem of agency (someone other than the investigator may carry out the experiment and record the results), and applicability (when processes require the ISS to be applied).
We appreciate that there are additional questions that could be asked about valuing ISS research, such as what are the appropriate metrics for the value of commercial products derived from ISS research? What are appropriate metrics for the value of engineering technology demonstration and materials testing on ISS to NASA and the aerospace industry? And, of keen importance, what are appropriate metrics for the value of ISS educational activities leveraging students’ interest in space? Our framework and approach are likely to be useful for future research related to these issues. We therefore see our research as a prototype of an interesting approach to evaluate some important intangible benefits of ISS science. Our proposed research emphasizes the evaluation of knowledge diffusion through the metric of peer-reviewed publications as the first application of our overall approach.
We will combine the NASA ISS databases5 which track and record ISS achievements with additional extensive and comprehensive existing bibliometric databases. To this merged database, we will then apply state-of-the-art tools — based largely in information science and the well-established literature known as the value of information – to carry out multiple statistical analyses. These analyses will allow us to identify changes in the direction, rate, and other characteristics pertaining to the creation and diffusion of new knowledge attributable to ISS science. This research will probe traditional boundaries within and across different scientific disciplines and relative to science carried out in other nonspace-based laboratory settings. Our approach is inspired by and modeled after state-of-the-art practices of the National Science Foundation, the National Institutes of Health, the US Department of Agriculture, and other agencies that now use science-of-science techniques to evaluate advances and patterns in knowledge diffusion.6
Our key outputs will be a computer-based statistical methodology, results from preliminary application of this methodology to the existing ISS publications database, results from tested hypotheses, and research findings centered on the questions above. We will also discuss the advantages and limitations of bibliometric approaches to the special circumstances that pertain to ISS science. In that discussion, we note that although peer-reviewed publications are often seen as a ‘gold standard’ of accomplishment in the scientific community and are a traceable, tractable output, there are recognized limitations and biases in these measures. Even so, we argue that including analysis of peer-reviewed output is a necessary and useful step to fully understanding the value of ISS science.
We anticipate that our methodology will be useful for ISS science management and performance reporting. Our results will be presented in both tabular and atlas (mapping) formats, using the latest advances in the visual display of quantitative information for science of science analyses (see Atlas of Science: Visualizing What We Know (2010, MIT Press at http://scimaps.org/atlas/)). These results will be of value to NASA managers in tracking, assessing, and communicating the value of ISS science. Furthermore, we expect that these methods will enhance management and communication of ISS science and its return to the nation. At the project’s end, ISS management can choose to maintain the performance measurement architecture either within NASA or with a third party, consider whether to extend the approach to include additional dimensions of ISS science performance, and possibly, consider it a viable template for other NASA directorates for analysis and communication of science impact.
Our project aligns with the goals of the Broad Agency Announcement for support and services for ISS by providing a service in support of an ISS laboratory-wide need to define, track, and measure laboratory achievements. It also will provide a basis for enhancing lab management by creating an information architecture that can be used to help maximize these achievements. Our project is also responsive to the request from the Office of Management and Budget (OMB 201 0, pp. 1-2) to heads of executive departments and agencies directing them “to develop outcome-oriented goals for their science, technology, and innovation activities, and target investments toward high-performing programs in their budget submissions.” The OMB further directed that “agencies should support the development and use of ‘science of science policy’ tools that can improve management of their R&D portfolios and better assess the impact of their science, technology, and innovation investments.” As the ISS approaches its second decade of full operation, establishing a framework and methodology for systematic tracking of ISS scientific return will not only respond to the OMB directive and provide information for ISS management, but will enhance public understanding of some of the more intangible benefits of the ISS. Finally, the “atlas of ISS science” we will create will be a compelling, data rich representation of the value of the nation’s investment in the ISS program.
1 Office of Management and Budget, National Aeronautics and Space Administration, The Federal Budget FYI2 Fact Sheet, at http://www.whitehouse.gov/omb/factsheet_department_nasa , accessed 11/30/2011.
2 Applications research is a critical component of the ISS as a national laboratory; however, the goal of this proposal is to design a science of science framework for the basic science research on ISS. The protocol we develop has applicability to applications research, however. The Center for the Advancement of Science in Space (CASIS) is developing a database to attempt to evaluate economic benefits/performance through its portfolio, and we may be able to leverage this to some extent if CASIS has advanced its model of their early research portfolio around year 3 of our proposed effort. We have included the opportunity for CASIS to send a representative to our proposed external steering group to allow regular exchange of information.
3 Blog of Julie Robinson, ISS Chief Scientist.
4 Waleed Abdalati, at open session of Spring 2011 Meetings of the Aeronautics and Space Engineering Board and Space Studies Board, Washington, DC, 6 April 2011.
5 We have worked with NASA’s ISS database managers to identify these data resources.
6 For example, see Office of Technology Assessment 1986, Borner et a1 2009, Fuglie and Heisey 2007
