Convergent validity of several indicators measuring disruptiveness with milestone assignments to physics papers by experts

Highlights

• Disruptive research diverges from current lines of research by opening up new lines.

• We investigated the convergent validity of several indicators measuring disruptiveness.

• We used a list of milestone papers, selected and published by editors of Physical Review Letters.

• We investigated whether this editor-based list is related to values of the disruption indicators variants.

• The results show that the indicators correlate differently with the milestone paper assignments by the editors.

Abstract

This study focuses on a recently introduced type of indicator measuring disruptiveness in science. Disruptive research diverges from current lines of research by opening up new lines. In the current study, we included the initially proposed indicator of this new type (Funk & Owen-Smith, 2017; Wu, Wang, & Evans, 2019) and several variants with DI₁: DI₅, DI_1n, DI_5n, and DEP. Since indicators should measure what they propose to measure, we investigated the convergent validity of the indicators. We used a list of milestone papers, selected and published by editors of Physical Review Letters, and investigated whether this human (experts)-based list is related to values of the several disruption indicators variants and – if so – which variants show the highest correlation with expert judgements. We used bivariate statistics, multiple regression models, and (coarsened) exact matching (CEM) to investigate the convergent validity of the indicators. The results show that the indicators correlate differently with the milestone paper assignments by the editors. It is not the initially proposed disruption index that performed best (DI₁), but the variant DI₅ which has been introduced by Bornmann, Devarakonda, Tekles, and Chacko (2020a). In the CEM analysis of this study, the DEP variant – introduced by Bu, Waltman, and Huang (in press) – also showed favorable results.

Introduction

The quantitative study of science using bibliometric indicators (publication and citation data) is widespread today (Fortunato et al., 2018). Research into science is aimed “at advancing our knowledge on the development of science and its communication structure, as well as in relation to social, technological, and socioeconomic aspects” (van Raan, 2019, p. 238). This research mainly began in the 1960s when Eugene Garfield created the Science Citation Index and developed into the broad activity of many scientists with the launch of the Web of Science (WoS, now Clarivate Analytics; Wilsdon et al., 2015). Professional bibliometrics is characterized by the use of advanced bibliometric indicators (Leydesdorff, Wouters, & Bornmann, 2016; van Raan, 2019). These indicators mainly refer to field-normalized indicators such as citation percentiles (Bornmann, 2019). However, the use of these indicators in research evaluation resulted in the critique that “citation-based funding could favor mainstream research and bias decisions against more original projects” (Jappe, Pithan, & Heinze, 2018). Citation counts (and indicators derived from these) might not be able to identify discoveries that are defined by Ziman (1987) as follows: “research results that make a significant change in what we thought we already knew” (p. 98). Highly cited papers might follow short-lasting trends, review the existing literature, or introduce research methods and algorithms (van Raan, 2019).

In recent years, several alternative indicators to citations have been developed measuring novelty of research (Bornmann, Tekles, Zhang, & Ye, 2019). Novelty is related to creativity (novelty is an ingredient of creativity, see Puccio, Mance, & Zacko-Smith, 2013) that involves “the production of high-quality, original, and elegant solutions to complex, novel, ill-defined, or poorly structured problems” (Hemlin, Allwood, Martin, & Mumford, 2013, p. 10). The newly developed novelty indicators are based on the view of creativity as novel recombination of existing elements. For example, Uzzi, Mukherjee, Stringer, and Jones (2013) interpreted more frequent unusual cited references (cited journal) combinations as hints to more novel knowledge (see Lee, Walsh, & Wang, 2015). An overview of the different available approaches in scientometrics to measuring novelty based on unusual combinations of existing elements (e.g. cited references or key words) can be found in Wang, Lee, and Walsh (2018) and Wagner, Whetsell, and Mukherjee (2019).

Based on the fact that impact metrics only measure possible use of new ideas, Funk and Owen-Smith (2017) introduced an indicator (based on patent citations) that is intended to measure newness challenging the existing order. Wu, Wang, & Evans (2019) transferred the idea by Funk and Owen-Smith (2017) to bibliometrics by introducing a disruption index (Shibayama & Wang, 2019, proposed similar measures). Azoulay (2019) explains the intuition behind the new indicator as follows: “when the papers that cite a given article also reference a substantial proportion of that article’s references, then the article can be seen as consolidating its scientific domain. When the converse is true – that is, when future citations to the article do not also acknowledge the article’s own intellectual forebears – the article can be seen as disrupting its domain”. A similar index (the so-called dependency indicator, DEP) has been proposed by Bu, Waltman, and Huang (2021), who introduced the dependency indicator from a multi-dimensional perspective of impact measurement. According to this perspective, further information from the citing and cited side is considered to measure performance (which is in contrast to the one-dimensional times cited indicator).

The various disruption indicator variants are connected to the distinction by Kuhn (1962) between normal science and scientific revolutions. Kuhn (1962) presents a theory of scientific knowledge: according to Wray (2017), in this theory, “growth of science is not a continuous march closer and closer to the truth. Instead, periods of rapid growth are interrupted by revolutionary changes of theory. These revolutionary changes of theory are disruptive” (p. 66; see also Casadevall & Fang, 2016). In the context of Kuhn’s theory of scientific knowledge, Foster, Rzhetsky, and Evans (2015) differentiate between two strategies of conducting research (see also Merton, 1957). The different strategy poles can be denoted as traditional and risky innovative thinking or as convergent and divergent thinking. Divergent thinking increases the chance of conducting breakthrough research. Various papers have been published in scientometrics to date that focus on identifying breakthrough, landmark, or milestone papers (e.g. Chen, 2004; Schneider & Costas, 2016; Thor, Bornmann, Haunschild, & Leydesdorff, 2021). An overview of these papers can be found in Winnink, Tijssen, and van Raan (2016).

In this paper, we follow the approach of Bornmann, Devarakonda, Tekles, and Chacko (2020) and investigate the convergent validity of the new disruption indicators. We are interested in the question of whether the indicators measure what they intend to measure, namely the disruptiveness of research. The journal Physical Review Letters (PRL) published a list of milestone papers from the journal. The list can be used to validate the indicators: one can expect that milestone papers have higher indicator values than PRL papers not selected for the list. We investigated whether the indicators are able to identify milestone papers and whether there are differences observable between the indicators in this ability.