Article Text
PDF
Abstract
Metrics which quantify the impact of a scientist are increasingly incorporated into decisions about how to rate and fund individuals and institutions. Several commonly used metrics, based on journal impact factors and citation counts, have been criticised as they do not reliably predict real-world impact, are highly variable between fields and are vulnerable to gaming. Bibliometrics have been incorporated into systems of research assessment but these may create flawed incentives, failing to reward research that is validated, reproducible and with wider impacts. A recent proposal for a new standardised citation metric based on a composite indicator of 6 measures has led to an online database of 100 000 of the most highly cited scientists in all fields. In this perspective article, we provide an overview and evaluation of this new citation metric as it applies to mental health research. We provide a summary of its findings for psychiatry and psychology, including clustering in certain countries and institutions, and outline some implications for mental health research. We discuss strengths and limitations of this new metric, and how further refinements could align impact metrics more closely with wider goals of scientific research.
- adult psychiatry
- child & adolescent psychiatry
Statistics from Altmetric.com
Introduction
Estimating the impact of a scientist’s research can contribute to individual-level decisions, such as funding and promotion, and the evaluation of institutions as they compete for public funding and seek to enhance their reputation and attract students.1 Increasingly, quantitative metrics are used in these decisions as one way to include more objectivity, accountability and transparency in decision-making. In so doing, metrics complement alternative approaches that typically use a system of peer or expert review, as used in the UK’s Research Excellence Framework. Although peer review is associated with substantial resource implications and possible risk of bias, many have argued it is necessary as metric-based approaches used in isolation may not provide meaningful estimates of research impact and are highly variable between fields.
The need for improving these metrics is not only a question of optimising decision-making processes. Research assessment creates a system of rewards and incentives that can shape what is researched and communicated.2 3 At their worst, poor systems can promote low quality research, undermine scientific integrity and hinder translation of research findings to clinical practice.2 Alternatively, if fairly determined and used in responsible ways, metrics have the potential to promote research relevant to clinical practice, replicable and engaging to the wider public.
In this perspective article, we summarise and evaluate a new composite citation metric for individual scientists proposed by Ioannidis and colleagues as it applies to mental health research and discuss the wider implications of metrics in research assessment.
Current metrics
Currently some metrics are used in research assessment, but they all have significant limitations. The Journal Impact Factor is not always strongly correlated with research quality and heavily influenced by a high number of citations to a minority of publications within a particular journal.4 5 Simple citation counts are also inadequate, as they do not account for an individual author’s contribution.6 Genetic consortia provide a clear example of this where an author may contribute information on a few individuals, and this is used in several publications that report on different outcomes or updated analyses. This is complicated by differences in how institutions and academic cultures determine what constitutes authorship. Consistent with this, ‘hyperprolific’ authors, who have published >70 papers in a year, cluster in certain institutions and countries.7 Such authorship practices may have been encouraged by research assessment that privileges authorship, making the number of publications the most valuable currency by which a scientist’s output is rated. An additional challenge to any metric is the extent to which it can be gamed with tactical citation practices or collaborative citation schemes.1
A new impact metric
A recent paper published in PLoS Biology by Ioannidis and colleagues proposes a new citation metric to assess individual scientists.8 This composite score aims to address some of the problems associated with commonly used metrics by accounting for differences in author contributions and reducing the risk of questionable citation practices. The unit of assessment is an individual scientist. The score is based on routinely collected citation and authorship data from Scopus. It is a composite of six citation metrics: total citations; Hirsch h-index; Schreiber hm-index (which adjusts for co-authorship); number of citations to papers where the scientist is the only author; number of citations to papers where the scientist is the first or only author; and number of citations to papers where the scientist is the first, only or last author.6 These six metrics are equally weighted and combined to create the c score out of a total score of 6. This composite score has been calculated in two freely available databases on Mendeley of the 100 000 top-cited scientists across all fields, which include each of the components of the composite score with and without self-citations, and the country and institution for each scientist. The two databases differ in the timescale of the citation metrics reported: one is based on citations in all years providing an overview of career impact; the second provides each of the component metrics and the composite score based on citations in 2017; for example the 1-year h-index is based on citations in 2017. We have focused on the 2017 database as it is less influenced by long publication histories and author age. One problem is that the institutional affiliation may be unavailable or misclassified in this database for some authors, particularly those with multiple affiliations (available at https://data.mendeley.com/datasets/btchxktzyw/1).
From this database, we extracted the 7116 most highly cited researchers publishing in psychiatry, psychology and related fields according to their composite score. This version uses citations in the year 2017 to papers published since 1960 (see online supplementary file 1 ranked by composite score, and online supplementary file 2 ordered by institution). These scientists were selected if they were categorised as publishing in psychiatry or psychology-related fields as primary or secondary fields. This list is therefore inclusive, incorporating for example experimental/developmental psychology, those for whom psychiatry/psychology is the second-ranked field for publications, and many individuals working primarily in neurology and neuroscience. However, we have excluded ‘social’ and ‘comparative’ psychology as they are not directly relevant to mental health research. Table 1 shows citation metrics for the 20 highest-ranked of these researchers in 2017 (ranked according to their composite score removing self-citations). Table 1 also contains the five next most highly ranked scientists with psychiatry as a first of second field, giving a narrower definition of the field (see extended list of scientists working psychiatry in online supplementary file 3).
Citation metrics based on 2017 citations for the 20 top-ranked scientists publishing in psychiatry, psychology and related fields, and additional top five scientists publishing in psychiatry, ranked according to composite score (c score)
To examine how the c score differs from other common metrics for individual authors, we compared ranking based on composite score and 1-year h-index for the top ten highest-cited scientists in psychiatry, psychology and related fields according to the composite score (online supplementary file 4). For two scientists, the h-index resulted in much lower rankings: from 7 (c score) to 260 (1-year h-index), and from 9 (c score) to 119 (1-year h-index). These differences are driven by large number of citations to papers as single/first author for these scientists, which have contributed to the relatively high composite score compared with the h-index which does not account for co-authorship.
Supplemental material
Table 2 shows the country of affiliation for the top-cited scientists in psychiatry, psychology, and related fields. 83% of the highly ranked scientists (5528 out of 6664 with a known affiliation) were based in five countries: USA, UK, Canada, Germany and Australia (online supplementary file 5). These five countries may be distinguished by well-funded research networks, but the relatively large number of highly cited scientists may also be influenced by publication and citation practices within these countries.9 The top-cited mental health scientists had 819 institutional affiliations, and Harvard University had the largest number of affiliated scientists (n=177) followed by the University of Toronto (n=109), King’s College London (n=108), University of California at Los Angeles (n=106) and Columbia University (n=100) (online supplementary file 6).
Supplemental material
Supplemental material
Country-level data on top-cited scientists in psychiatry, psychology and related fields
We have also cross-referenced this database with a list of ‘hyperprolific authors’ (defined as publishing >70 articles in a year), which was outlined in another paper in Nature 2019 (online supplementary file 7).7 Fourteen authors appear both among highly cited scientists in psychiatry/psychology and the hyperprolific list (out of a total of 140 hyperprolific authors in biology/health sciences).
Supplemental material
Strengths and limitations
This new approach to measuring impact (and the online database) is an advance as it attempts to balance citations with authorship, and provides one way to compare citation practices and metrics between individuals and institutions. At the same time, although the database allows comparison between fields, this needs to be done with caution. The equal weighting of the six citation metrics that make up the composite score may favour some subfields. One example is that single-author papers, which are one of the six metrics, are rare in some disciplines.
In addition, the classification of researchers into subject fields is problematic for identifying comparable researchers. Categorisation is based on classification of journals in which the authors publish. As a result, authors who collaborate across multiple fields may be more likely to be misclassified, especially if they publish in more general journals, although most researchers are unlikely to have all their research output in such venues. More problematic is that this type of categorisation by journal means that the author listing is over-inclusive, in particular for neurologists, social psychologists, statisticians and some others. The relative citation rate (RCR) is an example of a recently developed metric which defines scientific fields using a different approach. This article-based metric measures citation rates relative to a co-citation network of papers which are cited alongside the article of interest.10 Thus, the RCR employs a user-driven, flexible definition of the field. Compared with the field definitions provided by Ioannidis et al, the co-citation network may better reflect interdisciplinary fields; however, it has been criticised for potentially penalising papers that are cited outside of their subject area.11 The approach of the RCR is to normalise to a field which is specific to each paper and not readily scrutinised, whereas the proposed c score is more transparent and can be adjusted according to the needs of the user. The author-level RCR metric that can be generated through the NIH’s (National Institutes of Health) iCite interface12 does not account for co-authorship unless users manually select only a subset of an author’s papers. This differs from the approach of the c score that incorporates the Hm-index (accounting for the number of co-authors) and citations to papers as first/last/only author.
A further limitation is that the composite citation score is higher for scientists with a long publication history, and thus who are older. As such, the database contains few individuals who are early in their research career and does not capture individual research trajectories. Incorporating only citations to papers published in recent years might avoid this limitation, but is problematic because recent publications may not yet have accumulated citations.6
In addition, how author contribution is characterised can be criticised. Using single/first/last author status as a proxy for author contribution partly depends on authorship practices.13 As such, clarification of authorship contributions, if done precisely, provides an important complement to this. However, how and whether this can be metricised need further thought. One approach could be the use of structured author taxonomies that categorise types of contributions such as the CRediT (contributor role) taxonomy used by PLoS journals.14 To incorporate these taxonomies into impact metrics, they would ideally need to be used routinely by journals and published.
Impact metrics in context
The use of impact metrics within mental health needs more debate. A range of approaches should be considered. For applied research, a change in clinical practice may be the most important outcome.15 This might be reflected in the uptake of research findings in clinical guidelines, which may not be reflected in citation counts. Other individual contributions are less easily counted such as mentoring junior colleagues, development and maintenance of software and research infrastructure (including databases), and leadership roles within departments or collaborative projects.2
Measures of wider public interest in an individual’s research could also be considered.16 However, their utility should be appraised carefully as they may not measure forms of attention which reflect valuable impact or research quality. Some alternative metrics aim to incorporate news and social media coverage. One example is the number and extent of twitter mentions, which contribute to the Altmetric attention score. Case studies of highly tweeted papers have highlighted examples where large numbers of mentions are driven by one or two accounts which focus repeatedly on a single topic, or patterns of repetitive tweeting likely to come from automated ‘bots’.17 Another example showed that allegations of scientific misconduct drove high levels of twitter attention.18 In addition, twitter counts are poorly correlated with judgements of research quality made by experts or peers.19 20
Several suggestions have been made for ‘best practice’ in research assessment which may guide decisions about the use of impact metrics in mental health.1 4 21 Among these is the need to first establish clear, context-specific priorities for research. Changes in research assessment processes (including the use of novel metrics) should be assessed with outcomes relevant to these priorities. One such priority is the need to reward replication or validation studies. Psychiatry and psychology are among several fields where replication attempts have been disappointing.22 23
Another priority for mental health might be rewarding interdisciplinarity, which may not be adequately acknowledged by current metrics. One example is reflected in priorities for mental health research that were recently set out by the UK’s Department of Health.24 This report emphasises the need to research mental health interventions in non-healthcare settings such as schools, prisons and care homes. Further cross-disciplinary work will be necessary to meet this goal. Each of the fields involved may have different approaches to citation and authorship practices which could affect the comparability of impact metrics.
In conclusion, Ioannidis and colleagues have produced a composite metric that clearly improves on simple publication or citation counts by accounting for authorship contribution and providing tools to examine citation practices. In this paper, we have provided a ranking of individuals in psychiatry, psychology and related fields based on 2017 metrics, and provided an overview of the countries and institutions with the highest ranked scientists. We have argued that impact metrics need interpretation in a context-specific manner, designing and assessing processes which align to the wider priorities of the particular field. This is also the case for journals5 and individual publications,16 and this new score provide a way to evaluate individual scientists. Further refinements to this approach might assist in developing more scalable and transparent ways to assess individual research contributions.
Additional supplementary files are available as a spreadsheet at https://www.psych.ox.ac.uk/research/forensic-psychiatry."
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Correction notice This paper has been corrected since it appeared online first. Supplementary files 1-3 have now been hosted on an external website due to formatting issues.
Contributors MS and SF both contributed to conception of this article, data analysis, drafting and critical revision of the manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.