Do bibliometrics introduce gender, institutional or interdisciplinary biases into research evaluations?
Name:
Publisher version
View Source
Access full-text PDFOpen Access
View Source
Check access options
Check access options
Authors
Thelwall, MikeKousha, Kayvan
Abdoli, Mahshid
Stuart, Emma
Makita, Meiko
Wilson, Paul
Levitt, Jonathan
Issue Date
2023-06-12
Metadata
Show full item recordAbstract
Systematic evaluations of publicly funded research sometimes use bibliometrics alone or bibliometric-informed peer review, but it is not known whether bibliometrics introduce biases when supporting or replacing peer review. This article assesses this by comparing three alternative mechanisms for scoring 73,612 UK Research Excellence Framework (REF) journal articles from all 34 field-based Units of Assessment (UoAs) 2014-17: REF peer review scores, field normalised citations, and journal average field normalised citation impact. The results suggest that in almost all academic fields, bibliometric scoring can disadvantage departments publishing high quality research, as judged by peer review, with the main exception of article citation rates in chemistry. Thus, introducing journal or article level citation information into peer review exercises may have a regression to the mean effect. Bibliometric scoring slightly advantaged women compared to men, but this varied between UoAs and was most evident in the physical sciences, engineering, and social sciences. In contrast, interdisciplinary research gained from bibliometric scoring in about half of the UoAs, but relatively substantially in two. In conclusion, out of the three potential sources of bibliometric bias examined, the most serious seems to be the tendency for bibliometric scores to work against high quality departments, assuming that the peer review scores are correct. This is almost a paradox: although high quality departments tend to get the highest bibliometric scores, bibliometrics conceal the full extent of departmental quality advantages, as judged by peer review. This should be considered when using bibliometrics or bibliometric informed peer review.Citation
Thelwall, M., Kousha, K., Abdoli, M., Stuart, E., Makita, M., Wilson, P. and Levitt, J. (2023) Do bibliometrics introduce gender, institutional or interdisciplinary biases into research evaluations? Research Policy, 52 (8) Article Number 104829. DOI: https://doi.org/10.1016/j.respol.2023.104829Publisher
ElsevierJournal
Research PolicyAdditional Links
https://www.sciencedirect.com/science/article/pii/S0048733323001130Type
Journal articleLanguage
enDescription
© 2023 The Authors. Published by Elsevier. This is an open access article available under a Creative Commons licence. The published version can be accessed at the following link on the publisher’s website: https://doi.org/10.1016/j.respol.2023.104829ISSN
0048-7333Sponsors
This study was funded by Research England, Scottish Funding Council, Higher Education Funding Council for Wales, and Department for the Economy, Northern Ireland as part of the Future Research Assessment Programme (https://www.jisc.ac.uk/future-research-assessment-programme).ae974a485f413a2113503eed53cd6c53
10.1016/j.respol.2023.104829
Scopus Count
Collections
Except where otherwise noted, this item's license is described as https://creativecommons.org/licenses/by/4.0/