Loading...
Predicting article quality scores with machine learning: The UK Research Excellence Framework
Abdoli, Mahshid Levitt, Jonathan Knoth, Petr Cancellieri, Matteo
Abdoli, Mahshid
Levitt, Jonathan
Knoth, Petr
Cancellieri, Matteo
Editors
Other contributors
Affiliation
Epub Date
Issue Date
2023-05-15
Submitted date
Alternative
Abstract
National research evaluation initiatives and incentive schemes choose between simplistic quantitative indicators and time-consuming peer/expert review, sometimes supported by bibliometrics. Here we assess whether machine learning could provide a third alternative, estimating article quality using more multiple bibliometric and metadata inputs. We investigated this using provisional three-level REF2021 peer review scores for 84,966 articles submitted to the UK Research Excellence Framework 2021, matching a Scopus record 201418 and with a substantial abstract. We found that accuracy is highest in the medical and physical sciences Units of Assessment (UoAs) and economics, reaching 42% above the baseline (72% overall) in the best case. This is based on 1000 bibliometric inputs and half of the articles used for training in each UoA. Prediction accuracies above the baseline for the social science, mathematics, engineering, arts, and humanities UoAs were much lower or close to zero. The Random Forest Classifier (standard or ordinal) and Extreme Gradient Boosting Classifier algorithms performed best from the 32 tested. Accuracy was lower if UoAs were merged or replaced by Scopus broad categories. We increased accuracy with an active learning strategy and by selecting articles with higher prediction probabilities, but this substantially reduced the number of scores predicted.
Citation
Thelwall, M., Kousha, K., Wilson, P. et al. (2023) Predicting article quality scores with machine learning: The UK Research Excellence Framework. Quantitative Science Studies, 4(2), pp. 547–573. doi: https://doi.org/10.1162/qss_a_00258
Publisher
Journal
Research Unit
PubMed ID
PubMed Central ID
Embedded videos
Type
Journal article
Language
en
Description
© 2023 The Authors. Published by MIT Press. This is an open access article available under a Creative Commons licence.
The published version can be accessed at the following link on the publisher’s website: https://doi.org/10.1162/qss_a_00258
Series/Report no.
ISSN
2641-3337