Loading...
Assessing the societal influence of academic research with ChatGPT: Impact case study evaluations
Authors
Editors
Other contributors
Affiliation
Epub Date
Issue Date
2025-05-15
Submitted date
Alternative
Abstract
Academics and departments are sometimes judged by how their research has benefitted society. For example, the UK’s Research Excellence Framework (REF) assesses Impact Case Studies (ICSs), which are five-page evidence-based claims of societal impacts. This article investigates whether ChatGPT can evaluate societal impact claims and therefore potentially support expert human assessors. For this, various parts of 6,220 public ICSs from REF2021 were fed to ChatGPT 4o-mini along with the REF2021 evaluation guidelines, comparing ChatGPT’s predictions with published departmental average ICS scores. The results suggest that the optimal strategy for high correlations with expert scores is to input the title and summary of an ICS but not the remaining text, and to modify the original REF guidelines to encourage a stricter evaluation. The scores generated by this approach correlated positively with departmental average scores in all 34 Units of Assessment (UoAs), with values between 0.18 (Economics and Econometrics) and 0.56 (Psychology, Psychiatry and Neuroscience). At the departmental level, the corresponding correlations were higher, reaching 0.71 for Sport and Exercise Sciences, Leisure and Tourism. Thus, ChatGPT-based ICS evaluations are simple and viable to support or cross-check expert judgments, although their value varies substantially between fields.
Citation
Kousha, K. and Thelwall, M. (2025) Assessing the societal influence of academic research with ChatGPT: Impact case study evaluations. Journal of the Association for Information Science and Technology, 76 (10), pp. 1357–1373.
Publisher
Research Unit
PubMed ID
PubMed Central ID
Embedded videos
Additional Links
Type
Journal article
Language
en
Description
This is an author's accepted manuscript of an article published by Wiley in Journal of the Association for Information Science and Technology (JASIST) on 15 May 2025, available online https://doi.org/10.1002/asi.25021. The accepted manuscript may differ from the final published version.
Series/Report no.
ISSN
2330-1635
EISSN
2330-1643