Identifying the Invisible Impact of Scholarly Publications: A Multi-Disciplinary Analysis Using Altmetrics

Abstract

The field of ‘altmetrics’ is concerned with alternative metrics for the impact of research publications using social web data. Empirical studies are needed, however, to assess the validity of altmetrics from different perspectives. This thesis partly fills this gap by exploring the suitability and reliability of two altmetrics resources: Mendeley, a social reference manager website, and Faculty of F1000 (F1000), a post- publishing peer review platform. This thesis explores the correlations between the new metrics and citations at the level of articles for several disciplines and investigates the contexts in which the new metrics can be useful for research evaluation across different fields. Low and medium correlations were found between Mendeley readership counts and citations for Social Sciences, Humanities, Medicine, Physics, Chemistry and Engineering articles from the Web of Science (WoS), suggesting that Mendeley data may reflect different aspects of research impact. A comparison between information flows based on Mendeley bookmarking data and cross-disciplinary citation analysis for social sciences and humanities disciplines revealed substantial similarities and some differences. This suggests that Mendeley readership data could be used to help identify knowledge transfer between scientific disciplines, especially for people that read but do not author articles, as well as providing evidence of impact at an earlier stage than is possible with citation counts. The majority of Mendeley readers for Clinical Medicine, Engineering and Technology, Social Science, Physics and Chemistry papers were PhD students and postdocs. The highest correlations between citations and Mendeley readership counts were for types of Mendeley users that often authored academic papers, suggesting that academics bookmark papers in Mendeley for reasons related to scientific publishing. In order to identify the extent to which Mendeley bookmarking counts reflect readership and to establish the motivations for bookmarking scientific papers in Mendeley, a large-scale survey found that 83% of Mendeley users read more than half of the papers in their personal libraries. The main reasons for bookmarking papers were citing in future publications, using in professional activities, citing in a thesis, and using in teaching and assignments. Thus, Mendeley bookmarking counts can potentially indicate the readership impact of research papers that have educational value for non-author users inside academia or the impact of research papers on practice for readers outside academia. This thesis also examines the relationship between article types (i.e., “New Finding”, “Confirmation”, “Clinical Trial”, “Technical Advance”, “Changes to Clinical Practice”, “Review”, “Refutation”, “Novel Drug Target”), citation counts and F1000 article factors (FFa). In seven out of nine cases, there were no significant differences between article types in terms of rankings based on citation counts and the F1000 Article Factor (FFa) scores. Nevertheless, citation counts and FFa scores were significantly different for articles tagged: “New finding” or “Changes to Clinical Practice”. This means that F1000 could be used in research evaluation exercises when the importance of practical findings needs to be recognised. Furthermore, since the majority of the studied articles were reviewed in their year of publication, F1000 could also be useful for quick evaluations.