A systematic method for identifying references to academic research in grey literature

Abstract

Grey literature is research that has not been written with the intent to publish in a traditional journal or book. From this, and due to its unstandardised nature, its impact in academia can be difficult to identify. Research impact can be assessed in multiple ways, with citation analysis a usual method. Impact can include the citing of an output, but in some situations, cited references may be useful in assessing ‘academic impact’. Cited references in grey literature, however, may reflect ‘non-academic impact’ of research such as in policy making, clinical practice or legislation. This study introduces and tests a semi-automatic method to measure cited references in grey literature with unknown standardisation of references. Metadata (lead author surname, title, year) of 2.45 million Russell Group university outputs were collected, added to known citation metadata from a 100-document sample of UK government grey literature, and then searched within each document, assessing the accuracy of 21 proposed variations of matching terms. A ‘best method’ is proposed (lead author surname and title in either order, maximum of 200 characters apart) to show cited references present, enabling the ability to analyse impact differences impact across subject areas and years within grey literature in future studies.