Systematic reviews are vital to the pursuit of evidence-based medicine within healthcare. Screening titles and abstracts for inclusion in a systematic review is an intensive, and often collaborative, step. The use of appropriate tools is therefore important. A new study identifies and evaluates the usability of software tools that support the screening of titles and abstracts for systematic reviews within healthcare research. Covidence and Rayyan are the two software tools that scored most highly in this study.
The research team, in the Prevention Group at the Primary Care Unit, University of Cambridge, was Hannah Harrison with Isla Kuhn (Head of Medical Library Services at the University of Cambridge), Juliet Usher-Smith and Simon Griffin. The work was funded by NIHR through the award of the Systematic Review Methods Fellowship held by Hannah Harrison and by Cancer Research UK.
The research team identified software tools using three search methods: a web-based search; a search of the online “systematic review toolbox”; and screening of references in existing literature. They included tools that were accessible and available for testing at the time of the study (December 2018) and do not require specific computing infrastructure. To be included, they also had to provide basic screening functionality for systematic reviews. Key properties of each software tool were identified using a feature analysis adapted for this purpose. This analysis included a weighting developed by a group of medical researchers, therefore prioritising the most relevant features. The highest scoring tools from the feature analysis were then included in a user survey, in which we further investigated the suitability of the tools for supporting title and abstract screening amongst systematic reviewers working in medical research.
Fifteen tools met the inclusion criteria. They vary significantly in relation to cost, scope and intended user community. Six of the identified tools (Abstrackr, Colandr, Covidence, DRAGON, EPPI-Reviewer and Rayyan) scored higher than 75% in the feature analysis and were included in the user survey. Of these, Covidence and Rayyan were the most popular with the six survey respondents. Their usability scored highly across a range of metrics, with all surveyed researchers stating that they would be likely (or very likely) to use these tools in the future.
Based on this study, the research team recommend Covidence and Rayyan to systematic reviewers looking for suitable and easy to use tools to support title and abstract screening within healthcare research. They report that these two tools consistently demonstrated good alignment with user requirements.
We hope that the illustration of the tools available and they features they support will be a valuable resource to other researchers looking to select the right tool for screening in their systematic review project. Although we suggest that certain tools may be more appropriate for healthcare researchers looking to incorporate a new tool into their screening process, the variety of tools available means there will be something for every systematic reviewer.”
– Hannah Harrison, Systematic Review Methods Fellow, Prevention Group
H Harrison, S Griffin, I Kuhn, J Usher-Smith. Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation. BMC Medical Research Methodology. 15 January 2020
Lucy Lloyd, Communications Manager, Primary Care Unit