Background:

The volume of published peer-reviewed literature has been increasing exponentially in every discipline. This has implications for the time required for conducting systematic reviews (SRs) both at the review and data extraction and synthesis stages, especially when answering less focused, general questions or working in widely researched healthcare topics.

Objectives:

To explore the methods utilised by authors of SRs to manage high volumes of evidence, using the field of COVID-19 vaccine research as an example.

Methods:

This umbrella review was conducted according to a prespecified protocol. In February 2023, we searched MEDLINE, Embase and Cochrane Database of Systematic Reviews (via Ovid) for SRs focusing on efficacy or safety of COVID-19 vaccines in unselected populations. Records were reviewed by a single individual at the title and then full-text review stage. Information was extracted on whether any restrictions were placed (either a priori or a posteriori) on the search terms or characteristics of studies included in the SRs: publication date, study design, location, sample size, vaccine type, quality.

Results:

We initially identified 1,221 records, of which 960 were excluded at title review. A further 158 were excluded at full-text review, leaving 103 for analysis. Overall, none of the SRs explicitly stated having to use an approach to limit the analysed evidence. However, five SRs only completed data extractions for articles that were judged to be of sufficient quality following a risk of bias assessment. Many SRs used restrictive search terms (beyond COVID-19 and vaccine efficacy/safety), which resulted in few records entering the SR in the first place, making the review less resource-intensive but likely to miss relevant data. A large proportion of SRs that had high database hits excluded substantial numbers of records before the title/abstract screening step but did not report further details on the rationale for this. The majority also excluded large numbers of records at the abstract review stage, resulting in small numbers of full texts screened.

Conclusions:

SR authors do not openly discuss issues of dealing with high volumes of evidence when conducting reviews. An open debate on how to manage the growing body of published data is warranted.