Peer reviewing search strategies
Search strategy peer review is a process by which the searches designed and conducted for a Health Technology Assessment (HTA) or systematic review are designed, ideally, by an Information Specialist and subsequently reviewed by another Information Specialist. The goal of peer review of search strategies is to detect errors, to improve quality, and to reduce not only the risk of missing relevant studies but also the risk of identifying unnecessarily large numbers of irrelevant records.
As the search strategy is the cornerstone of a well-conducted HTA or systematic review, its quality could affect the results of the final review. A study found that errors in search strategies are common: the principal mistakes being spelling errors, missed spelling variants, truncation errors, logical operator errors, use of wrong line numbers, missed or incorrect use of Medical Subject Heading index terms (e.g. MeSH), and the search strategy not being tailored for use in other databases (1).
How is peer review of search strategies performed?
Peer review of search strategies has been performed informally since literature searching for HTA and systematic reviews began. HTA Information Specialists who are part of information teams have always been able to check colleagues’ search strategies for mistakes, if project time allowed. The search strategy peer reviewer and the Information Specialist who designed the search strategy have been able to meet face-to-face to discuss errors and revisions. However, not all Information Specialists are based in teams and so may be less able to call on colleagues to peer review their search strategies. A forum has been established to enable Information Specialists to submit their searches for peer review by a fellow Information Specialist, on a reciprocal basis (http://pressforum.pbworks.com).
In addition to the forum mentioned above, a tool has been developed which enables Information Specialists to check search strategies in a more formal, structured way. The PRESS (Peer Review of Electronic Search Strategies) checklist is an evidence-based checklist that summarizes the main potential mistakes made in search strategies. This checklist can help Information Specialists to improve and assure the quality of their own search strategies and those of their colleagues. It provides clear guidance for peer reviewers to follow. It can also help non-searchers understand how search strategies have been constructed and what it is they have been designed to retrieve. Full details about the original PRESS project can be found in the original funder’s report (2) and the accompanying journal article (3). Further information, including the original PRESS checklist itself can be found elsewhere (4). An update of the PRESS processes has recently been published (5). This involved an updated systematic review, a web-based survey of experts and a consensus meeting to update the PRESS tools. The 2015 Guideline Explanation and Elaboration (PRESS E&E) incorporates four components:
- six PRESS 2015 recommendations for librarian practice
- four PRESS 2015 implementation strategies
- an updated PRESS 2015 evidence-based checklist
- an updated PRESS 2015 assessment form.
A Guideline Statement has been published as a companion document to the updated CADTH report (6).
The six main domains of the updated PRESS 2015 evidence-based checklist are:
- translation of the research question
- Boolean and proximity operators
- subject headings
- text word searching (free text)
- spelling, syntax and line numbers
- limits and filters
It is recommended that the peer review of search strategies is undertaken at the research protocol phase. If peer review of the search strategy is performed before the searches are conducted, the results downloaded and researchers start the selection of studies, there will be no need to revisit strategies and rerun searches.
Is there any evidence of the value of the peer review of search strategies?
The Agency for Healthcare Research and Quality (AHRQ) has conducted a study assessing use of the PRESS checklist and found that it “seems to cut down the time needed to do the review, increase response, and do a better job of identifying actual errors in search strategies” (7). The time burden of the review process using the PRESS checklist was less than two hours. There is no evidence of whether this tool affects the final quality of systematic reviews or its economic cost. However, CADTH conducted an internal investigation to see whether peer review of search strategies has an effect on the number and quality of articles included in CADTH Rapid Responses reports (8, 9), and found that both the number and quality of relevant articles retrieved were improved.
- (1) Sampson M, McGowan J. Errors in search strategies were identified by type and frequency. J Clin Epidemiol 2006;59(10):1057-63. [Further reference details] [Publication appraisal] [Free Full text]
- (2) Sampson M, McGowan J, Lefebvre C, Moher D, Grimshaw J. PRESS: Peer Review of Electronic Search Strategies. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2008. [Further reference details] [Publication appraisal] [Free Full text]
- (3) Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol 2009;62(9):944-52.
[Further reference details] [Publication appraisal] [Free Full text]
- (4) McGowan, Sampson M, Lefebvre C. An evidence based checklist for the Peer Review of Electronic Search Strategies (PRESS EBC). Evidence Based Library and Information Practice 2010;5(1):149-54 [Further reference details] [Publication appraisal] [Free Full text]
- (5) PRESS – Peer Review of Electronic Search Strategies: 2015 Guideline Explanation and Elaboration (PRESS E&E). Ottawa: CADTH; 2016 Jan. [Further reference details] [Publication appraisal] [Free Full text]
- (6) McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. J Clin Epidemiol 2016;75:40-6. [Further reference details] [Publication appraisal] [Free Full text]
- (7) Relevo R, Paynter R. Peer Review of Search Strategies. Methods Research Report. (Prepared by the Oregon Evidence-based Practice Center under Contract No. 290-2007-100572.) AHRQ Publication No. 12-EHC068-EF. Rockville, MD: Agency for Healthcare Research and Quality. June 2012. [Further reference details] [Publication appraisal] [Free Full text]
- (8) Spry C, Mierzwinski-Urban M, Rabb D. Peer review of literature search strategies: does it make a difference? Presented at Canadian Health Libraries Association (CHLA) Conference; 22-25 May 2013; Saskatoon, Saskatchewan: Canada. [Further reference details] [Publication appraisal] [Free Full text ]
- (9) Spry C, Mierzwinski-Urban M, Rabb D. Peer review of literature search strategies: does it make a difference? Presented at the 21st Cochrane Colloquium; 19-23 Sep 2013; Quebec: Canada.
[Further reference details] [Publication appraisal] [Free Full text]