Value of using different search approaches

Last revised: 
2017-05-10

The conventional search approach of applying Boolean logic to subject heading and free-text queries continues to dominate literature reviews as it remains an effective method for searching bibliographic databases. However, sensitivity and specificity issues relating to Boolean searching have led researchers to investigate a variety of alternative search approaches: checking reference lists (backward citation), citation tracking (forward citation), using the “similar articles” function in the database, hand searching, and methods of automated retrieval are some examples.

The evidence available indicates that especially so-called “indirect citation relationships”, such as the “similar articles” function or checking “co-cited articles” seem to be superior to the commonly used forward and backward citations.

Direct citation relationships: checking reference lists (backward citation), citation tracking (forward citation)

A Cochrane review (1) included 12 studies examining manually checking reference lists and concluded that there is some evidence to support this method when traditional searching is difficult, but that the studies were heterogeneous and at a high risk of bias. A recent study by Preston et al. (2) showed that search strategies in MEDLINE and Embase identified 85% of studies on diagnostic test accuracy included in 9 systematic reviews; 24 further studies (8%) were identified by checking reference lists. A single study (3) comparing an automated system using the Scopus database with manual reference checking found the automated system to be equally sensitive but considerably more (62.5%) time efficient than the manual method.

Citation tracking seems not to be a useful approach to supplement traditional searching. Three studies (4-6) showed that citation tracking could only identify between 0-12% unique references on a certain subject compared with other search approaches. However, citation tracking appears to offer some added value compared with Boolean searching when performing more challenging searches (e.g. for outcome measures) (7).

Indirect citation relationships: similar articles, co-citing articles

There is evidence that the “similar articles” link in PubMed (formerly known as “related citations”) is an efficient search approach (8-10) that can be used for scoping searches or for identifying the need to update a systematic review. The results of 3 analyses suggest that there is real benefit in using the 'similar articles' link and a simple Boolean search in PubMed. In the study by Waffenschmidt et al (10) the combination of these two search methods resulted in 98% sensitivity when searching for RCTs of drugs. In the studies of Sampson a variation of topics were used and the combination of the search methods reached 90% sensitivity in the original testing (8) and 91% (9) when retested.

In 2 independent studies, Janssens and Gwinn (11) analysed co-cited articles (i.e. the reference lists of articles citing key articles). Sensitivities between 79 and 82% were achieved. The number needed to screen was reduced by 50 to 89% (median). Belter (12) used a combination of approaches (co-cited articles, similar articles, citation tracking, checking reference lists) in the Web of Science database. The method retrieved 74% of the references included in 14 Cochrane reviews. “In progress” references and conference abstracts were rarely identified. Indirect citation relationships (“similar articles” function and co-citing) outperformed simple reference checking and citation tracking of key articles and retrieved about 96% of the relevant references.

Full text search

A paper by Linder (7) evaluated Google Scholar's full text search feature in order to find studies on 'outcome measurement instruments' - terms for which are often omitted from the title, abstract and subject headings of articles. The keyword (full text) search using Google Scholar yielded the highest sensitivity (70%). However, one must bear in mind that searching Google Scholar is time-consuming and difficult, as its functionality (e.g. incomplete bibliographic information, no reference export) remains limited (13).

Automated retrieval methods

At the moment automated retrieval methods implemented e.g. in public available interfaces have not been sufficiently evaluated to decide whether they are a useful approach for performing sensitive searches. One study comparing Boolean searching with ranked querying in MEDLINE (Ovid) reported that ranked retrieval alone was not reliable for a search task requiring high recall (14, 15).

Hand searching

Several current studies comparing hand searching with electronic database searching concluded that there was little or no benefit offered by hand searching. This was found to be the case when searching for additional RCTs (16), diagnostic test accuracy (DTA) studies (17), and occupational health intervention studies published in a language other than English (18). This contradicts the conclusions of a 2007 Cochrane review which found hand searching to be an effective approach when searching for systematic reviews; however, this review is now outdated as its most recent search was performed in 2002 and its main comparison was between hand searching and the "old" (1994), three-section Cochrane Highly Sensitive Search Strategy for RCTs (19).

 

Reference list

  • (1) Horsley T, Dingwall O, Tetzlaff JM, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst Rev. 2009(1):MR000026. [Further reference details] [Publication appraisal] [Free Full text]
  • (2)  Preston L, Carroll C, Gardois P, Paisley S, Kaltenthaler E. Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking. Syst Rev. 2015;4:82. [Further reference details] [Publication appraisal] [Free full text]
  • (3) Chapman AL, Morgan LC, Gartlehner G. Semi-automating the manual literature search for systematic reviews increases efficiency. Health Info Libr J. 2010;27(1):22-7.  [Further reference details] [Publication appraisal] [Free Full text]
  • (4) Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Info Libr J. 2010;27(2):114-22. [Further reference details] [Publication appraisal] [Free Full text]
  • (5) Westphal A, Kriston L, Holzel LP, Harter M, von Wolff A. Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. J Public Health Res. 2014;3(2):177. [Further reference details] [Publication appraisal] [Free Full text]
  • (6) Wright K, Golder S, Rodriguez-Lopez R. Citation searching: a systematic review case study of multiple risk behaviour interventions. BMC Med Res Methodol. 2014;14:73. [Further reference details] [Publication appraisal] [Free Full text]
  • (7) Linder SK, Kamath GR, Pratt GF, Saraykar SS, Volk RJ. Citation searches are more sensitive than keyword searches to identify studies using specific measurement instruments. J Clin Epidemiol. 2015;68(4):412-7. [Further reference details] [Publication appraisal] [Free Full text]
  • (8) Sampson M, Shojania KG, McGowan J, Daniel R, Rader T, Iansavichene AE, et al. Surveillance search techniques identified the need to update systematic reviews. J Clin Epidemiol. 2008;61(8):755-62. [Further reference details] [Publication appraisal] [Free Full text]
  • (9) Sampson M. Complementary approaches to searching MEDLINE may be sufficient for updating existing systematic reviews. J Clin Epidemiol. 2016;78:108-15. [Further reference details] [Publication appraisal] [Free Full text]
  • (10) Waffenschmidt S, Janzen T, Hausner E, Kaiser T. Simple search techniques in PubMed are potentially suitable for evaluating the completeness of systematic reviews. J Clin Epidemiol. 2013;66(6):660-5. [Further reference details[Publication appraisal] [Free Full text]
  • (11) Janssens AC, Gwinn M. Novel citation-based search method for scientific literature: application to meta-analyses. BMC Med Res Methodol. 2015;15:84. [Further reference details[Publication appraisal] [Free full text]
  • (12) Belter CW. Citation analysis as a literature search method for systematic reviews. J Assn Inf Sci Tec. 2016;67(11):2766-77. [Further reference details] [Publication appraisal] [Free Full text]
  • (13) Levay P, Ainsworth N, Kettle R, Morgan A. Identifying evidence for public health guidance: a comparison of citation searching with Web of Science and Google Scholar. Res Synth Methods. 2015;7(1):34-45. [Further reference details] [Publication appraisal] [Free Full text]
  • (14) Karimi S, Pohl S, Scholer F, Cavedon L, Zobel J. Boolean versus ranked querying for biomedical systematic reviews. BMC Med Inform Decis Mak. 2010;10:58. [Further reference details] [Publication appraisal] [Free Full text]
  • (15) Pohl S. Boolean and ranked information retrieval for biomedical systematic reviewing [Dissertation]. Melbourne: University of Melbourne; 2012. [Further reference details] [Publication appraisal] [Free Full text]
  • (16) Armstrong R, Jackson N, Doyle J, Waters E, Howes F. It's in your hands: the value of handsearching in conducting systematic reviews of public health interventions. J Public Health. 2005;27(4):388-91. [Further reference details] [Publication appraisal] [Free Full text]
  • (17) Glanville J, Cikalo M, Crawford F, Dozier M, McIntosh H. Handsearching did not yield additional unique FDG-PET diagnostic test accuracy studies compared with electronic searches: a preliminary investigation. Res Syn Meth. 2012;3(3):202-13. [Further reference details] [Publication appraisal] [Free Full text]
  • (18) Mattioli S, Farioli A, Cooke RM, Baldasseroni A, Ruotsalainen J, Placidi D, et al. Hidden effectiveness? Results of hand-searching Italian language journals for occupational health interventions. Occup Environ Med. 2012;69(7):522-4. [Further reference details] [Publication appraisal] [Free Full text]
  • 19. Hopewell S, Clarke M, Lefebvre C, Scherer R. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev. 2007(2):MR000001. [Further reference details] [Publication appraisal] [Free Full text]