Assessing the performance of methodological search filters to improve the efficiency of evidence information retrieval: five literature reviews and a qualitative study

Phase 1: semistructured interviews

As this project was funded to inform NICE methods development, the involvement of NICE staff was central to it. We contacted NICE information specialists and project managers and offered them the opportunity to participate in the project. Each interview, which was recorded, lasted for no more than 45 minutes. Once the interview time and date were agreed, confirmation details (date, time, length of interview and interviewer details), along with a topic guide and assurance of anonymity, were sent to each interviewee. After each interview, an e-mail containing a summary of the key points raised during the interview was sent to each interviewee, who was offered the opportunity to check the notes for accuracy and add any additional points that may have occurred to him or her after the interview had ended.

Phase 2: questionnaire survey

Information from the literature reviews and the interviews was used to inform the design and content of a web-based questionnaire. NICE information specialists and project managers were invited to complete the questionnaire but it was also used to collect the views of the wider (national and international) systematic review, HTA and guidelines information community. This information community is well networked and was reached via e-mail lists, as described in Chapter 4 (see Questionnaire methods).

Introduction

Although there are a large number of search filters in existence, many have been developed pragmatically and have not undergone validation. Even for those search filters that have been validated, few have been validated beyond the data in the original publication. This method is described as internal validation and is a less rigorous approach than external validation, in which a filter is tested using a different gold standard from the one used to develop the filter. External validation provides an independent assessment of filter performance and gives a better indication of how a filter is likely to perform in the real world.

Selection of a search filter will depend on the particular searching task and on the performance of the search filter. Thus, it is important to report performance measures for search filters. There are a few tools available that can be used to assess or appraise search filters and these can help in the selection of search filters for specific tasks. 3–5

The aim of this review was to look at the performance measures that are reported for search filters (single studies) and how they are presented. Single studies were defined as those in which a new search filter (or series of filters) was developed, or a search filter was revised, and in which performance measures of the search filter(s) were also reported.

The objectives of the review were to:

• identify and summarise the methods used to develop and validate search filters

• identify and summarise the performance measures used in single studies of search filters

• describe how these performance measures are presented.

Methods

Results

Fifty-eight studies were identified from the ISSG Search Filters Resource. After applying the outlined inclusion and exclusion criteria, 23 studies were identified for inclusion in the review. 7–29 Details from the included studies, grouped according to type of methodological search filter (economic, diagnostic, systematic review and RCT), are provided in Tables 1–4.

Of the 35 studies excluded, 19 were rejected because they were published before 2001. The reasons why the remaining 16 studies were excluded are presented in Table 5.

Discussion

Introduction

A variety of methodological search filters are already available to find RCTs, economic evaluations, systematic reviews and many other study designs. In principle, these filters can offer efficient, validated and consistent approaches to study identification within large bibliographic databases. Search filters, however, are an under-researched tool. Although there are many published search filters, few have been extensively validated beyond the data offered in the original publications. 47–49 This means that their performance in the real-world setting of day-to-day information retrieval across a range of search topics is unknown. 50 Furthermore, search filters are seldom assessed against common data sets, which makes a comparison of performance across filters problematic. Consequently, the use of search filters as a standard tool within technology assessment, guideline development and other evidence syntheses may be pragmatic rather than evidence based. 50,51

As search filters proliferate, the key question becomes how to choose between them. The most useful information to assist search filter choice is likely to be performance data derived from well-conducted and well-reported performance tests or comparisons. Methods exist to test search filter performance and to build the performance picture, including reviews of search filter performance. 48,49,52–54 There is no formal guidance, however, on the best methods for testing filter performance, on which performance measures are valued by searchers and on which measures should ideally be reported to assist searchers in choosing between filters. The performance picture for filters across different disciplines, questions and databases is therefore largely unknown. Different performance measures are reported in studies describing search filters and the process whereby searchers choose a filter remains unclear.

The purpose of this review was to consider the measures and methods used in reporting the comparative performance of multiple methodological search filters.

Objectives

This review addressed the following questions:

• What performance measures are reported in studies comparing the performance of one or more methodological search filters in one or more sets of records?

• How are the results presented in studies comparing the performance of one or more methodological search filters in one or more sets of records?

• How reliable are the methods used in studies comparing the performance of methodological search filters?

• Are there any published methods for synthesising the results of several filter performance studies?

• Are there any published methods for reviewing the results of several syntheses?

Methods

Results

Twenty-one studies were identified as potentially meeting the inclusion criteria for this review based on titles and abstracts2,10,14,15,17,19,22,23,25,33,48,49,55–63 Of these studies, 10 reported the development of one or more search filters, whose performance was then compared against the performance of existing filters10,14,15,17,19,22,23,25,56,57 and 11 reported the comparative performance of existing filters. 2,33,48,49,55,58–63 On receipt of the full articles, three studies55,60,62 were excluded from the review based on the criteria outlined in the methods section. The 18 included studies are listed in Tables 8 and 9 and the excluded studies are listed in Table 10. No studies were identified that synthesised the results of several performance reports or reviewed the results of several syntheses.

Of the 18 studies included in the review:

• eight reported the performance of DTA search filters2,10,14,15,48,49,57,58

• five reported the performance of RCT filters22,23,25,33,61

• three reported the performance of systematic review filters17,19,56

• one reported the performance of filters for economic evaluations59

• one reported the performance of RCT and systematic review filters. 63

The methodological filters evaluated in the included studies had been developed in a variety of interfaces including the interfaces to LILACS, Ovid, PubMed and SilverPlatter. Most studies, however, did not specify the interface used in the development of some or all of the filters being compared. 2,15,17,19,22,23,49,56–59,61 This absence of detail was particularly common in studies in which performance comparison was secondary to the development of one or more new filters. 15,17,19,22,23,56,57

Fourteen studies compared the performance of filters in MEDLINE (various interfaces). 2,10,14,17,19,22,23,33,48,49,56–58,61 Two studies tested filters in MEDLINE and EMBASE. 59,63 One study only tested EMBASE filters15 and one study compared filters in LILACS. 25 Seven of the eight studies comparing DTA filters used MEDLINE to test performance, although the interface used varied. 2,10,14,48,49,57,58

Studies included in the review used a variety of methods to identify relevant filters for comparison, including database searches,2,14,48,49,61 consulting relevant websites14,23,59,61 and contacting experts in the field. 2,49,59 Ten studies used other methods of identifying filters such as using studies that they already knew about or studies that they had conducted themselves. 2,10,17,22,23,49,57,58,61,63 Five studies did not provide explicit details on how the filters for testing were identified. 15,19,25,33,56

The number of filters compared in a single study ranged from 2 to 38. DTA study and RCT filters were the most common filters compared and systematic review and economic evaluation filters were the least common.

Methods used to display performance comparisons/data

All of the studies included in the review displayed the results using a table format, with only two studies supplementing tables of results with graphical (non-tabular) displays of comparative data. 2,48 None of the studies reporting the development of new filters displayed comparative performance in a graphical format. 10,14,15,17,19,22,23,25,56,57

The majority of tables presenting performance comparison data displayed the filters in rows and performance measures in columns (an example is provided in Table 12). The results in the tables in all included studies were provided as percentages or proportions. Within tables, authors generally listed filter results in descending order by the measure of interest, for example decreasing sensitivity. Four studies reporting the development of a filter only included data on comparative performance in the text of the study report. 10,23,25,57

TABLE 12 - Review B: example of a filter performance comparison table as commonly presented in the literature

Reproduced with permission from Harbour et al. 46 © 2014 The authors. Health Information and Libraries Journal © 2014 Health Libraries Journal. Health Information & Libraries Journal, 31, pp. 176–194.

Filter	Number of records retrieved	Filter
		Sensitivity (%)	Precision (%)
RCT filter A	n	X	Y
RCT filter B	n	X	Y
RCT filter C	n	X	Y

Tables that did not list filter results in descending order by the measure of interest instead arranged results by:

• the databases in which the filters were tested15,63

• strategy type (sensitive strategy, specific strategy, optimised strategy)15,63

• filter criteria (sensitive, accurate, etc.)48

• filter alone compared with a clinical subject strategy58

• use or not of an exclusion strategy59

• clinical topic considered in the performance testing33,58

• subject search alone compared with the same subject search with each test filter2

• author or source of published filters15,17

• descending order of cumulative precision or cumulative sensitivity. 56

Tables were also used to present information on the number of studies retrieved58 and the specificity, sensitivity and precision of single terms. 63 One study that reported highest precision combined with sensitivity of > 69% showed the results of the filters meeting these criteria in a separate table. 49

Leeflang et al. 48 used a bar graph to display the average proportion of retrieved and missed gold standard records per filter tested (Figure 1). Whiting et al. 2 presented the overall sensitivity and specificity of each filter tested in a forest plot, including confidence intervals (Figure 2).

FIGURE 1.

Review B: bar chart displaying the comparative performance of filters for DTA studies as published by Leeflang et al. 48 Republished with permission of Elsevier from the Journal of Clinical Epidemiology, Use of methodological search filters to identify diagnostic accuracy studies can lead to the omission of relevant studies, Leeflang MM, Scholten RJ, Rutjes AW, Reitsma JB, Bossuyt PM, 59(3), pp. 234–40, copyright 2006;48 permission conveyed through Copyright Clearance Centre, Inc.

Republished with permission of Elsevier from the Journal of Clinical Epidemiology, Use of methodological search filters to identify diagnostic accuracy studies can lead to the omission of relevant studies, Leeflang MM, Scholten RJ, Rutjes AW, Reitsma JB, Bossuyt PM, 59(3), pp. 234–40, copyright 2006;48 permission conveyed through Copyright Clearance Centre, Inc.

FIGURE 2.

Review B: forest plot of overall sensitivity and precision for each filter in the study by Whiting et al. 2 CEBM, Centre for Evidence Based Health; CRD, Centre for Reviews and Dissemination; HTBS, Health Technology Board for Scotland. Republished with permission of Elsevier from the Journal of Clinical Epidemiology, Inclusion of methodological filters in searches for diagnostic test accuracy studies misses relevant studies, Whiting P, Westwood M, Beynon R, Burke M, Sterne JA, Glanville J, 64(6), pp. 602–7, copyright 2011;2 permission conveyed through Copyright Clearance Centre, Inc.

Republished with permission of Elsevier from the Journal of Clinical Epidemiology, Inclusion of methodological filters in searches for diagnostic test accuracy studies misses relevant studies, Whiting P, Westwood M, Beynon R, Burke M, Sterne JA, Glanville J, 64(6), pp. 602–7, copyright 2011;2 permission conveyed through Copyright Clearance Centre, Inc.

Discussion

Eighteen published articles met the criteria for inclusion in this review. No numerical syntheses of filter performance comparisons were identified, which may be because of the limited availability of performance comparison articles. The majority of included studies reported the development of one or more new filters and compared performance against the performance of existing filters as an adjunct to the main research. This would seem to indicate a focus within filters research on the development of new, ‘better’ filters rather than on a comparison of performance across existing filters. The proliferation in search filters, however, may make it more difficult for searchers to quickly select the most appropriate filter for their particular purpose. The development of increasingly effective filters and the transparent reporting of performance comparisons are important in demonstrating improvements in the performance of new filters compared with current methodological filters.

The number of comparisons of performance varied across study designs. A single study was identified that compared the performance of economic evaluation filters59 whereas studies reporting on the performance of DTA study and RCT filters were much more common. As there have been, until recently, several specialist economics databases [NHS EED, the Health Economic Evaluations Database (HEED) and the Cost-effectiveness Analysis Registry], it may be that filters for the retrieval of economic evaluation studies have been given a lower research priority than filters for other study designs such as RCTs and DTA studies.

Recommendations

The following recommendations for the presentation of filter performance comparisons are made based on the results of this review.

• Studies that compare search filter performance should explicitly report the methods and results to help searchers identify the most appropriate filter for their particular purpose.

• Studies presenting the development of new search filters that include comparisons with existing filters should present detailed methods describing how the performance comparisons were undertaken.

• One or more gold or reference standards should be used for testing filter performance.

• Search filters should be validated on gold or reference standards that are different from those from which they were developed.

• The size of the gold or reference standard(s) should be clearly stated and a sample size calculation presented to justify the size of the standard(s).

• Any translation of filters should be specifically reported in all articles in which a filter has been used in a different interface from that in which it was developed.

• Results should be presented systematically, identifying clearly the best-performing filter for specific purposes (sensitive strategy, specific strategy, balanced strategy).

• When tables of performance results are provided, a consistent format and order should be used to make the information easy to extract.

Introduction

Performance measurement of search filters can be seen as analogous to DTA in that DTA studies aim to reliably differentiate those with a specific disease (relevant studies for searchers) from those who do not have the disease (irrelevant studies for searchers). They also aim to be as accurate as possible in distinguishing cases of disease from cases of non-disease, by minimising false positives (positive results for those who do not have the disease) and false negatives (missing cases of people with a disease). Similarly, search filters aim to identify all relevant studies (true positives) while aiming to minimise the retrieval of irrelevant studies (false positives).

This review explores published guidance and recommendations that inform best practice in the measurement and reporting of DTA and assesses their applicability to the area of search filter performance.

Objectives

• To identify recommended methods for conducting DTA studies and evaluating test performance.

• To identify the diagnostic test performance measurements that have been reported and presented.

• To identify methods to compare DTA performance from primary studies.

• To assess how applicable these measures and methods are to search filter performance and how these measures might add value to the filter selection process.

Methods

We undertook literature searches of electronic databases to identify articles that reviewed methodological aspects of undertaking DTA studies and DTA reviews or provided guidelines and other recommendations on how DTA studies or reviews should be carried out and how the results should be reported. These searches were supplemented by consulting key HTA agencies and Cochrane websites for relevant reports or recommendations.

The following databases were searched in October 2011: Cochrane Methodology Register, The Cochrane Library (Issue 4, 2011), Medion (October 2011), MEDLINE (1950 to October Week 3 2011), MEDLINE In-Process & Other Non-Indexed Citations (28 October 2011) and EMBASE (1980 to Week 43 2011). Full details of the strategies used are reproduced in Appendix 2 along with a list of websites that provided potentially useful reports.

From the electronic database searches, 1454 records were retrieved, which was reduced to 972 records after deduplication. After screening titles and abstracts, 97 records were selected as being potentially useful (Figure 3). The full articles were obtained and read for relevance. In addition, eight reports were obtained from organisation websites. Forty-seven of these reports contributed information to the review. 36,67–112 A list of the remaining 58 retrieved documents that were excluded from the review is provided in Appendix 3. Studies were excluded because they were considered to be irrelevant, described issues or methods that were better expressed or more thoroughly considered in another publication or were duplicate publications. A flow chart showing the selection process for inclusion of studies in the review is provided in Figure 3.

FIGURE 3.

Review C: selection of reports for inclusion in the review. CMR, Cochrane Methodology Register.

We acknowledge that there has been a regrettable delay between carrying out the project, including the searches, and the publication of this report, because of serious illness of the principal investigator. The searches were carried out in 2010/11.

Results for diagnostic test accuracy studies

Measuring diagnostic test accuracy

Contingency table

The primary outcomes of interest in DTA studies are the data required to populate 2 × 2 contingency tables presenting the presence or absence of the target condition or disease, as defined by the reference standard against the result of the index test (Table 13). From this all DTA measures can be derived.

TABLE 13 - Review C: contingency table

Test result	Disease		Total
	Present	Absent
Positive	A (true positive)	B (false positive)	A + B (test positive)
Negative	C (false negative)	D (true negative)	C + D (test negative)
Total	A + C (disease)	B + D (no disease)	A + B + C + D

Measures

Table 14 describes the measures of diagnostic accuracy that are commonly calculated, namely sensitivity, specificity, likelihood ratio (LR), DOR and predictive value.

TABLE 14 - Review C: measures of diagnostic accuracy

LR–, negative likelihood ratio; NPV, negative predictive value.

Measurement	Formula	Definition
Sensitivity	A/(A + C)	Proportion of patients with the disease correctly identified by the test
Specificity	D/(D + B)	Proportion of patients without the disease correctly identified by the test
LR	LR for positive result (LR+) = [A/(A + C)]/[B/(B + D)]	How many times a person with the disease is more likely to receive a particular test result (positive or negative) than a person without the disease
	LR for negative result (LR–) = [C/(A + C)]/[D/(B + D)]
DOR	[(A/C)/(B/D)] = (AD/BC)	Summary measure of the diagnostic accuracy of a diagnostic test
Predictive value	PPV = A/(A + B)	Proportion of patients with a positive test result who are correctly diagnosed
	NPV = D/(C + D)	Proportion of patients with a negative test result who are correctly diagnosed

Two statistical measures of diagnostic accuracy are traditionally used in a clinical setting: the true positive rate or the sensitivity of the test (the proportion of those with the disease who have an abnormal test result) and the specificity of the test (the proportion of those without the disease who have a normal test result). To rule out a diagnosis a test must have high sensitivity whereas to confirm a diagnosis a test must have high specificity. 69,73,80 Both measures are susceptible to spectrum bias76,86 but are not directly influenced by prevalence. 76

The predictive value is the probability of the test correctly diagnosing patients. The PPV is the proportion of patients with a positive test result who are correctly diagnosed. Conversely, the negative predictive value (NPV) is the proportion of patients with a negative test result who are correctly diagnosed. Predictive values depend on the prevalence of the condition in the population being tested. When prevalence is high, it is more likely that a positive test result is correct and a negative result is wrong. 86,87

Likelihood ratios describe the performance of diagnostic tests and can be useful in a clinical setting. The ratio describes whether or not a test result usefully changes the probability that a condition exists. The LR+ is the probability of a person who has the disease testing positive divided by the probability of a person who does not have the disease testing positive. A LR+ of > 10 and a negative likelihood ratio (LR–) of < 0.1 are judged to provide convincing diagnostic evidence. 88 Their interpretation, however, depends on the clinical context. 87

The DOR is a summary measure of the diagnostic accuracy of a diagnostic test. It is calculated as the odds of positivity among diseased persons divided by the odds of positivity among non-diseased persons. When a test provides no diagnostic evidence then the DOR is 1.0. 89 This measure has a number of limitations. In particular, it combines sensitivity and specificity into a single value, hence losing the relative values of the two, and is difficult to interpret clinically. 87

Sensitivity and specificity are based on binary classification of test results (either positive or negative). Test measures, however, are often categorical or continuous and so a cut-off point must be defined to classify results as either positive or negative. As the threshold shifts, the sensitivity and specificity of a test will change, with an increase in one resulting in a decrease in the other. This trade-off at different thresholds can be presented graphically in a receiver operating characteristic (ROC) curve, describing the relationship between the true-positive value (sensitivity) and the false-positive value (1 – specificity), and can be used to identify a suitable threshold for clinical practice. 69 Figure 4 displays a sample ROC curve of test performance using different threshold values from ≥ 5 to > 25.

FIGURE 4.

Review C: example ROC curve.

The Q* value is the point on the ROC curve where sensitivity equals specificity and can be used as a single indicator of overall test performance when there is no preference for maximising sensitivity (minimising false negatives) or specificity (minimising false positives) but can give misleading results if used to compare performance between tests. 69,90 Overall, diagnostic accuracy is summarised by the area under the curve (AUC) and ranges from 0.5 (very poor test accuracy and equivalent to chance) to 1.0. 69,87 The more accurate the test, the more closely the curve approaches the top left hand corner and has a value close to 1.0.

Whiting et al. 87 have undertaken an overview of the various types of graphical presentations that have been used in the DTA literature and describe other graphical displays that could be used to present DTA data. These include dot plots, box-and-whisker plots and flow charts (Figure 5).

FIGURE 5.

Review C: example graphical displays for primary study data. (a) Dot plot; (b) box-and-whisker plot; (c) flow chart.

Dot plots are used for test results that take many values and display the distribution of results in patients with and without the target condition but do not directly display diagnostic performance. Box-and-whisker plots summarise the distributions of true-positive and true-negative groups by a continuous measure. Flow diagrams depict the flow of patients through the study, for example how many patients were eligible, how many entered the study, how many of these had the target condition and the numbers testing positive and negative.

Methods to compare and synthesise diagnostic test accuracy performance from primary studies

Several HTA organisations, in guidance for undertaking DTA evidence synthesis,69,76,90,103,104 recommend using the QUADAS tool or a modified version to assess the methodological quality of primary studies. Undertaking a formal assessment provides an indication of the degree to which the included studies are prone to bias100,102,105,106 and hence the reliability of the study results. A report from the Agency for Healthcare Research and Quality (AHRQ)100 found that there had been a trend in recent years for an increasing number of DTA reviews to formally assess study quality.

Several organisations have developed guidance on carrying out systematic reviews of DTA studies69,75,76,90,102–104 and agree that analysis is more complex than for clinical effectiveness. Combining results from individual studies can be problematic because of the methodological variability (heterogeneity) found across the studies. In particular, combining test accuracy studies with heterogeneity can produce biased, and hence inaccurate, results. 74,79,104,107,108

It is recognised that variability among studies is to be expected. Some of the variability is due to chance, because many diagnostic studies have small sample sizes. The remaining heterogeneity may be the result of differences in study populations or differences in study methods or the result of variation in the diagnostic threshold adopted. 74 Several methods have been described to measure heterogeneity, using graphical plots and statistical tests. 36,76,109 Although it is recommended that such a thorough investigation be undertaken prior to meta-analysis,69,75,76,86,90,100,102–104 this is often not carried out. In a review of 189 systematic reviews,109 only 32% investigated heterogeneity and the authors concluded that this underuse reflected uncertainty about the correct approach to adopt.

It is recommended that only studies using the same reference standard, including substantially similar patients and showing minimal heterogeneity should be synthesised by meta-analysis. 69,74,76,90,104 When this type of complex analysis is undertaken it has been recommended that reviewers should enlist the specialist support of an experienced statistician in the field. 36,69,109 When it is not suitable to undertake meta-analysis a narrative approach should be adopted using graphical presentations, such as forest plots and ROC space plots,69 to provide a visual overview of the results from the included studies.

Paired forest plots (Figure 6) can show the spread of estimated values for sensitivity and specificity for each study. Point estimates are shown as dots or squares and can be sized according to the precision of the estimate or sample size. Confidence intervals around the estimate are shown by horizontal lines either side of the point estimate. If meta-analysis is then undertaken, the pooled estimate is displayed as a diamond. ROC space plots (Figure 7) present the relationship between sensitivity and specificity, with each point representing the summary performance for each study. 69

FIGURE 6.

Review C: example of a paired forest plot. FN, false negative; FP, false positive; TN, true negative; TP, true positive.

FIGURE 7.

Review C: example of a ROC space plot showing summary sensitivity and specificity. df, degrees of freedom.

When performance measures are pooled, separate meta-analyses of sensitivity and specificity data are both the simplest and the most useful approach. 69,104 Such an approach, however, assumes that all included studies are using the same threshold value. Summary ROC (SROC) curves are a form of meta-analysis in which the result is a ROC curve with each data point representing the paired estimate of sensitivity and 1 – specificity from the separate studies (Figure 8). Hierarchical and bivariate statistical models have been developed to estimate the SROC curve. 110,111 The SROC curve is a useful presentation when a threshold effect is observed. The curve provides a global summary of test accuracy and, as with a ROC curve, shows the trade-off between sensitivity and specificity at different threshold levels. It does not, however, provide a single statistic of overall test performance104 and a review has indicated slow uptake of these newer methods. 112

FIGURE 8.

Review C: example of a paired SROC curve, comparing the accuracy of test 1 with that of test 2.

Other graphical methods that can be used to present data in a way that is useful in a clinical context have been suggested. 87 The two main methods are LR nomograms and the probability-modifying plot. These graphs enable the clinician to estimate the post-test probability of a patient having the disease, based on their pretest probability, when the LRs of tests are known.

Whiting et al. 87 reviewed the graphical presentation of diagnostic information in 49 systematic reviews. Just over half (53%) of the reviews used graphical displays to present the results. ROC plots were the most common type of graph and were included in 22 reviews (45%), whereas forest plots were used in 10 reviews (20%) to display individual study results. In another review of DTA reviews, Honest and Khan101 found that, when meta-analysis had been undertaken, pooled sensitivity or specificity was reported in 35 out of 60 (58%) reviews, pooled predictive values in 11 out of 60 (18%) reviews, pooled LRs in 13 out of 60 (22%) reviews and pooled DORs in five out of 60 (8%) reviews. SROC plots were reported in 44 out of 60 (73%) of the meta-analyses. Dinnes et al. 109 noted that, out of 189 systematic reviews included in their review, 30% had involved narrative analysis and, when meta-analysis had been undertaken, 52% statistically pooled data, 18% reported SROC plots and a further 30% employed both techniques.

Summary

• Diagnostic test accuracy studies should be carried out on a sample of patients who are representative of the target population, particularly in terms of disease state, and should use an appropriate reference standard with interpreter blinding to previous test results.

• Sensitivity (true-positive rate) and specificity (true-negative rate) are the most commonly reported outcomes and are subject to spectrum bias.

• Predictive values, used to calculate the probability of a test giving a correct result, are influenced by the disease prevalence in the population.

• LRs are useful in a clinical setting to determine the probability of a patient having the target disease.

• DORs provide a summary measure combining sensitivity and specificity but are difficult to interpret clinically.

• ROC curves present sensitivity and specificity pairs at different test thresholds, whereas the AUC gives an overall value of DTA.

• International HTA organisations that have addressed the issue recommend that DTA studies should present 2 × 2 contingency tables, sensitivity and specificity pairs and LR pairs.

• Several types of graphical presentations can be used to display DTA data but these have not been used extensively in the DTA literature.

• In undertaking systematic reviews of DTA studies, heterogeneity between studies is a common feature and should be investigated before combining data in a meta-analysis.

• A narrative approach, presenting forest plots and ROC space plots, is recommended when heterogeneity exists.

• Poor quality in relation to methodology and reporting affects the inferences that can be drawn from DTA studies.

Applicability to research in search filter performance

Diagnostic test accuracy and search filter studies share similar characteristics in that both evaluate the performance of an index test (or search filter) against that of a reference standard in the same sample of patients (or records). In the clinical literature, the reference standard should be the best available method to identify the ‘target condition’. In the search filter literature the reference standard usually refers not to the method per se but rather the set of relevant records that the method has been designed to identify. 51,76 Typically, the reference standard is described as the records obtained by hand-searching a set of journals over a specified time period (i.e. the ‘positive’ records in the sample to be tested) rather than describing the reference standard as the method used (i.e. ‘hand-searching’). Other reference standards used, such as the records of included studies from systematic reviews or studies held in a specialised register, again conflate the method and the sample. In these cases, the method used is implicit: searching and screening to identify relevant studies. Although the terminology is different, the principle is the same: the results of applying the index test or filter to a sample are compared with the results of a method that is considered to be robust.

Methods for conducting a search filter performance study

Guidance on measuring DTA performance emphasises the importance of using a sample of patients who are representative of the intended population, particularly in relation to the target condition, otherwise the study may be subject to spectrum bias. Likewise, when measuring search filter performance of a filter intended for a particular bibliographic database, the set of records on which the filter is tested should be representative of that database.

When hand-searching is undertaken, the selection of journals used should be representative of the journals that are indexed in the bibliographic database for which the filter is intended. In terms of subject/clinical focus this can be problematic because hand-searching is labour intensive and so the requirement to include a representative selection of journals has to be balanced against the need to obtain a sufficient yield of articles efficiently by using specialist high-yield journals. For example, when testing or developing a DTA study filter, hand-searching radiology journals may be an efficient way to provide a good yield of DTA studies but these will not be representative of health-care journals in general. The underlying prevalence in the test sample is likely to be much higher than for the whole database and will result in overestimation of the internal precision of the resulting filter. Other factors to consider in selecting journals might include language (including UK/US variations), impact factors and the inclusion of abstracts in the database records.

Using included studies from reviews or a study register such as CENTRAL is likely to provide a wider range of publication sources. The original search strategies used in the reviews should be sensitive and ideally not include methodological search filters so that bias is not introduced by limitations in the searches. However, the inclusion criteria used to select the studies for the reviews or registers may also introduce bias. For example, the reviews may include only large RCTs so the reference standard under-reports all RCTs on the review topic retrieved by the subject search. This will impact on the measurement of the performance of the search filter, particularly in terms of reducing precision. Reduction in the NNR, which calculates a reduction in the number of records to be screened, may be a more appropriate parameter in these circumstances.

As bibliographic databases have changed over time in terms of both content and indexing vocabulary, the publication span for hand-searched journals and included studies also deserves attention to ensure representative coverage.

The DTA literature mentions sample size as another important issue, although the literature suggests that this is seldom formally reported. This is also the case for search filter performance literature. The performance measures calculated for the test sample are an estimate of the population value and uncertainty around these performance measures (as demonstrated by the confidence intervals) decreases with an increase in the sample size.

Tables have been published to assist in sample size calculations for DTA studies and would be appropriate to use for search filter studies. 83 An example is shown in Table 15.

When the prevalence of relevant records across the results set is expected to be < 0.50 (which would be the case in search filter design studies), the following steps can be followed to calculate the sample size:

Reference set:

• for example, based on the assumption that the expected specificity of the filter will be 90% (see Table 15, seventh row) and

• if we specify that the minimal acceptable lower confidence limit is, for example, 0.75 (see Table 15, sixth column)

• then the minimal sample size for the reference set (Ncases) is read from the table as 70 records.

Results set:

• the minimum results set is calculated from the equation (Ncases) + Ncontrols, where Ncontrols = Ncases[(1 – prevalence)/prevalence]

• if we assume that the expected prevalence of relevant records is 5% of the hand-search or search results then the results set is calculated as 70 + 70[(1 – 0.05)/0.05] = 70 + 1330 = 1400 records

A lower assumed prevalence would increase the size of the required results set. For example, for a 1% assumed prevalence, the reference set should be 7000 records.

Other main sources of bias mentioned in the DTA literature relate to the suitability of the reference standard (appropriate to the target condition and independent from the index test) and to the methods used in carrying out the test (interpreter blinding and standard interpretation of the results). In terms of search filter testing, there are factors that might affect the independence between the index test and the reference test. For example, when screening journal abstracts, hand-searchers should be unaware of the indexed terms used in the corresponding database records and, when the included studies in a review are used as the reference set, the original search strategy terms should not include any of the search terms being tested. Ideally, the review’s search strategies should have no methodological terms.

Irrespective of how the reference standard is obtained, methods should be standardised to help limit variability. When multiple hand-searchers are involved in creating the reference standard, they should work to the same inclusion and exclusion criteria, which match the study type(s) that the test filter is intended to retrieve, and reviewers’ reliability should be formally assessed before commencement.

Checklists similar to the QUADAS tool85 and the STARD statement,68 but designed for search filter studies, would enable a formal assessment of study quality and might assist search filter researchers to adopt a more consistent and high-quality methodology. Examples of checklists for search filter studies have been reported,3,4,51 with only that of Bak et al. 3 including a scoring system.

Search filter performance measures

In DTA performance measurement, sensitivity and specificity are the most commonly reported values and are judged to be essential by most guidance. Other measures that tend to be reported are PPVs and NPVs, LRs and DORs. For search filter performance, sensitivity (or recall) is almost universally reported, with specificity and precision (equivalent to PPV) the next most frequently reported measures (see reviews A and B).

Specificity and precision (or PPV) are both measures of the false-positive rate; the former is measured in relation to the total number of negatives whereas the latter relates to the number selected by the filter or test. In situations in which data are highly skewed, as is the case with literature retrieval, when typically a very small fraction of records in a bibliographic database are relevant (positive), precision rather than specificity better captures changes in the false-positive rate. This is because the number of false positives is being compared with a relatively small number of true positives rather than the much larger number of true negatives. 113

This phenomenon is illustrated by the precision and specificity of the three filters shown in Table 16. Filter A has 83% sensitivity, 25% precision and 92% specificity. For filters B and C, the number of relevant records retrieved is the same and so sensitivity is maintained at 83%. The number of retrieved irrelevant records, however, varies. For filter B, the number has more than doubled from 750 to 1750 and consequently precision has been halved to 12.5% whereas specificity has been reduced from 92% to 82%, a reduction of only 11%. A large increase in the number of irrelevant records retrieved has led to a substantial change in precision but a relatively small change in specificity. For filter C, the number of retrieved irrelevant records has increased almost seven-fold, resulting in specificity being reduced by half to 46%. The resulting change in precision of approximately 80%, from 25% to 4.6%, again better reflects the huge increase in number of irrelevant records being retrieved.

In the context of evidence synthesis, a searcher’s primary interest is to know how many relevant records have been missed by the search as well as how many retrieved records are irrelevant but will still require to be screened. These factors affect how efficiently and accurately data gathering for evidence synthesis will be carried out. Sensitivity and precision are therefore of most interest. A busy clinician, however, may prefer to retrieve a small set of records in which a high proportion are relevant, and so high precision is very important whereas sensitivity is less important. Knowing the proportion of irrelevant records in a bibliographic database that have not been retrieved, as measured by specificity, is of lesser value.

Likelihood ratios, although useful in a clinical situation for indicating a patient’s probability of truly having the target condition, are probably of less use in literature searching because searchers are less interested in individual records. The DOR, sometimes referred to as ‘accuracy’, is a single indicator of diagnostic performance and has occasionally been calculated in search filter literature. As with a clinical situation, however, it provides a summary measure and hence does not provide as much useful information on performance as other measures.

Presentation of results

In search filter performance studies, tabular presentation of the results is the norm. DTA study guidance suggests several different graphical presentations that can be used, although they seem to be underused in the DTA literature.

In clinical situations, test measurements are frequently continuous in nature and so thresholds are set to define positive and negative results. The trade-off between sensitivity and specificity at different thresholds is often graphically presented in a ROC plot. This situation does not occur in standard literature searching: a search filter produces a binary result, either selected or not. At the filter development stage, however, a ROC plot could be a useful way to display the performance characteristics of variations in a filter, showing the change that results from the inclusion or exclusion of particular search terms.

Other graphical presentations that have been used in the DTA literature include dot plots, box-and-whisker plots and flow diagrams. Plots can be used for tests that can have a range of values so again would not be applicable to search filter performance. A flow diagram, however, could be considered as a method for presenting search filter performance.

Comparing the results of search filters

Systematic reviews of the DTA literature are complex, largely because of the variability (heterogeneity) between studies in terms of the reference standards that have been used and the populations that have been tested. When heterogeneity exists, meta-analysis is not recommended and a narrative approach is advised using graphical presentations such as forest plots and ROC space plots.

In the search filter literature, a variety of approaches have been adopted to test search filters using different search interfaces and so heterogeneity is likely to be present between filters. There have been few systematic reviews undertaken in the search filter literature and these have tended to adopt a different approach from that taken in the DTA literature. Although DTA reviews frequently compare studies that have evaluated the performance of one index test against the performance of the same reference standard but in different samples, search filter reviews published to date compare several search filters using both the same reference standard and sample (review B). In this situation, synthesising the results is not applicable; rather, we can directly compare performance between filters. These reviews have tended to display the results only in tabular form but ROC space plots or paired forest plots would be highly appropriate for displaying these comparisons. Displaying the results using graphs may convey them more effectively and assist users to choose between filters.

Conclusions

Guidance on conducting and analysing the results of DTA studies is applicable to several aspects of search filter research. The identification of a representative sample of records, of sufficient size and using a standardised approach will assist in producing robust and generalisable results. Although appropriate performance measurements are generally reported, the greater use of some graphical presentations may facilitate the dissemination and interpretation of results.

Objectives

The objective of this review was to identify any published research into how searchers (information specialists, librarians, researchers and clinicians) choose search filters based on the information presented to them.

Methods

Studies were eligible for inclusion if they reported criteria or methods that searchers used to choose filters, for example:

• the characteristics of the filter, such as how the filter was designed, what performance measurements were used and the currency of the filter

• how searchers appraised the filter designs, for example, did they use the ISSG critical appraisal tool,4 the Canadian Agency for Drugs and Technologies in Health (CADTH) tool3 or other methods to appraise search filters to inform their choice

• whether or not searchers asked for advice from others on the choice of filters, including colleagues, recognised experts in the field (such as members of the ISSG or the McMaster Hedges project team) or other professional networks

• where searchers found the filter; for example, did they choose the filter because they found it in a source they regarded as ‘reputable’ (such as MEDLINE/PubMed or the ISSG Search Filters Resource) or in published guidance documents [such as those produced by the Centre for Reviews and Dissemination (CRD)69 or Cochrane114].

Studies were excluded if they were not specifically about search filter choice or were in languages other than English. Studies from any discipline were eligible.

Although there is a large volume of literature on resource selection, this is not directly applicable to this very specific type of tool selection. At the protocol stage we decided against searching for generic literature about resource selection ‘choices’ as this was likely to retrieve a large number of records with little or no direct relevance to the review question.

To identify relevant studies we searched databases in a number of disciplines including information science and health care. Table 17 summarises the database and other resources searched to identify relevant studies.

The search strategy consisted of subject indexing (e.g. MeSH, Emtree) and free-text terms (in the title and abstract). It included search terms for ‘searchers/information specialists’ in combination with terms for ‘choice/decision’ and terms for ‘methodological search filters’. No date or language limits were applied to the search. Full search strategies are listed in Appendix 4. Records were downloaded from databases and then imported into EndNote X5 bibliographic software (Thomson Reuters, CA, USA), which allowed categorisation and coding, as well as streamlining of the production of draft and final reports. Duplicate records were then removed.

The titles and abstracts of the records identified in the searches were assessed for relevance. The intention was to select those studies reporting how searchers make choices about search filters. Studies not specifically about search filter choice and studies in languages other than English were excluded.

We acknowledge that there has been a regrettable delay between carrying out the project, including the searches, and the publication of this report, because of serious illness of the principal investigator. The searches were carried out in 2010/11.

Results

In total, 2266 records were identified by the searches. Table 18 shows the numbers of records by resource identified from the searches.

After the removal of duplicates, 837 records remained for assessment. The titles and abstracts of these 837 records were assessed for relevance and no records met the inclusion criteria (Figure 9).

FIGURE 9.

Review D: numbers of records retrieved and assessed for relevance. CMR, Cochrane Methodology Register; CPCI-S, Conference Proceedings Citation Index – Science; CPCI-SSH, Conference Proceedings Citation Index – Social Science and Humanities; LISTA, Library, Information Science and Technology Abstracts; MEDLINE In-Process, MEDLINE In-Process & Other Non-Indexed Citations; SCI, Science Citation Index; SSCI, Social Science Citation Index.

Discussion

The search strategy used search terms relevant to systematic review methods (‘search strategy’, ‘search filter’, ‘information specialist’, ‘choice/decision’) and as a result a high proportion of the records identified were systematic reviews, which typically report search strategies in their abstracts. In total, 48% (402/837) of the records assessed were Cochrane reviews, which report their methods in detail and whose abstracts tend to include search terms similar to those used in this search strategy. Many other non-Cochrane reviews were also identified for the same reason. This also explains the high number of duplicate records retrieved as Cochrane reviews were identified across most of the databases searched.

Studies about the creation, testing, evaluation and awareness of search filters were also identified because of the similarity of the search terms used in the strategy and those used in the bibliographic records. Other studies looked at search techniques for identifying study populations by age or sex; investigated the differences between databases and database interfaces; and discussed the growing importance of searching via the internet. In addition, a significant number of records were completely irrelevant, such as those about searching bioinformatics (genes, proteins) databases.

However, we did not identify any studies that had explored how searchers select search filters. The absence of studies was not unexpected, despite the fact that our searches were relatively sensitive and were undertaken across a wide range of resources (including databases covering health care and information science as well as HTA organisation websites).

It was decided when developing the protocol that, given the resources available for this project, it would not be possible to undertake broader searches to identify research about how searchers or information specialists (including librarians) make choices about the resources/tools they use. It was felt that this literature would be very large as it would include library stock selection, database selection and other situations in which informed choice is required. It may be that this literature could suggest how information seekers choose between tools. The literature would not be specific, however, to the choice of search filters and might be qualitatively different as many stock selection decisions may be governed by factors such as cost and subject coverage rather than sensitivity and precision.

There is literature about the development and quality of search filters, as well as research comparing published filters, but we did not identify any studies reporting the use and choice of filters by searchers in practice. A survey about the awareness of search filters among searchers was published in 2004 and, although awareness of filters was relatively high at that time, usage was still low. 5 Since that questionnaire was undertaken, the promotion of search filters through the ISSG Search Filters Resource, through training courses conducted in the UK, the USA and elsewhere and through the increasing numbers of published filters may have increased awareness and usage by searchers. We have not identified any current published evidence, however, to support this. Investigations of how searchers are choosing filters seem not to have been published.

Introduction

Database searchers have access to a range of methodological search filters that have been designed to retrieve records relating to studies that employ a particular research design. It is unclear, however, what factors influence the choice of an appropriate filter. As search filters can be viewed as analogous to diagnostic tests (as outlined above), it is hypothesised that the factors that lead clinicians to choose between diagnostic tests or health-care organisations to choose between screening tests might offer insights into how searchers do, or might in the future be encouraged to, make choices about search filters.

Objective

To identify and summarise evidence, in a narrative review, on factors that influence clinicians’ choice between diagnostic tests.

Methods

Evidence for this review was obtained from literature searches of the major health-care databases and consultation of national screening programme websites. MEDLINE, MEDLINE In-Process & Other Non-Indexed Citations and EMBASE were searched in March 2011 and CINAHL, PsycINFO and Applied Social Sciences Index and Abstracts (ASSIA) were searched in June 2011. The search strategies that were used are reproduced in Appendix 5. No date restrictions were applied but a pragmatic decision was taken to search only for English-language publications. Reference lists of relevant studies were scrutinised and citation searching of key articles was undertaken in Scopus and ISI Web of Knowledge. Results were downloaded into Reference Manager 12 (Thomson ResearchSoft, San Francisco, CA, USA). Titles and abstracts were screened and full-text copies of all studies deemed to be potentially relevant were obtained and assessed for inclusion by one researcher.

We acknowledge that there has been a regrettable delay between carrying out the project, including the searches, and the publication of this report, because of serious illness of the principal investigator. The searches were carried out in 2010/11.

Results

The electronic searches retrieved 1559 records after deduplication (Figure 10). Titles and abstracts were screened and 47 records were selected for full-text assessment. Seven studies met the inclusion criteria. 115–121 Table 19 provides details of the included studies. The references and citations of these seven publications generated an additional 38 articles for further checking, none of which met the inclusion criteria.

FIGURE 10.

Review E: numbers of records retrieved and assessed for relevance.

Studies were excluded for a variety of reasons. One-quarter (10/40) of the excluded studies considered the reasoning that underpins diagnostic decisions, mainly factors that can lead to errors and suboptimal diagnostic strategies, and one-quarter (10/40) surveyed the use of a range of tests for different conditions. Six articles examined factors that influence the diagnostic process or adopted strategy, characterised by a stepwise series of hypothesis testing using information from a variety of sources and series of tests. These included symptoms elicited from patients, patient and physician characteristics and structural issues.

Other reasons for exclusions were examination of patient choice or compliance (n = 4), use of interventions designed to influence test ordering behaviour (n = 2) and use of an economic model to assess screening strategies (n = 1). An additional two articles did examine test choice but did not elicit the reasons involved. Appendix 6 provides details of the excluded studies together with the primary reason for exclusion.

Of the seven studies that met the inclusion criteria, none was set in the UK. Four studies were set in the USA,115,116,118,120 one was set in Canada,121 one was set in Switzerland117 and one was multinational. 119 Information from the clinicians was obtained by survey (n = 3117,119,121), questionnaire (n = 2115,118) or interview (n = 2116,120) and the number of participants ranged from 11116 to 1184. 117 Three studies looked at cancer screening tests (two for colorectal cancer),117,120,121 two at imaging tests for pulmonary embolism,115,119 one at balance assessment tests116 and one at tests to diagnose pertussis. 118

Four studies mentioned high test performance as a reason in support of clinician choice. In the study by Jha et al. ,115 90% of emergency physicians and 95% of radiologists who responded to a questionnaire cited test accuracy as a reason for test choice. Both Stein et al. 119 and Zettler et al. 121 noted that perceived test performance was a factor in decision-making whereas Sox et al. 118 reported that 70% of participants who had received information on DTA performance chose the best-performing test compared with 21% of controls who had not received this information. One further study, which interviewed physiotherapists about balance assessment tests, found that the perceived value of information gathered was a deciding factor in clinician choice of test rather than the psychometric properties of the assessment tests. 116

Two studies reported economic factors: the perceived cost-effectiveness of colorectal cancer screening tests121 and the perceived added benefit as set against resource use of various diagnostic tests for pulmonary embolism. 119 One further study looked at the influence of equity in physician choice. 117 The participants were asked to choose between one test given to the whole population and a better (in terms of lives saved) and more expensive test given to half of the population. Three-quarters (75%) opted for the universal test although the better, more expensive test was seen as being more acceptable if clinical factors determined who would receive it.

Two studies reported patient characteristics as factors influencing test choice. Stein et al. 119 mentioned age and sex whereas Wackerbarth et al. 120 identified family history as an influencing factor for screening at an earlier age. Patient acceptance of the proposed tests and whether or not the tests were covered by patients’ insurance coverage were also mentioned. 120

Other factors considered were clinician experience (McGinnis et al. 116 reported this as the primary influence on test choice for balance assessment), mortality reduction121 and adverse events, primarily in terms of radiation exposure. 119 The study by Jha et al. ,115 which took place in an emergency department, found that ready access to the test and whether or not 24-hour interpretation support was available were the two most frequently reported factors after test performance.

In addition to the studies identified in the review, information on selection criteria for four screening programmes was identified (Table 20). Three of the four screening programmes that provided information were national, set in the UK,122 USA123 and Australia. 124 The fourth, providing criteria for cancer screening, was produced by the World Health Organization. 125 Most programmes identified high test performance in terms of sensitivity,123–125 specificity,124,125 PPV124,125 and/or NPV124,125 as important. The UK programme122 stipulates that the test should be precise and that the distribution of test values in the target population should be known and a suitable cut-off level should be defined.

Other characteristics listed included being safe,122,124,125 being reliable,124 having been validated,122,124 easy to administer122,124 and being acceptable to the target population. 122,124,125 All of the programmes consider factors other than test performance. The effectiveness of undertaking a screening programme, in terms of morbidity and mortality reduction, should be established,122,123,125 with effective identification of disease at an early disease stage124 and the availability of effective treatment. 123,125 The condition under investigation should be sufficiently prevalent123,125 so that a screening programme can be effective. The UK programme122 adds that an agreed policy of further diagnostic investigation and disease management should have been agreed. Both the UK122 and the USA123 programmes mention that the perceived benefits of the screening programme should outweigh any harms resulting from screening and treatment.

Discussion

From this overview it seems that there is limited evidence to clarify how clinicians choose between diagnostic tests. What evidence there is suggests that test performance is the main factor that informs their choice. It has been reported, however, that a substantial proportion of clinicians have an inaccurate understanding of test performance parameters and apply them inaccurately126–131 and so it may be the case that choices are being based on false assumptions. Other factors mentioned in more than one study were the pretest probability of having the condition, as defined by patient characteristics, patient acceptance of the test and the costs involved in carrying out the test, which are factors that are not readily transferable to the search process. Additional attributes reported related to the particular scenario being investigated: the harmful effect of radiation when imaging tests were being considered and the need for immediate testing and interpretation in an emergency department were important criteria in two studies.

The screening programmes also valued high test performance but add that a test should have been proven to be valid and reliable. Furthermore, the screening committees set other criteria to ensure the effectiveness of public health programmes: the prevalence of the target disease or condition as well as whether or not there is effective disease management and treatment available. In a screening setting, where patients are asymptomatic, acceptability was mentioned as crucial by three of the screening programmes and the need to evaluate benefits against harms was also considered to be an important criterion.

Conclusion

From the very limited evidence available in a clinical setting, it is difficult to gain much insight into how searchers might make choices about search filters. Diagnostic test performance (perceived or known) was the most frequent factor mentioned and is the main factor that is readily applicable to search filter choice. However, it may be beneficial to provide additional explanatory information when reporting search filter performance to ensure that searchers make choices based on an accurate understanding of test performance parameters.

Databases used by interviewees

The interviewees use or have used a range of databases, many of which are health related (Table 22). Other databases mentioned were project specific and included databases that focused on social care, transport, criminology and humanitarian aid.

Interviewees’ use of search filters

Circumstances under which NICE searchers did not tend to use search filters included the following:

For short clinical guidelines, the team only use search filters on the rare occasions when the PICO [population, intervention, comparison and outcome] is restricted to study design.

Filters do not work very well when searching for diagnostic studies.

There is only a small volume of literature relating to new procedures/interventions, so filters are not necessary.

Searches carried out at the point in time when products get a CE [Conformité Européene] mark (or before), tend to be internet-based as any publications are very new and may not yet be included in databases.

The ERGs’ use of search filters for NICE work was limited because:

Single Technology Appraisals involved a review of the work of organisations submitting to NICE. The ERG staff only developed searches to test the searches carried out by the submitting body.

Multiple Technology Appraisals (MTAs) are very PICO [population, intervention, comparison and outcome]-driven.

However, the occasions when filters are used by an ERG included:

When carrying out searches for systematic reviews that include RCTs.

To help focus the question further than PICO [population, intervention, comparison and outcome] permits, to make the project manageable in terms of record numbers retrieved.

With projects looking at a single study type which are usually small projects with limited resources.

To build searches to answer guideline questions, except on the occasions where search results were small in number.

To identify economic evidence.

To carry out limited focused searches.

The filters that interviewees said that they used were:

Cochrane RCT filters and RCT filters [unspecified].

Diagnostic test accuracy filters.

Qualitative filters [drafted by the interviewee].

Filters produced by HIRU [Health Information Research Unit]/the McMaster Hedges Team.

Where would you look for a search filter?

Interviewees provided several responses to this question:

CRD website/blog/ISSG search filters page/InterTASC website [note that the last two sites mentioned here are the same].

Would post a question to discussion lists.

Look in the Cochrane Handbook.

Speak to colleagues.

Consult an in-house methodology database.

Consult an in-house search manual.

Look at methods used in previous project.

Developing and amending search filters

Some interviewees were comfortable with translating filters for use in different databases and some were not. Interviewees were comfortable translating MEDLINE search filters for use in other databases but were not comfortable translating non-MEDLINE filters for use in other databases. In the absence of objectively derived filters, however, interviewees said that they would have translated non-MEDLINE filters to run them in other databases.

Some interviewees said that to identify qualitative research they would tend to write/amend their own filters. Some respondents said that they would amend filters for scoping searches. A number of respondents noted that filters need to be written (or adapted) on a review-by-review basis depending on what was needed from the search. Several respondents indicated that they would adapt filters occasionally, for example if a filter was too sensitive they would take out a few lines to make it more specific.

Reporting the use of search filters

A number of approaches were reported around the documentation of the use of search filters:

Citing the search filters used and reporting if amendments had been made to an existing filter or if the strategy had been based on an existing filter.

Writing search strategies up fully without explicitly citing the filters used.

Including search filters as part of published strategies but not explicitly identifying them.

Documenting the use of all agreed filters, amendments and the rationale behind the amendments.

Keeping a record of search strategies but not describing them when the strategy is not written up for publication.

Using a process document which included a section about which filters have been used; document the filter when there is a need to justify the use of a filter.

Methods of keeping up to date

Interviewees’ attitudes towards keeping up to date ranged from ‘Difficult – something that is always on the “to do” list’ to ‘As we are such a small community I feel that it is unlikely that important information about a new filter will be missed’. Interviewees reported using the following methods to keep up to date:

A NICE internal current awareness bulletin.

E-mail lists and specialist groups (e.g. Cochrane IRMG [Information Retrieval Methods Group], HTAi IRG [Information Resources Group], National Library of Medicine list for MeSH changes and updates).

Meetings {e.g. ISSG, groups in the wider NICE family (e.g. NCC [National Collaborating Centre] Information Specialist Network meetings)}.

Websites (e.g. ISSG and McMaster Hedges team).

Conferences (e.g. Cochrane, HTAi).

Journal publications.

Setting up a citation search.

Choosing between filters

Interviewees reported a wide variety of actions that they might take when choosing between two or more filters, including:

In cases where two search filters appear similar (in terms of sensitivity and specificity) I tend to take the good parts from each and test to see that the results still include benchmark papers and then make sure client is happy with the approach.

Sensitivity and specificity figures can give a guide but they are still reporting the results from that instance. They may have been combined with a specific topic or used in a specific context and will still have to be investigated for appropriateness.

I would test for sensitivity and specificity. The final choice, however, is still arbitrary, relying on gut-feeling and the requirements of the specific project.

Test against a set of target references.

If I had sufficient time would try both and test results against each other; if there was a lack of time I would use the most current.

I assess the methods used to develop the filter and the extent to which it matched needs of the search (sensitivity, precision, a mixture of the two).

I would back up my decision with academic literature.

I run both filters and compare the results to see where there are gaps/duplications in retrieval between the two and to see which retrieves the more relevant papers.

I try both – I use my gut feeling rather than anything formal.

Provenance – I judge according to who developed the filter.

Look on ISSG website to see if anyone has completed an appraisal.

Search testing is pragmatic/intuitive rather than being a formal scientific process (there is not enough time to do this).

I would like someone to be quite directive about which are the best filters to use in different situations and be able to quickly see how these filters have been evaluated and how decisions have been reached (e.g. as in the Cochrane Handbook).

As a junior information specialist, the decision on which filter to use is made by senior colleagues.

It would be easier to choose if the Collaborating Centres and NICE were using the same filters and then informed everyone when changes/updates have occurred.

The YHEC [York Health Economics Consortium] ‘Getting the best out of search filters’ training course has been useful information to help critically appraise filters.

What would help you choose between filters?

Interviewees provided a range of responses when asked what would help them choose between filters:

The interpretation of the filter in simple terms – such as power calculations, statistical methods etc. that are difficult to understand, particularly for those with limited time – a synopsis would be a great help.

It is difficult to fully understand all of the complicated technical methods used to devise and test search filters. There is an element of trusting the researchers involved – I can critically appraise to a certain level but not entirely.

A summary documenting sensitivity/specificity would help to choose between filters, although this might be subjective (e.g. a document would be good for one search but not for another).

Sensitivity and specificity are important.

Some measure of rating would be useful but I would need to have confidence in whoever had carried out the rating.

Benefits of filters

Interviewees said that the main benefits of filters were that they could target the results of searches and reduce the volume of literature retrieved. It was also mentioned that the use of established filters (e.g. the Cochrane RCT filters), which have been evaluated and tested, reflected well on search quality. Additionally, using filters means that the searchers can benefit from someone else’s expertise and time spent developing the filters.

Limitations of filters

Interviewees expressed a range of concerns about search filters:

There is always a chance something has been missed.

Filters still identify a lot of irrelevant records.

Poor indexing doesn’t help searching.

Few, if any, are used appropriately.

If there is a mistake, it will be replicated through all searches/databases.

Transferability can be a problem, e.g. the Fleming qualitative filter was originally devised for use in nursing topics and probably works fine there but it was not appropriate for a diagnostic type study.

Not many filters are reliable, there are only a few databases that you can use them in and databases keep changing so it is important to check that the filter is up-to-date.

A filter gets published, people talk about it and it gets known and it starts getting used. But negative results/experiences tend not to be talked about or published and therefore there can be bias.

Areas where filters are needed/existing filters need to be improved

Interviewees were asked if there were any topics for which filters are needed or any filters that could be improved. The responses included the following:

Population age.

HRQoL (including topic-specific instruments).

Tested filters for observational studies.

Epidemiology.

Diagnostics.

Adverse events and safety issues.

Prognostic filters.

Qualitative research filters.

An improvement to the diagnostic accuracy filter.

Other comments

Interviewees were asked for other final comments and responded with both general and specific points:

The methods behind derivation of filters are impenetrable, so there is a certain amount of trust involved in using them. But this is an improvement on pragmatically deriving a study design filter from scratch.

Databases need some/better coding for SRs [systematic reviews] and DTAs [DTA studies].

PRISMA [Preferred Reporting Items for Systematic Reviews and Meta-Analyses] guidelines about reporting search strategies need to be reviewed.

There is a need for academics to recognise the importance of searching in its own right – this might be helped if information specialists were to routinely write-up protocols and include academic arguments for the approach they took.

Perhaps developers of similar filters could work collaboratively/liaise with one another to see if there is really a need for two or more filters which (appear to) carry out the same role.

There are issues with different database interfaces. For example, a filter devised for use in Ovid is likely to work very differently if translated into another interface, such as EBSCO (or Web of Science or Dialog DataStar, etc.).

More education on what filters can and can’t do is important as there are still examples of filters being used incorrectly, for example, an economic filter being used in NHS EED, an RCT filter used in CENTRAL.

Filters are needed for more databases, rather than more filters for MEDLINE (and EMBASE).

There are problems with EMBASE and the number of Emtree terms attached to the records leading to the retrieval of more irrelevant records.

It would be useful if the ISSG Search Filter Resource website indicated when something new had been added.

Patient experience/issues filter (SIGN) [Scottish Intercollegiate Guidelines Network] needs to be disaggregated – it is over 200 lines long.

What is your job title?

Forty-three different job titles were provided. Seventy of the 88 respondents who answered this question (79.5%) reported a job title that included the word ‘library’, ‘librarian’ or ‘information’. The remaining respondents reported one of the following job titles (two respondents did not answer this question):

• Assistant Professor

• Associate Scientist

• Consultant Physician and PhD candidate

• Director, systematic review research unit

• e-resources Co-ordinator

• Health Communication Specialist

• Learning Resources Officer

• Medical Documentalist

• Research Assistant

• Research Fellow

• Senior Lecturer

• TSC.

Over 75% of the respondents worked directly in information or library services, with the remaining respondents holding positions in which research and information finding would seem to be a key aspect of the role and knowledge of search filters could be assumed.

How long have you been searching databases such as MEDLINE?

The questionnaire was completed by experienced searchers, all with a minimum of 1 year’s experience of database searching and with nearly half (48.9%; 44/90) having > 10 years of database searching experience (Table 23).

How often do you develop new search strategies as part of your work?

Three-quarters of questionnaire respondents (75.6%; 68/90) reported that they developed searches at least once a week and half of these said that they developed searches daily (Table 24).

For what purposes do you carry out searches within your organisation?

The questionnaire sought information on what types of searches were carried out. Respondents were presented with the following three options and were asked to tick all that applied:

• rapid searches to answer brief questions (78.9%; 71/90)

• scoping searches to estimate the size of the literature on a topic (81.1%; 73/90)

• extensive searches to inform evidence synthesis such as guidelines, systematic reviews and technology assessments (94.4%; 85/90).

The most common searches that were carried out by respondents to the survey appear to be extensive searches to inform reviews and guidelines, but almost 80% of respondents reported that they also carried out rapid searches to answer brief questions and/or scoping searches.

Respondents also reported that they carried out searches for purposes other than those mentioned above. These were focused around teaching/education or were carried out in response to direct questions (Table 25).

Which databases do you search regularly?

Respondents were presented with a list of six databases (Table 26) (which are often cited for searches in HTA and systematic reviews) and were asked which they searched regularly. They were also asked to indicate any other databases that they use on a regular basis.

All respondents reported that they use MEDLINE and most (93.3%; 84/90) used The Cochrane Library databases. Over 75% of respondents indicated that they used EMBASE (77.8%; 70/90), nearly 75% used CINAHL (74.4%; 67/90) and > 60% (62.2%; 56/90) used PsycINFO. In total, 10% of respondents used HEED (10.0%; 9/90).

Other databases that were used by four or more respondents are documented in Table 27.

Have you ever used a methodological search filter?

Over 90% of respondents indicated that they had used methodological search filters (94.4%; 85/90); five respondents reported that they had not used a methodological search filter (5.6%; 5/90).

In what circumstances would you use methodological search filters?

Respondents were provided with five options to capture the circumstances in which they would use a methodological filter and were asked to tick all that they felt applied to their own situation (Table 28). Over 75% of respondents (76.7%; 69/90) indicated that they would use search filters for extensive searches carried out to find studies to inform guidelines or systematic reviews. Over 60% (61.1%; 55/90) indicated that they would use filters for rapid searches to answer brief questions and a similar number (58.9%; 53/90) said that they would use filters for scoping searches to estimate the size of the literature on a topic.

Respondents provided many other reasons for using search filters. Some related to developing the search strategy, some to the type of research that the search was informing and some to specific objectives:

• to practise search techniques

• to begin to identify MeSH and text words to use in developing a strategy

• if the customised limits provided by the databases cannot be relied on

• to reduce the results to a manageable size/narrow down results

• to locate research conforming to appropriate methodology to inform systematic reviews or other research/clinical practice

• to meet client need/interest

• in analysis of a situation/bibliographic analysis/identifying trends

• in health research and policy, especially questions related to economics/cost-effectiveness

• to monitor trends

• for drug trials

• to keep updated regarding competitors’ clinical trials.

Do you always use a filter when providing searches for similar types of projects?

Just over one-third of respondents (37.8%; 34/90) indicated that they would always use a filter when providing searches for similar types of projects. Just over half (56.7%; 51/90), however, would not and five respondents (5.6%; 5/90) did not respond to the question.

Four respondents indicated that they use filters only as a starting point when developing strategies and two respondents said that they rarely used filters, with one explaining that a filter would be used only when the topic had been well covered and a quick search was required. The circumstances in which respondents would not use a filter can be summarised as follows:

• when the volume of literature is manageable (21 respondents)

• client preference/specification (eight respondents)

• when looking for multiple study designs (six respondents)

• when looking for DTA studies (one respondent)

• on questions that encompass social issues (e.g. the social determinants of health, as much of the research is qualitative) (one respondent)

• when searching the literature for information neither directly for nor oriented towards clinical practice (e.g. physiology research) (one respondent)

• if it is important to be sure of finding all relevant references (one respondent)

• depending on the topic – it is not always appropriate and when undertaking scoping searches it is not always useful to narrow down these searches at an early stage (one respondent)

• when not sure that the filter is sufficiently sensitive (one respondent).

Typical practice when using search filters

Respondents were presented with different options describing how they might typically use search filters. The majority of respondents (81.1%; 73/90) indicated that they used different filters depending on whether their search needed to be sensitive or precise. However, 11% (10/90) of respondents reported using the same filter irrespective of the search focus (Table 29).

If you had to find a methodological search filter for a specific study design, where would you look?

Respondents reported a range of resources that they would use to identify search filters for specific study designs. Some respondents reported using more than one resource. When the respondents used varying designations for the same resource, these have been grouped together, for example responses such as ‘Haynes’, ‘Hedges team’, ‘HIRU’ and ‘McMaster’ have been grouped together as denoting the output of the McMaster Hedges team. Although respondents searched a range of resources, the most frequently searched resource for filters for a specific topic seemed to be the Cochrane filters to identify RCTs in MEDLINE133 (36.7%; 33/90).

Across a range of topics, the most widely reported filters were those produced by the McMaster Hedges team, which are included in many interfaces to MEDLINE, as well as the filters reported on the ISSG Search Filters Resource. Search filters for RCTs and systematic reviews were more frequently reported than filters for other study designs.

In terms of search filters to find guidelines, respondents reported:

• using no filters (five respondents)

• developing their own filters (four respondents)

• using PubMed clinical queries systematic reviews or clinical queries filters (four respondents)

• using Health Evidence Bulletins Wales filters (two respondents)

• using McMaster Hedges filters (two respondents)

• searching using ‘practice guideline.pt.’ in MEDLINE (two respondents)

• using Scottish Intercollegiate Guidelines Network (SIGN) filters (two respondents)

• using various guideline producers’ or guidelines.gov filters (two respondents)

• using Guidelines International Network filters (one respondent)

• using ISSG Search Filters Resource filters (one respondent)

• not needing filters to search for guidelines (one respondent).

In terms of search filters to find economic evaluations, four respondents indicated that they do not use filters and nine indicated that they have developed their own or adapted published filters. Other economics filters used by respondents were:

• SIGN filters (five respondents)

• CRD (NHS EED) filter (four respondents)

• MEDLINE/PubMed built-in queries (four respondents)

• McMaster Hedges filters (three respondents)

• specific databases rather than filters (two respondents)

• CADTH filters (one respondent)

• Comparative Effectiveness Research filters (one respondent)

• Guidelines International Network filters (one respondent)

• Health Services Research Queries (one respondent)

• ISSG Search Filters Resource (one respondent)

• McKinlay filters (one respondent)

• William Witteman’s filter (from the Toronto HTA) (one respondent).

In response to a question about the types of filters (other than those for RCTs, systematic reviews, DTA studies, prognosis and aetiology) that they might use, five respondents indicated that they did not use a filter when looking for other types of studies and four respondents reported that they devised their own filters. Two respondents would search for filters on the ISSG Search Filters Resource. Other respondents suggested specific filters for a range of topics.

How do you decide which filter to use?

Respondents replied to this question by selecting one or more options from a list of options (Table 30). Respondents reported that they generally used the available filters that best suited their purpose (56.7%; 51/90) or the filters that were already available in the database being searched (53.3%; 48/90).

Respondents also noted other approaches that they used to help them decide which filters to use, namely:

• trial and error/comparing results

• reverse engineering

• based on sensitivity and precision

• depends on the study design.

Apart from adding a subject search, do you amend methodological search filters?

The questionnaire sought to find out whether or not searchers amend filters. Four respondents (4.4%; 4/90) indicated that they always amend search filters. Over half of the respondents said that they sometimes amended filters (55.6%; 50/90) and one-third indicated that they do not make changes to filters (33.3%; 30/90) (Table 31).

Why, typically, do you amend search filters?

Twenty-six (28.9%) out of 90 respondents indicated that they amended filters to improve sensitivity and/or specificity, for example:

We are afraid to miss things so we amend filters to enhance sensitivity.

Sometimes to make them a little shorter or to increase/decrease sensitivity.

Where there are inappropriate results returned I may be able to improve specificity.

Either to broaden or narrow the scope of a search.

How do you amend search filters?

Twenty-eight (31.1%) out of 90 respondents indicated that they amended search filters by adding or removing terms. Other forms or methods of amendment reported by respondents were:

• adapting to another database

• looking at adjacency or truncation

• researching MeSH terms and adding free text

• examining which lines of syntax are producing zero or too many results

• by inclusion of keywords and weighting word algorithms

• based on advice from other librarians.

Do you test and document the effects of any amendments you make?

All who responded to this question indicated that they always or sometimes amend search filters and, of these, a majority (83.3%; 45/54) also indicated that they tested the effects of the amendment (Table 32).

Respondents reported that they test the effects of any amendments by:

• ‘eyeballing’ results

• conducting a ‘before and after’ comparison

• assessing whether or not key relevant articles have been identified.

About three-quarters of the respondents (75.9%; 41/54) who make changes to search filters document the changes that they make (Table 33).

A wide range of approaches was reported for documenting amendments to search filters, with about three-quarters of respondents indicating that they comprehensively documented changes. Some examples of the broad nature of responses to this question are shown in the following quotations:

Usually reproduce entire search string and provide written summary of rationale for changes and effects.

I keep spreadsheets of search terms where each column is an iteration of my search, with notes on why changes occur so that I have a record and a rationale.

I make a note of where I adapted the strategy from and then save the search strategy in a word document and also in the database where possible.

Narrative included in both the methods section of the review and in the annexe.

I record that the filter has been adapted for use in other databases.

I add some comments to the search line.

Save the searches for future reference but how they are written up depends on client requirements.

Only to the degree that I may save the search in my saved search file . . . and save the search to our search recording software.

Yes, but not always!

Keeping up to date

Respondents were asked to select from a list the method(s) that they use to keep up to date with search filters (Table 34). The most frequently reported method of keeping up to date was through professional development meetings and training events (74.4%; 67/90).

Over 60% of respondents reported keeping up to date by the following methods:

• reading journal articles

• reading e-mail lists

• through information provided by managers/work colleagues.

Respondents were asked to indicate which specific current awareness resources they used to keep up to date. Some respondents indicated that they used more than one resource. Current awareness searches set up in databases such as MEDLINE and EMBASE were the most frequently cited approaches, followed by tables of contents services. The resources cited were:

• database alerts/current awareness searches (nine respondents)

• tables of contents services (six respondents)

• e-mail discussion lists (two respondents)

• really simple syndication (RSS) feeds (two respondents)

• American College of Physicians journals (otherwise unspecified) (one respondent)

• AETMIS (Agence d’évaluation des technologies et des modes d’intervention en santé) current awareness service (one respondent)

• AHRQ current awareness service (one respondent)

• Cochrane (otherwise unspecified) (one respondent)

• discussion with colleagues (one respondent)

• end-of-life care (otherwise unspecified) (one respondent)

• library blogs [including Krafty Librarian, iLibrarian, Phil Bradley, OCLC, ScienceRoll, MedScape] (one respondent)

• NICE internal current awareness bulletin (one respondent)

• Palliative Care Journal Club (one respondent)

• WebSite-Watcher e-mail alerts (one respondent).

Respondents were asked to indicate which websites they used to keep up to date. Some respondents provided more than one resource. The ISSG Search Filters Resource was the most frequently cited website (25.6%; 23/90). The SIGN, McMaster Hedges team, MEDLINE and Cochrane resources were also mentioned by between four and nine respondents. As previously, some resources were described using various names and some assumptions have been made about groupings. The websites cited were:

• InterTASC/ISSG/CRD (23 respondents)

• SIGN (nine respondents)

• McMaster Hedges team (seven respondents)

• MEDLINE/PubMed/US NLM (National Library of Medicine) (otherwise unspecified) (five respondents)

• Cochrane (Collaboration/IRMG/Handbook/Library) (four respondents)

• BMJ Clinical Evidence (three respondents)

• Cindy Smith’s blogspot (three respondents)

• University of British Columbia Library Health Library Wiki (two respondents)

• BestBETs (one respondent)

• Centre for Evidence-Based Medicine (one respondent)

• Google (one respondent)

• government and non-governmental health organisations (one respondent)

• HTAi Vortal (one respondent)

• Knowledge Network – shared space for Scottish librarians (one respondent)

• national and university websites (otherwise unspecified) (one respondent)

• US NLM e-text on HTA (one respondent)

• Ovid databases (otherwise unspecified) (one respondent)

• World Health Organization (one respondent).

Respondents were asked to indicate which e-mail lists they used to keep up to date. The most frequently reported lists were Cochrane e-mail discussion lists (19 respondents) and expertsearching (nine respondents). National medical librarian discussion lists for the USA, Canada and the UK were frequently mentioned. The e-mail lists cited were:

• Cochrane information specialist e-mail discussion lists (IRMG/librarians/methods/TSCs) (19 respondents)

• expertsearching (nine respondents)

• CANMEDLIB (eight respondents)

• LIS-MEDICAL (eight respondents)

• MEDLIB-L (eight respondents)

• HTAi Information Resources Group (seven respondents)

• Evidence-Based-Health (five respondents)

• InterTASC ISSG (five respondents)

• CLIN-LIB (two respondents)

• local health libraries network (unspecified) (two respondents)

• EAHIL-L (one respondent)

• Evidence-Based-Libraries (one respondent)

• Health Sciences Libraries Group discussion group for the Health Sciences Libraries Group of Ireland (one respondent)

• LIB-HELIX discussion group for library staff in the NHS South Central area of the UK (one respondent)

• LIS-NURSING (one respondent)

• medical librarian lists (unspecified) (one respondent)

• NCC-information specialists (one respondent)

• professional e-mail lists (unspecified) (one respondent)

• SYS-REVIEW (one respondent)

• WEBENZ e-mail list for medical information specialists in the Netherlands (one respondent).

Respondents were asked to indicate which RSS feeds they subscribed to to help to keep up to date. RSS feed usage was low, with the most frequently reported feed being Cindy Smith’s (most likely Schmidt’s) blogspot (http://pubmedsearches.blogspot.ca/) and Evidence Based Library and Information Practice (EBLIP). 134 The RSS feeds cited were:

• Cindy Smith’s (most likely Schmidt’s) blogspot (three respondents)

• EBLIP (three respondents)

• PubMed New and Noteworthy/PubMed Technical Bulletin (two respondents)

• Health Information and Libraries Journal (one respondent)

• journal tables of contents (unspecified) (one respondent)

• librarian blogs (unspecified) (one respondent)

• LISNews (one respondent)

• medical libraries (unspecified) (one respondent)

• OvidSP Updates (one respondent)

• websites (unspecified) (one respondent).

Respondents were also asked to provide other methods (not listed above) that they used to help keep up to date. The methods reported included:

Check my file of papers on search filters.

Search for filters when one is required (PubMed or Google or post a query to MEDLIB-L or CANMEDLIB).

If there were changes in the Cochrane Handbook I would incorporate these into my work.

Meetings of the Cochrane Information Retrieval [Methods] Group.

My colleagues and I have a journal club where articles such as these are often chosen.

Other colleagues within my unit.

Attend workshops.

If you have had to choose between methodological search filters, what features or information has helped you to do so?

Respondents were asked how, when faced with a choice of methodological filters, they chose a filter. Several respondents (16.7%, 15/90) said that they required information on the performance of a filter (sensitivity, specificity and precision), with other respondents (11.1%; 10/90) requiring published reports and evaluations of the filter. Five respondents (5.6%, 5/90) required information on authorship and five looked to colleagues for advice.

Other approaches reported by respondents included:

Personal knowledge and testing.

Length – the shorter the better.

Relevant database.

Focus.

Flexibility/modifiability.

Testing.

InterTASC site/ISSG.

Choose the ones that look logical based on my experience.

Search words used.

If you report your search process do you describe the filters that you have used?

Most respondents (86.7% 78/90) reported that they described the search filters that they used, with 4.4% (4/90) of respondents reporting that they did not (Table 35).

If you report your search process do you justify your choice of filters used?

Just over half of respondents (57.8%; 52/90) reported that they did not justify their choice of filter when writing up their search, whereas approximately one-third (32.2%, 29/90) of respondents reported that they do provide a justification (Table 36).

What do you think are the benefits of using methodological search filters?

The most frequently reported benefits of using methodological search filters were that they helped to focus results (42.2%; 38/90), they are tried and tested (18.9%; 17/90), they save time (10%; 9/90) and they offer transparency and consistency (5.6%; 5/90).

Respondents also reported other benefits, including:

help estimate workload in project planning.

[to enable] conceptual mapping of thoughts.

rerunning is easy, results are comparable.

What do you think are the limitations of using methodological search filters?

Respondents reported that the most frequent concerns they had about using a methodological search filter were that studies would be missed (37.8%; 34/90), filters were not always fit for purpose (22.2%; 20/90), filters lacked transparency or were hard to appraise (10%; 9/90) and filters were reliant both on the competence of the filter developer and on the adequacy of record indexing (14.4%; 13/90).

Other limitations included:

• can sometimes be hard to choose between filters (one respondent)

• lack of instructions for publishing (one respondent)

• sometimes hard to explain filters to researchers (one respondent)

• the ‘perfect filter’ is not always available and so ‘the next best thing’ is used, which is not ideal (one respondent)

• too many results (one respondent).

What information would help you to choose which filter to use?

Respondents reported that they would like information on filter performance measures such as validation (27.8%; 25/90) and sensitivity and specificity (20%; 18/90), and a description of the filter (16.7%; 15/90).

Other information requirements included:

• results of own testing (11 respondents)

• colleague recommendations/discussion (six respondents)

• the database (four respondents)

• knowledge of the creator/developer (three respondents)

• simplicity/understandability (three respondents)

• ease of use (including automatic loading) (two respondents).

Respondents reported that the main factors that would make choosing a filter easier were the availability of a critical appraisal or evaluation (17.8%; 16/90) and more information (such as the effectiveness of the filter, what it does/provides, what it excludes, its limitations, when it was last updated; advantages and disadvantages; sensitivity and specificity; how it has been tested) (16.7%; 15/90). Respondents also reported that they wanted to be confident in the author/developer (11.1%; 10/90).

Other factors cited as making it easier to choose which filter to use were:

• the presence of a central storage location (seven respondents)

• better expression/presentation of results (four respondents)

• greater consistency in the methods used (one respondent)

• availability/accessibility in all databases (problem with CINAHL on EBSCOhost) (one respondent)

• better labelling/indexing of articles so that they might be more easily retrieved (one respondent)

• more up-to-date coverage on the CRD (i.e. ISSG Search Filters Resource) website (one respondent)

• more ‘professional noise’ about a new filter (one respondent)

• the availability of synopses of filters (one respondent).

What methodological search filters would be useful to you?

The respondents had a wide range of requirements for new filters:

• economic/economic evaluation/cost–benefit/cost–utility studies (five respondents)

• all research/study designs (in one filter) (two respondents)

• controlled trials/controlled studies in the public health field (two respondents)

• diagnosis/diagnostic studies (two respondents)

• a combination of RCTs and systematic reviews/meta-analysis in one filter

• aetiological studies

• burden of illness studies

• case–control studies

• case series

• clinical audits

• clinical trials

• cross-sectional studies

• epidemiological studies

• full-text searches

• guidelines

• HTAs

• interrupted time series

• meta-analyses

• non-RCTs

• observational studies

• process evaluations

• qualitative studies

• quasi-experimental studies

• RCTs

• social sciences methodologies (other)

• specific methodologies

• systematic reviews.

Respondents also had requirements for filters capturing other issues:

• adverse effects/events/harms

• age groups

• children/paediatrics

• demography

• disease specific/technology specific

• emergency departments

• errata

• hospital management (non-clinical)

• hospital setting

• magnetic resonance imaging

• older people

• patient-centred outcomes

• patient experience

• prognosis/prognostic studies

• programmes and services

• public health, especially health protection and infection control

• retracted or withdrawn articles

• therapy.

With respect to databases, respondents expressed an interest in the following database-specific filters:

• a more precise RCT filter for EMBASE

• a validated/Cochrane-recommended RCT filter for EMBASE and other databases, for example CINAHL

• a definitive filter per methodology per database

• Education Resources Information Center (ERIC) filters

• filters validated for more databases than EMBASE and MEDLINE.

Other comments about filters were invited and the following were noted:

Clearer ones.

Please, no further filters – more work on limitations of filters and dissemination work on alternative/better methods is necessary.

UK studies.

Further observations on methodological search filters as a tool for information retrieval

Respondents provided additional views on methodological search filters as a tool for information retrieval, which have been grouped in the following sections under limitations and benefits.

When do searchers and researchers use search filters?

The awareness and use of search filters seems to have developed considerably in the decade since the publication of the article by Jenkins and Johnson. 5 Most respondents seem to know where to look for filters from well-established producers and collections. The responses, however, demonstrate a wide variation in the confidence with which questionnaire respondents choose filters. There are also contradictions between the difficulties that respondents express in terms of selecting between filters (acknowledging the possible complexities of filter design) and the commonplace practice of searchers adapting published strategies to fit their own requirements (ignoring the fact that many filters are designed to perform in a quite specific way). Several respondents have developed their own filters for local use. The responses indicate that search filters are used more frequently for large-scale reviews and slightly less often for simpler scoping and rapid searches. This may reflect different practices in scoping and rapid searches because fewer resources will be searched and less sensitive subject searches will be employed because of the limited timescale. Adding a filter to an already focused search might be seen as risking missing studies. For all types of searches, search filters offer an opportunity to focus the numbers of records retrieved, which can be helpful when time is limited. Search filters are predominantly viewed by respondents as a tool to maximise sensitivity rather than precision (although this is not the intended objective of all filters), but seem to be used to achieve optimal sensitivity and precision.

What information would help researchers choose between filters?

The responses to the questionnaire have many messages for search filter designers. Filter performance measures need to be signposted more clearly and succinctly to help searchers make better use of the available filters. Filter and website designers should present less information (to avoid information overload) and ensure that performance information can be clearly seen. Respondents also reported that they wanted to be confident in the author/developer. While the provenance of filters is clearly important to some searchers, there are no established parameters to measure this confidence. Clear authorship labelling and the provision of detailed methods to show the robustness of the development methods would not only assist users of filters but also help filter designers achieve recognition for their filters. The convenience of having filters by well-established producers available within database interfaces (such as the PubMed Clinical Queries filters) encourages their use. The most convenient search filters, however, may not always be the best for particular tasks and searchers and researchers need to know how to choose when a range of sensitive, precise or ‘optimal’ strategies is offered. Respondents require more information on the validation of search filters. They value and use resources such as the ISSG Search Filters Resource and the filters of the McMaster Hedges team. The former provides a list of all identified methodological search filters in one place, by study design and by database, which has a convenience factor. The latter provides search filters developed using documented methods within database interfaces, with filters ‘badged’ with the authority of both the research team and the US NLM. In contrast to the methodological and publisher quality seals of the McMaster filters, the BMJ Clinical Evidence and SIGN websites provide little information on filter production and/or validation. The filters on these websites, however, seem to be widely used, suggesting that authorship is the seal of quality.

Respondents did not necessarily feel that all of their requirements were currently being met. They would like translations of filters for different databases and interfaces, more strategies independent of indexing language (to facilitate transferral across databases) and filters for a wider range of study designs and other topics. This provides a research agenda for any search filter authors willing to take up the challenge.

Respondents keep informed about developments in search filters through a wide variety of methods and resources, which suggests that search filter and website designers face a marketing challenge. Highlighting new filters to key audiences such as information specialists and systematic reviewers by inclusion in resources such as the Cochrane Handbook133 and the ISSG Search Filters Resource6 would help to promote new filters beyond the simple publication of a journal article. In addition, a large number of e-mail lists are used for current awareness purposes, and the promotion of new filters through these lists would seem to be an efficient way to reach potential users.

Although the use of search filters seems to be quite widely documented and amendments are noted in search reports, there seems to be scope for promoting clarity around the use and amendment of search filters. This, again, is an issue for filter authors and website producers. There is clearly a large amount of ad hoc filter amendment work being undertaken: searchers take filters and adapt them for their own purposes. This would seem to indicate a lack of awareness that the filters may be designed for a purpose or have been arrived at after extensive exploration (increasingly using textual analysis techniques) to justify the use of specific terms and the absence of others. The performance assessment of amended search filters does not seem to be a priority for many searchers. Filter developers should consider how they want their filters to be used and perhaps attach guidance or caveats to the filters. Guidance for filter adaptation may also be merited so that filter developers are credited for the original work but absolved from the effects of the adaptations. Many filter developers retain their gold or reference standards and might be willing to test adaptations.

The original impetus for many search filters was to maximise sensitivity but, increasingly, possibly because of limited resources, searchers seem to be demanding improvements in precision. Future filter developments (for interfaces that use Boolean searching) need to continue to improve precision while maintaining sensitivity. The advent of full-text searching and semantic analysis of both full-text and bibliographic records may see filters used in different ways in the future. For example, sensitive filters might be used to identify records from databases and these results might then be processed using semantic analysis software trained to identify records of specific types. The results could then be used to revise the search filter and improve the precision of the search results. This approach will have search algorithms (filters) that are more like semantic rules than the dichotomous (relevant/not relevant) search filters that we see used in bibliographic databases such as MEDLINE. Textual analysis approaches have been used in the design of searches. 22,135 The extent to which textual analysis alone can be relied on in the future to distinguish relevant records from irrelevant records is under investigation. 59,135 When using semantic analysis approaches the onus will be on the searcher to select the performance levels, that is, to choose an acceptable probability of a record being relevant.

Conclusion

Search filters are used mainly for reducing the size of large result sets (introducing focus) and assisting with searches that are focused on a single study type. Searchers use several key resources to identify search filters but may find choosing between filters problematic. Features that would help with filter choice include making information about filters less technical, offering ratings and providing more detail about filter validation strategies and filter provenance.

Which performance characteristics should be measured?

When considering the measures that are most useful to users of search filters, the following are recommended based on the responses to the interviews and the questionnaire survey carried out to inform this research. These measures are also those most frequently reported in the literature (reviews A and B):

• sensitivity

• precision or NNR.

Sensitivity is defined as the number of records in the reference set that are retrieved by a search filter as a proportion of the total number of records in the reference set. It is therefore a crucial performance issue for many searchers, especially within the context of many systematic reviews and HTAs, in which searchers are usually focused on retrieving as much relevant evidence as possible. This may be less of a concern in reviews of qualitative evidence. 136

Precision is defined as the number of reference set records retrieved by a search filter as a proportion of the total number of records (relevant and irrelevant) retrieved. It is also a crucial issue for searchers involved in evidence synthesis because, in seeking to achieve high sensitivity, retrieval rates are often high and the precision tends to be low. One study reports that 2–3% precision is typical of searches undertaken in systematic reviews137 but experience suggests that precision is often much lower than that. Precision is also a concept that is of relative importance to searchers as they are likely to be more tolerant of low precision when low numbers of records are retrieved (e.g. in a search topic when there are few research reports) than when high numbers of records are retrieved (e.g. in a search of breast cancer). NNR offers a precision-based metric to indicate the workload involved in identifying relevant records when using a specific filter.

An additional performance measure could be collected that might assist with estimating workload. We have called this ‘reduction in number needed to read’ (see Glossary and also review C and Whiting et al. 2). This indicates how far adding a filter to a subject search will reduce the workload involved in processing records by showing the reduction in the number of records that will need to be screened. A small reduction in the number needed to screen may indicate that using a filter is not helpful in reducing the workload involved in assessing retrieved records for relevance, whereas a large reduction in number needed to screen may encourage the use of a filter.

How should a performance measure be ascertained?

If sensitivity and precision are the focus of performance measurement, the following issues are crucial for robust measures:

• having a definition of the criteria for building a reference set

• having a reference set of relevant records (to measure sensitivity and precision/NNR)

• having a results set containing all records retrieved by hand-searching or records retrieved by RR methods or the total number of records retrieved by a search of a database using a subject search strategy (to measure precision/NNR)

• having search filters that are suitable for the database interface being used to search for records or that have been translated carefully to be used in another database interface.

These issues are discussed in more detail in the following sections.

How can performance measurement be carried out most efficiently?

A flow diagram showing the key steps in conducting search filter performance measurement using a hand-searched reference set is shown in Figure 11. The process should be fully documented as it is undertaken and ideally the search filter should be tested on its own, without the addition of a subject filter, as the hand-searched documents provide the test bed. The question of how far the hand-searched documents are representative of all documents that might yield relevant records should be discussed.

FIGURE 11.

Search filter performance measurement using a hand-searched reference set.

A flow diagram showing the key steps in conducting search filter performance measurement using a RR reference set is shown in Figure 12. As above, the process should be fully documented as it is undertaken and the same caveats apply.

FIGURE 12.

Search filter performance measurement using a RR reference set.

Search strategy

#1 “diagnostic test accuracy”:kw in Methods Studies

#2 “diagnostic test accuracy”:kw and “search strategies”.kw in Methods Studies

#3 (#1 AND NOT #2)

Search strategy

1. *”diagnostic techniques and procedures”/ or *diagnostic imaging/ or *diagnostic tests, routine/ use mesz

2. *diagnostic accuracy/ or *diagnostic procedures/ or *Diagnostic test/ use emez

3. diagnostic.ti.

4. *roc curve/

5. *”sensitivity and specificity”/

6. or/1-5

7. *guidelines as topic/ use mesz

8. *practice guidelines/ use emez

9. *meta-analysis as topic/ use mesz

10. *meta-analysis/ use emez

11. *review literature as topic/ use mesz

12. *systematic review/ use emez

13. *Evidence-Based Medicine/mt, st use mesz

14. *evidence based medicine/ use emez

15. guideline?.ti.

16. (method$ adj1 standard$).ti.

17. methodological.ti.

18. (statistic$ adj1 method$).ti.

19. (working adj1 (party or committee or group)).ti.

20. or/7-19

21. 6 and 20

22. limit 21 to english language

23. remove duplicates from 22

24. limit 23 to yr=”1980 –Current” (993)

Search strategy

1. (methodolog$ adj3 filter$).ti,ab. (78)

2. (search adj3 filter$).ti,ab. (164)

3. (search adj strateg$).ti,ab. (9588)

4. (quality adj3 filter$).ti,ab. (278)

5. hedge$.ti,ab. (6400)

6. (clinical adj queries).ti,ab. (66)

7. ((economic or random$ or systematic or diagnostic) adj3 (filter? or search strateg$)).ti,ab. (618)

8. or/1-7 (16,583)

9. Choice Behavior/ (16,343)

10. (choice$ or choose or chose or choosing).ti,ab. (201,696)

11. select$.ti,ab. (944,947)

12. prefer$.ti,ab. (229,163)

13. (decid$ or decision$).ti,ab. (190,433)

14. judgment$.ti,ab. (21,984)

15. or/9-14 (1,468,055)

16. 8 and 15 (8714)

17. Librarians/ (600)

18. librarian$.ti,ab. (1773)

19. (information adj2 (specialist$ or officer$ or scientist$)).ti,ab. (474)

20. (searcher$ or researcher$).ti,ab. (56,147)

21. or/17-20 (58,457)

22. 16 and 21 (638)

Search strategy

1. information retrieval/ and methodology/ (6053)

2. (methodolog$ adj3 filter$).ti,ab. (93)

3. (search adj3 filter$).ti,ab. (189)

4. (search adj strateg$).ti,ab. (11,533)

5. (quality adj3 filter$).ti,ab. (370)

6. hedge$.ti,ab. (6951)

7. (clinical adj queries).ti,ab. (75)

8. ((economic or random$ or systematic or diagnostic) adj3 (filter? or search strateg$)).ti,ab. (709)

9. or/1-8 (25,162)

10. decision making/ (101,825)

11. (choice$ or choose or chose or choosing).ti,ab. (243,717)

12. select$.ti,ab. (1,095,085)

13. prefer$.ti,ab. (256,127)

14. (decid$ or decision$).ti,ab. (226,199)

15. judgment$.ti,ab. (24,055)

16. or/10-15 (1,746,266)

17. 9 and 16 (11,520)

18. librarian/ (736)

19. librarian$.ti,ab. (1650)

20. (information adj2 (specialist$ or officer$ or scientist$)).ti,ab. (553)

21. (searcher$ or researcher$).ti,ab. (65,524)

22. or/18-21 (67,855)

23. 17 and 22 (824)

Search strategy

1. (methodolog$ adj3 filter$).ti,ab. (9)

2. (search adj3 filter$).ti,ab. (29)

3. (search adj strateg$).ti,ab. (1062)

4. (quality adj3 filter$).ti,ab. (19)

5. hedge$.ti,ab. (620)

6. (clinical adj queries).ti,ab. (5)

7. ((economic or random$ or systematic or diagnostic) adj3 (filter? or search strateg$)).ti,ab. (65)

8. or/1-7 (1743)

9. choice behavior/ (11,420)

10. (choice$ or choose or chose or choosing).ti,ab. (102,121)

11. select$.ti,ab. (190,035)

12. prefer$.ti,ab. (81,835)

13. (decid$ or decision$).ti,ab. (112,977)

14. judgment$.ti,ab. (49,128)

15. or/9-14 (460,385)

16. 8 and 15 (521)

17. exp information specialists/ (174)

18. librarian$.ti,ab. (515)

19. (information adj2 (specialist$ or officer$ or scientist$)).ti,ab. (176)

20. (searcher$ or researcher$).ti,ab. (72,543)

21. or/17-20 (73,226)

22. 16 and 21 (30)

Search strategy

S22 S15 and S21 (164)

S21 S16 or S17 or S18 or S19 or S20 (118,186)

S20 TX searcher* or researcher* (14,140)

S19 TX information N2 specialist* or information N2 officer* or information N2 scientist* (5812)

S18 TX librarian* (102,368)

S17 DE “INFORMATION professionals” (3235)

S16 DE “LIBRARIANS” (18,635)

S15 S11 and S14 (468)

S14 S12 or S13 (122,568)

S13 TX select* or prefer* or decid* or decision* or judgment* (56,972)

S12 TX choice* or choose or chose or choosing (68,911)

S11 S1 or S2 or S3 or S4 or S5 or S6 or S7 or S8 or S9 or S10 (2074)

S10 TX diagnostic N3 filter? or random* N3 search* (61)

S9 TX systematic N3 filter? or random* N3 search* (62)

S8 TX random* N3 filter? or random* N3 search (50)

S7 TX economic N3 filter? or economic N3 search* (55)

S6 TX “clinical queries” (20)

S5 TX hedge* (423)

S4 TX quality N3 filter* (54)

S3 TX search N1 strateg* (1400)

S2 TX search N3 filter* (106)

S1 TX methodolog* N3 filter* (13)

Search strategy

#1 (methodolog* NEAR/3 filter*):ti,ab,kw (30)

#2 (search NEAR/3 filter*):ti,ab,kw (85)

#3 (search NEXT strateg*):ti,ab,kw (5136)

#4 (quality NEAR/3 filter*):ti,ab,kw (20)

#5 (hedge*):ti,ab,kw (32)

#6 (clinical NEXT queries):ti,ab,kw (24)

#7 (economic or random* or systematic or diagnostic) NEAR/3 (filter? or search strateg*):ti,ab,kw (236)

#8 (#1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7) (5214)

#9 MeSH descriptor Choice Behavior explode all trees (696)

#10 (choice* or choose or chose or choosing):ti,ab,kw (1727)

#11 (select*):ti,ab,kw (5287)

#12 (prefer*):ti,ab,kw (11,236)

#13 (decid* or decision*):ti,ab,kw (10,668)

#14 (judgment*):ti,ab,kw (1335)

#15 (#9 OR #10 OR #11 OR #12 OR #13 OR #14) (70,646)

#16 (#8 AND #15) (4704)

#17 MeSH descriptor Librarians explode all trees (5)

#18 (librarian*):ti,ab,kw (144)

#19 (information NEAR/2 (specialist* or officer* or scientist*)):ti,ab,kw (34)

#20 (searcher* or researcher*):ti,ab,kw (2534)

#21 (#17 OR #18 OR #19 OR #20) (2677)

#22 (#16 AND #21) (458)

Search strategy

#18 420 #13 and #17

Databases=SCI-EXPANDED Timespan=All Years

#17 71,421 #14 or #15 or #16

Databases=SCI-EXPANDED Timespan=All Years

#16 66,699 TS=(searcher* or researcher*)

Databases=SCI-EXPANDED Timespan=All Years

#15 2970 TS=(information) SAME TS=(specialist* or officer* or scientist*)

Databases=SCI-EXPANDED Timespan=All Years

#14 2057 TS=librarian*

Databases=SCI-EXPANDED Timespan=All Years

#13 8269 #8 and #12

Databases=SCI-EXPANDED Timespan=All Years

#12 > 100,000 #9 or #10 or #11

Databases=SCI-EXPANDED Timespan=All Years

#11 20,409 TS=judgment*

Databases=SCI-EXPANDED Timespan=All Years

#10 > 100,000 TS=(select* or prefer* or decid* or decision*)

Databases=SCI-EXPANDED Timespan=All Years

#9 > 100,000 TS=(choice* or choose or chose or choosing)

Databases=SCI-EXPANDED Timespan=All Years

#8 30,806 #1 or #2 or #3 or #4 or #5 or #6 or #7

Databases=SCI-EXPANDED Timespan=All Years

#7 5102 TS=(economic or random* or systematic or diagnostic) SAME TS=(filter* or search strateg*)

Databases=SCI-EXPANDED Timespan=All Years

#6 46 TS=(“clinical queries”)

Databases=SCI-EXPANDED Timespan=All Years

#5 14,192 TS=hedge*

Databases=SCI-EXPANDED Timespan=All Years

#4 3183 TS=(quality SAME filter*)

Databases=SCI-EXPANDED Timespan=All Years

#3 7524 TS=(“search strateg*”)

Databases=SCI-EXPANDED Timespan=All Years

#2 1102 TS=(search SAME filter*)

Databases=SCI-EXPANDED Timespan=All Years

#1 814 TS=(methodology* SAME filter*)

Databases=SCI-EXPANDED Timespan=All Years

Search strategy

“methodological filter” choice librarian

“methodological filter” choice specialist

“methodological filter” choice searcher

“methodological filter” choice researcher

“methodological filter” choice officer

“methodological filter” decide librarian

“methodological filter” decide specialist

“methodological filter” decide searcher

“methodological filter” decide researcher

“methodological filter” decide officer

“search filter” choice librarian

“search filter” choice specialist

“search filter” choice searcher

“search filter” choice researcher

“search filter” choice officer

“search filter” decide librarian

“search filter” decide specialist

“search filter” decide searcher

“search filter” decide researcher

“search filter” decide officer

“search strategy” choice librarian

“search strategy” choice “information specialist”

“search strategy” choice searcher

“search strategy” choice “information officer”

“search strategy” decide librarian

“search strategy” decide “information specialist”

“search strategy” decide searcher

“search strategy” decide “information officer”

View all resources

Searching the HTA Literature

MEDLINE/PubMed

Clinical Trial Registries

Evaluated Sources

Grey Literature

Information on Literature Searching

Searching on the Web

Clinical Practice Guidelines

Reference tools

Keeping Up: stuff for Librarians and Information Specialists

General search

+search +filter

+methodological +filter

+search +strategy

+search +strategies

Tools

EUnetHTA Planned and Ongoing Projects (POP) Database

EUnetHTA Database on additional evidence

EUnetHTA News Aggregator

HTA Core Model

All resources require username and password to access – EUnetHTA membership (partners and associates)

Search this group

“search filter”

“methodological filter”

“search strategy”

“search strategies”

hedges

Search the IFM Healthcare website (www.ifmh.org.uk/)

“search filter”

“methodological filter”

“search strategy”

“search strategies”

hedges

Search

“search filter”

“methodological filter”

“search strategy”

“search strategies”

Hedges

Search

“search filter”

“methodological filter”

“search strategy”

“search strategies”

Hedges

Search strategy

1. choice behavior/ use mesz

2. decision making/

3. professional practice/

4. physician's practice patterns/ use mesz

5. clinical practice/ use emez

6. ((clinician$ or physician$ or doctor$ or practitioner$) adj3 (choice$ or chos$ or choos$)).ti.

7. ((clinician$ or physician$ or doctor$ or practitioner$) adj3 (select$ or decid$ or decision$)).ti.

8. ((clinician$ or physician$ or doctor$ or practitioner$) adj3 prefer$).ti.

9. 1 or 2

10. 3 or 4 or 5

11. 9 and 10

12. 6 or 7 or 8 or 11

13. exp “diagnostic techniques and procedures”/ use mesz

14. exp diagnosis/ use emez

15. (diagnosis or diagnostic$).ti,hw.

16. (test or tests).ti,hw.

17. 13 or 14 or 15 or 16

18. 12 and 17

19. remove duplicates from 18

20. limit 19 to english

21. (abstract or comment or conference or letter).pt

22. 20 not 21

Search strategy

S1 DE “Choice Behavior” OR DE “Decision Making”

S2 TX clinician* n3 prefer* or TX physician* n3 prefer* or TX doctor* n3 prefer* or TX practitioner* n3 prefer*

S3 TX clinician* n3 decision* or TX physician* n3 decision* or TX doctor* n3 decision* or TX practitioner* n3 decision*

S4 TX clinician* n3 decid* or TX physician* n3 decid* or TX doctor* n3 decid* or TX practitioner* n3 decid*

S5 TX clinician* n3 select* or TX physician* n3 select* or TX doctor* n3 select* or TX practitioner* n3 select*

S6 TX clinician* n3 choos* or TX physician* n3 choos* or TX doctor* n3 choos* or TX practitioner* n3 choos*

S7 TX clinician* n3 chos* or TX physician* n3 chos* or TX doctor* n3 chos* or TX practitioner* n3 chos*

S8 TX clinician* n3 choice* or TX physician* n3 choice* or TX doctor* n3 choice* or TX practitioner* n3 choice*

S9 S1 or S2 or S3 or S4 or S5 or S6 or S7 or S8

S10 DE “Screening” OR DE “Health Screening”

S11 DE “Diagnosis” OR DE “Differential Diagnosis” OR DE “Medical Diagnosis”

S12 TX test* n3 order* or TX diagnos* n3 test* or TX screen* n3 test*

S13 S10 or S11 or S12

S14 S9 and S13 Limiters - English

Search strategy

S1 (MM “Decision Making”)

S2 (MM “Practice Patterns”) OR (MM “Professional Practice”)

S3 TX clinician* n3 prefer* or TX physician* n3 prefer* or TX doctor* n3 prefer* or TX practitioner* n3 prefer*

S4 TX clinician* n3 decision* or TX physician* n3 decision* or TX doctor* n3 decision* or TX practitioner* n3 decision*

S5 TX clinician* n3 decid* or TX physician* n3 decid* or TX doctor* n3 decid* or TX practitioner* n3 decid*

S6 TX clinician* n3 select* or TX physician* n3 select* or TX doctor* n3 select* or TX practitioner* n3 select*

S7 TX clinician* n3 choos* or TX physician* n3 choos* or TX doctor* n3 choos* or TX practitioner* n3 choos*

S8 TX clinician* n3 chos* or TX physician* n3 chos* or TX doctor* n3 chos* or TX practitioner* n3 chos*

S9 TX clinician* n3 choice* or TX physician* n3 choice* or TX doctor* n3 choice* or TX practitioner* n3 choice* Search modes - Boolean/Phrase

S10 S1 or S2 or S3 or S4 or S5 or S6 or S7 or S8 or S9

S11 TX test* n3 order* or TX diagnos* n3 test* or TX screen* n3 test* -

S12 MW diagnosis

S13 (MH “Diagnosis”)

S14 (MH “Health Screening+”)

S15 S11 or S12 or S13 or S14

S16 S10 and S15 Limiters - English Language

Search strategy

Search Query #29 (((DE = choice) or(DE = clinical decision making) or(DE = clinical practice) or(TI = ((clinician* or physician* or doctor* or practitioner*) within 3 (choice* or chos* or choos*))) or(AB = ((clinician* or physician* or doctor* or practitioner*) within 3 (choice* or chos* or choos*))) or(TI = ((clinician* or physician* or doctor* or practitioner*) within 3 (select* or decid* or decision*))) or(AB = ((clinician* or physician* or doctor* or practitioner*) within 3 (select* or decid* or decision*))) or(TI = ((clinician* or physician* or doctor* or practitioner*) within 3 (prefer*))) or(AB = ((clinician* or physician* or doctor* or practitioner*) within 3 (prefer*)))) and((DE = diagnostic testing) or(TI = (diagnosis or diagnostic* or test or tests)))) or(TI =  (test* within 3 order* within 3 (choice* or chos* or choos*))) or(AB =  (test* within 3 order* within 3 (choice* or chos* or choos*))) or(AB =  (test* within 3 order*) AND (choice* or chos* or choice*)) or(TI =  (test* within 3 order*) AND (choice* or chos* or choice*)) or(TI =  (diagnos* within 3 test*) AND (choice* or chos* or choice*)) or(AB =  (diagnos* within 3 test*) AND (choice* or chos* or choice*)) or(AB =  (diagnos* within 3 test*) AND (select* or decid* or decision*)) or(TI =  (diagnos* within 3 test*) AND (select* or decid* or decision*)) or(TI =  (test* within 3 order*) AND (select* or decid* or decision*)) or(AB =  (test* within 3 order*) AND (select* or decid* or decision*)) or(AB =  (test* within 3 order*) AND (prefer*)) or(TI =  (test* within 3 order*) AND (prefer*)) or(TI =  (diagnos* within 3 test*) AND (prefer*)) or(AB =  (diagnos* within 3 test*) AND (prefer*))