Digital and online symptom checkers and assessment services for urgent care to inform a new digital platform: a systematic review

Article history

The research reported here is the product of an HS&DR Evidence Synthesis Centre, contracted to provide rapid evidence syntheses on issues of relevance to the health service, and to inform future HS&DR calls for new research around identified gaps in evidence. Other reviews by the Evidence Synthesis Centres are also available in the HS&DR journal. The research reported in this issue of the journal was funded by the HS&DR programme or one of its preceding programmes as project number 16/47/17. The contractual start date was in April 2018. The final report began editorial review in June 2018 and was accepted for publication in October 2018. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HS&DR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

Andrew Booth is a member of the National Institute for Health Research Complex Review Support Unit Funding Board.

Copyright statement

© Queen’s Printer and Controller of HMSO 2019. This work was produced by Chambers et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2019 Queen’s Printer and Controller of HMSO

Literature search and screening

Initial scoping searches revealed that a highly sensitive search strategy (designed to retrieve all relevant items), typically conducted for systematic reviews, retrieved a disproportionately high number of references on GPs’ decision-making and triage. These references are outside the scope of the NHS Digital 111 review and would have unnecessarily diverted resources from the review focus. Therefore, to optimise retrieval and sifting we devised a three-stage retrieval strategy as an acceptable alternative to comprehensive topic-based searching. This involves:

1. Targeted searches of precise, high-specificity terms (designed to limit the search to items likely to be relevant). Box 1 shows an example for MEDLINE.

   We searched the following databases –The searches were not restricted by language or date.

   • MEDLINE via Ovid

   • EMBASE via Ovid

   • The Cochrane Library via Wiley Online Library

   • CINAHL (Cumulative Index to Nursing and Allied Health Literature) via EBSCOhost

   • HMIC (Health Management Information Consortium) via OpenAthens

   • Web of Science (Science Citation Index and Social Science Citation Index) via the Web of Knowledge at the Institute for Scientific Information, now maintained by Clarivate Analytics

   • the Association of Computing Machinery (ACM) Digital Library.

2. Phrase searching for named and generic systems. The list below was used for phrase searching and was compiled from a study of symptom checker performance6 and an examination of internet search results for symptom checkers –The phrase searches were conducted on the databases listed in point 1, with no language or date restrictions.

   • Askmd

   • askmygp

   • BetterMedicine

   • DocResponse

   • Doctor Diagnose

   • Drugs.com

   • EarlyDoc

   • Econsult

   • engage consult

   • Esagil

   • Family Doctor

   • FreeMD

   • gp at hand

   • Harvard Medical School Family Health Guide

   • healthdirect

   • Healthline

   • Healthwise

   • Healthy Children

   • Isabel

   • iTriage

   • Mayo Clinic

   • MEDoctor

   • NHS Symptom Checkers

   • online triage

   • push doctor

   • Steps2Care

   • Symcat

   • Symptify

   • Symptomate

   • webgp

   • WebMD.

3. Citation searches of key included studies and reviews. Reference checking of included studies and key reviews. This was complemented by contact with service providers, directly and via websites.

Search results were stored in a reference management system (EndNote; Clarivate Analytics, Philadelphia, PA, USA) and imported into EPPI-Reviewer software for screening (version 4; Evidence for Policy and Practice Information and Co-ordinating Centre, University of London, London, UK). The search results were screened against the inclusion criteria by one reviewer, with a 10% sample screened by a second reviewer. Uncertainties were resolved by discussion among the review team.

Inclusion and exclusion criteria

Data extraction and quality/strength of evidence assessment

We extracted and tabulated key data from the included studies, including study design, population/setting, results and key limitations. The full data extraction template is provided in Appendix 1. Data extraction was performed by one reviewer, with a 10% sample checked for accuracy and consistency by another reviewer.

To characterise the included digital and online systems as interventions, we identified studies reporting on a particular system and extracted data from all relevant studies using a modification of the Template for Intervention Description and Replication (TIDieR) checklist8 that we designated Template for Intervention Description for Systems for Triage (TIDieST). The checklist is presented in Appendix 2 and the completed checklists in Appendix 3. Characteristics of included systems are summarised in Chapter 3, Characteristics of included systems.

Quality (risk-of-bias) assessment was undertaken for peer-reviewed full publications only (i.e. not grey literature publications or conference abstracts). The rationale for this approach was that non-peer-reviewed publications tend to lack the detail required for assessment of risk of bias and/or tend not to follow standard study designs. Randomised controlled trials (RCTs) were assessed using the Cochrane Collaboration risk-of-bias tool. 9 For diagnostic accuracy type studies, we used the Cochrane Collaboration version of Quality Assessment of Diagnostic Accuracy Studies (QUADAS)10 and for other study designs we used the National Heart, Lung and Blood Institute’s tool for observational cohort and cross-sectional studies. 11 Details of quality assessment tools can be found in Appendix 4. Quality assessment was performed by one reviewer, with a 10% sample checked for accuracy and consistency by another reviewer. Assessment of the overall strength (quality and relevance) of evidence for each research question is part of the narrative synthesis. Overall strength of the evidence base for key outcomes was assessed using an adaptation of the method described by Baxter et al. 12 This involves classifying evidence as ‘stronger’, ‘weaker’, ‘inconsistent’ or ‘very limited’, based on study numbers and design. Specifically, ‘stronger evidence’ represented generally consistent findings in multiple studies with a comparator group design or comparative diagnostic accuracy studies, ‘weaker evidence’ represented generally consistent findings in one study with a comparator group design and several non-comparator studies or multiple non-comparator studies, ‘very limited evidence’ represented an outcome reported by a single study and, finally, ‘inconsistent evidence’ represented an outcome for which < 75% of the studies agreed on the direction of effect. All studies included in the review were included in the analysis of the overall strength of evidence.

Evidence synthesis

We performed a narrative synthesis structured around the prespecified research questions and outcomes. This included an ‘evidence map’ summarising the quantity and strength of evidence for each outcome and identifying gaps that may need to be filled by further research. We did not perform any meta-analyses because the included studies varied widely in terms of design, methodology and outcomes.

Public and patient involvement in the study

The study aimed to be informed by patient and public involvement (PPI) at all stages of the research process. The Sheffield Health Services and Delivery Research (HSDR) evidence synthesis centre patient and public advisory group provided input during the design, analysis and reporting phases, including exploration of the study parameters, discussion regarding the meaning and interpretation of the study findings, drafting of the Plain English summary and help with disseminating the findings and maximising the impact of the research.

The advisory group comprised nine members drawn from the Yorkshire and Humber region and two members from other regions of England. Because this study was of a relatively short duration, the group provided input at two advisory group meetings. At the meetings there was discussion regarding the focus of the work, including a presentation on previous research on NHS111 services to provide a context for understanding the current work. The meetings also included the presentation and discussion of the findings of the review in order to explore key messages for patients that could inform the dissemination of the findings.

The discussion during one meeting was structured using a SWOT (strengths, weaknesses, opportunities and threats) analysis approach that revealed a number of potential concerns among patients as well as potential perceived benefits. The group members expressed some concern over the reliability and consistency of symptom checker algorithms, particularly if different systems are being used in different parts of the country. The economic benefits of the systems were also questioned in view of the high costs of programming and system development. Possible equity issues were identified because digital 111 might be less accessible to some groups (people with cognitive impairment were mentioned). The group expressed uncertainty about the impact of this type of service on the wider urgent and emergency care system (e.g. would the creation of a new access point complicate patient pathways and increase demand overall?). The possible vulnerability to external threats leading to the breakdown of the system or the loss of data was identified as a weakness. There was discussion of whether or not patients would feel able to trust the advice they were given. Some group members saw the desire of patients to be able to talk to someone about their problems for reassurance and empathy as a threat to the success of digital 111. Potential benefits included increased access to urgent-care advice at any time and appeal to younger people and to those who might feel anxious or embarrassed about discussing their problem with a health professional.

Involvement of the advisory group was also beneficial in highlighting some issues that had emerged from the systematic review and enabled the reviewers to structure the review findings while taking this into account. The group’s uncertainty about the probable impact of digital 111 was reflected in the report’s findings and recommendations for ongoing evaluation and further research. The review report also reflects the group’s relatively cautious attitude (while recognising the need to update the way services are accessed), which contrasts with the strong belief in some quarters that digital 111 will help to ensure that patients receive appropriate care more quickly while reducing ‘inappropriate’ visits to EDs and GP appointments. Advisory group members reported that they had limited experience of using symptom checkers for real-life health problems in the urgent-care setting, which echoes the limited available information and uncertainty regarding the probable effects of digital 111 services.

Study registration and outputs

The protocol was registered prospectively with PROSPERO (registration number CRD42018093564) and is also available via the National Institute for Health Research (NIHR) HSDR programme website (www.journalslibrary.nihr.ac.uk/programmes/hsdr/164717/) and the Sheffield HSDR Evidence Synthesis Centre website (https://scharr.dept.shef.ac.uk/hsdr/).

Results of literature search

This chapter presents the studies that were included in the review. A PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) flow diagram (Figure 1) details the search process.

FIGURE 1.

The PRISMA flow diagram. Some parts of this chapter have been reproduced from Chambers et al. 1 © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY. Published by BMJ. This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original text.

Some parts of this chapter have been reproduced from Chambers et al. 1 © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY. Published by BMJ. This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original text.

All titles and abstracts were screened by one researcher from the review team, with a subset (about 10%) of the titles and abstracts screened by one other researcher from the review team. A calculation of the inter-rater agreement was made. A kappa coefficient was calculated, demonstrating moderate agreement between reviewers [κ = 0.582, 95% confidence interval (CI) 0.274 to 0.889]. Any queries were resolved by discussion. A similar process was followed for final decisions on inclusion/exclusion, based on full-text documents.

Characteristics of included studies

We included 29 publications that represented 27 studies (one study13 had two associated comments14,15). Nine studies were performed in the UK. Seventeen studies (Table 1) evaluated symptom checkers as a self-contained intervention, of which eight covered a limited range of symptoms (e.g. respiratory16,17,25 or gastrointestinal28,30 symptoms that we considered to be ‘urgent’). The remaining studies in this group evaluated symptom checkers that covered a wider range of common urgent-care symptoms. Studies evaluated either a single system21,22,24,27 or multiple systems. 6,13 We found only one study of a symptom checker specifically intended for the assessment of children’s symptoms, a development of the Strategy for Off-Site Rapid Triage (SORT) system for influenza-like illnesses. 23 Two reports with some overlap of content evaluated the ‘babylon check’ application (app). 26,27

Five studies7,31–34 evaluated symptom checkers as part of a broader self-assessment and consultation system (often referred to as electronic consultation or e-consultation). Study characteristics are summarised in Table 2. In this type of system, the role of symptom checkers is to help patients decide whether their symptoms require a consultation with a doctor or other health professional or can be dealt with by self-care. If a consultation is required, details of the symptoms and a request for an appointment or a call-back can be submitted electronically. This type of study is important because it considers the service in the broader context of the urgent and emergency care system. A limitation is that some studies focused mainly on the ‘downstream’ elements of the pathway (e.g. consultation with GPs) and provided limited data on the symptom checker element of the system.

A final group of five studies (Table 3) examined patient and/or public attitudes to online self-diagnosis in the context of urgent care. 35–39

The included studies used a wide range of designs, some of which were challenging to classify. Studies that assessed the systems’ performance in terms of clinical and service use outcomes, including patient satisfaction, were generally observational or qualitative, although one web-based system was evaluated in two RCTs. 17,25 Diagnostic accuracy was assessed by measuring performance in simulations (e.g. using vignettes to describe symptoms for a known condition) or by comparing the system’s performance with that of doctors in diagnosing the cause of symptoms and choosing an appropriate level of triage. This was carried out using either simulated data or real-world data (e.g. by asking patients to complete a symptom checker assessment before seeing a doctor and comparing the two assessments). Risk of bias in studies and the overall strength of evidence for different outcomes are discussed further below (see Risk-of-bias assessment and Overall strength of evidence assessment, respectively).

The publication status of the included studies also varied. In addition to peer-reviewed journal articles and reports, we included four studies published as conference abstracts18,28–30 and three reports characterised as grey literature. Two of these reports27,34 were written and published by the developers of specific systems and may therefore be subject to potential conflicts of interest. The third grey literature report was a draft report of NHS England’s ongoing pilot studies of NHS111 online in four regions of England. 7 This report, dated December 2017, was not officially published by NHS England but was easily available online.

Characteristics of included systems

Using studies included in the review, we were able to extract data on eight systems using the TIDieST checklist. When appropriate, data from multiple studies were combined in one checklist. We noted that some of these systems were no longer in use. We were also unable to obtain sufficient data for some systems that are currently being used and/or evaluated. A summary of extracted data is presented below (Tables 4 and 5) and full data extractions can be found in Appendix 3.

Four of the included systems were designed to cover a full range of symptoms (see Table 4) and four covered a more limited range (three for influenza-like illness and one for minor respiratory symptoms; see Table 5). Most systems were accessed through web pages, often linked to health-care providers or government organisations. The ‘babylon check’ system was the main exception, being designed for access using a smartphone app. Published research studies provided relatively little detail about the systems that, in some cases, possibly reflected a need for commercial confidentiality. Details of published studies are included in Tables 4 and 5 for reference; the findings of these studies are summarised in the following section for each key outcome.

Results by outcome

Patient and public attitudes to online self-diagnosis

Seven cross-sectional papers were included in the review. 18,19,35–39 Study characteristics are summarised in Table 13. The studies were carried out in the USA,18,36,38,39 Norway19,37 and Sweden35 and all but one, a conference abstract,38 were published articles.

Included studies used a range of cross-sectional research methods, including a review of online resources, surveys, interviews and ‘think aloud’ sessions to assess the quality, usability, acceptability of and the intention to use internet-based health-care information. Given the generally small samples of specific populations (e.g. students, older people, affluent) the generalisation of findings is limited, but the issues identified could be useful when considering the development of a new service.

Six out of the seven papers were assessed for quality, given that the one conference abstract fell outside our criteria for appraisal. 38 The remaining studies were varied in terms of methods and how the methods fit with the appraisal criteria and, therefore, they were not graded; instead, strengths and weaknesses were discussed. All six papers included a clear statement of the research question and four of them presented clear outcome measures, but confounding variables were less clearly dealt with. Only two studies included a participant response rate,19,35 and both of these indicated that the rate was > 50%, but only five studies recruited participants. Blinding of assessors was either not appropriate or not stated for all of the studies. Other weaknesses included inconsistency of reporting the findings and a general lack of enough detail to make a decision regarding the risk of bias.

Aims of the included studies can be distinguished between those that examine characteristics of the supply side (online sources) and those of the demand side (users and potential users), or both. This indicates an interest in:

• the development of online resources for health-care information or advice

• behaviour of users in relation to online health-care information seeking.

These two issues are dealt with separately in the sections below.

Risk-of-bias assessment

We assessed risk of bias in the two included RCTs17,25 using the Cochrane risk-of-bias tool. 9 Thirteen studies16,18–21,24,31–33,35–37,39 were assessed with the tool for cross-sectional and cohort studies and four (six publications6,13–15,22,23) were assessed with the modified QUADAS tool. 10 Seven grey literature reports and conference abstracts were not formally assessed for risk of bias. 7,27–30,34,38 Identified limitations were extracted for all included studies, the risk-of-bias results are presented in Appendix 5.

No serious problems were identified with the two randomised trials, although blinding was somewhat unclearly reported. The studies evaluated using the cross-sectional/cohort study tool were variable both in design and risk of bias. Most studies addressed a clear research question in a defined population. Participation rates, when applicable, varied widely between studies. None of the studies provided a sample size justification, making their statistical power uncertain, and some findings were based on very small samples. Most of the studies had clearly defined outcome measures but none reported on blinding (e.g. of outcome assessment). Adjustment for confounders is important for observational studies but only three of the included studies were judged to have adjusted adequately for confounders (see Appendix 5).

The four ‘diagnostic’ studies assessed using the modified QUADAS tool had some important limitations. Two did not recruit representative samples of patients22,23 and the other two used patient vignettes, so were judged ‘unclear’ for this aspect. 6,13 There were also issues with blinding and potential partial verification bias in individual studies.

With the possible exception of the two RCTs,17,25 the included studies generally had at least a moderate risk of bias. However, the diverse designs and objectives of the studies made risk of bias difficult to assess in some cases with the available tools. Grey literature reports containing relevant data were included in the review but not formally assessed for risk of bias. Reports prepared by individuals with a commercial interest in a specific system and published without independent peer review27,34 should be treated with particular caution because of possible conflicts of interest.

Overall strength of evidence assessment/evidence map

The overall strength of evidence for key outcomes was assessed using a variant of the scale used by Baxter et al. ,12 which relates the evidence for particular outcomes to the size and quality of the whole body of evidence (Table 14). The rating scale was as follows: ‘stronger evidence’ represented generally consistent findings in multiple studies with a comparator group design or comparative diagnostic accuracy studies, ‘weaker evidence’ represented generally consistent findings in one study with a comparator group design and several non-comparator studies or multiple non-comparator studies, ‘very limited evidence’ represented an outcome reported by a single study and ‘inconsistent evidence’ represented an outcome where < 75% of the studies agreed on the direction of effect.

Using this scale, there is relatively strong evidence that digital and online symptom checkers have not been shown to present any major patient safety issues, that their diagnostic accuracy tends to be lower than that of health professionals and that patients who have used these systems generally show high levels of satisfaction (mainly in non-comparative studies). Areas where evidence is lacking or inconsistent include clinical effectiveness and cost-effectiveness, accuracy of disposition to appropriate services and patient compliance with advice received. The implications of these findings will be considered in Chapter 4.

Main findings

The literature search identified 29 publications describing 27 studies that met the inclusion criteria. Studies were published between 2006 and 2018 and were diverse in terms of their design and methodology. Four items of grey literature were included in view of their importance to the topic, although three were produced by the developers of specific systems and should be interpreted with that fact in mind. 26,27,34 At the time of writing this report, NHS England’s evaluation of four different systems in different regions was still in progress but an informally published draft report was located online7 and we have used that in our review while acknowledging that it represents an incomplete preliminary data set.

The overall strength of the evidence base varied between outcomes (see Table 14), but in absolute terms the evidence is weak, being based largely on observational studies. A substantial component of grey literature of uncertain quality complicates the interpretation of the evidence. Interpretation of the evidence should also take into account the risks of bias in individual studies, as discussed in Chapter 3, Risk-of-bias assessment.

We found little evidence to support the hypothesis that digital and online symptom checkers are detrimental to patient safety. Although the evidence was comparatively strong relative to other outcomes assessed in the review, it falls far short of being an adequate assessment of the safety of these systems. The studies that reported on the outcome were mostly short term and involved relatively small samples. Some were limited to people with specific types of symptoms (e.g. influenza-like illness23 or respiratory infections17) and others recruited from specific population groups (e.g. students22) who were not representative of typical users of urgent-care services. This body of evidence should, therefore, be interpreted cautiously and not be extrapolated to the possible impact of a nationally available digital urgent-care service being used by millions of people annually. Anecdotal reports of informal tests circulated via social media show symptom checkers making serious errors, but we were unable to locate any research to substantiate these claims.

The evidence on patient satisfaction with digital and online systems also had some limitations, but these findings appear more likely to be generalisable. Study participants generally expressed high levels of satisfaction, albeit in uncontrolled studies. For example, in the NHS England pilot evaluation, 70–80% of users were satisfied with their experience at each of the pilot sites. 7 This was based on a sample of > 1500 users, the great majority of whom were involved in the London pilot using the ‘babylon check’ app. Despite the unbalanced sample, the findings on satisfaction suggest that any national, digital urgent-care service may be well-used, particularly given the increasing reliance on digital technology in all areas of life. Satisfaction was measured in a number of ways, but generally the studies appeared to be rating usability rather than satisfaction with the advice received or the degree of reassurance provided. Failure to provide the patient with reassurance could result in them seeking further help elsewhere in the urgent and emergency care system.

The evidence from the studies included in the review suggests that digital and online systems have yet to achieve a high level of accuracy in the diagnosis of specific conditions. This finding applies both to ‘general purpose’ symptom checkers and to those limited to particular conditions. Although the evidence was classified as relatively strong, several caveats should be applied. For example, some of the included studies did not recruit representative populations and others were based on standardised vignettes rather than real-world data. Finally, those studies that compared symptom checkers with health professionals tended to use the doctors’ clinical diagnosis as the reference standard, which would bias the comparison in favour of the health professionals. There was an indication from one study that the symptom checker performed better when additional information was available. 29

The accuracy of signposting patients to the most appropriate level of service is closely related to diagnostic accuracy but results for this outcome were inconsistent between studies. Studies researching the accuracy of signposting used similar methods to those evaluating diagnostic accuracy. A review of 23 symptom checkers found that triage level was appropriate overall for 57% of patient vignettes, with considerable variation between systems and conditions. 6 The NHS England evaluation also found differing levels of agreement with clinical expert opinion across the four systems evaluated. 7 In general, algorithm-based triage tended to be more risk averse than that of health professionals, with 85% of respondents being advised to visit their doctor in one study. 21 Although there is considerable uncertainty about the magnitude of the effect, a national, digital urgent-care service could result in considerable numbers of patients receiving inappropriate advice to visit the ED or request an urgent general practice appointment. Middleton et al. 27 claimed that the ‘babylon check’ app had a high degree of triage accuracy for vignettes when compared with health professionals, but this report was not peer reviewed and so requires further validation.

We also found inconsistent evidence on effects on service use, but there was some indication that symptom checkers can influence the pattern of service use. The strongest evidence comes from a randomised trial of an intervention that was specifically designed to promote self-care and covered respiratory symptoms only. 17 In this study, the intervention group had less contact with doctors (but more contact with NHS Direct) than the control participants despite having a longer duration of illness and illness of a greater severity. It is uncertain whether or not these results are generalisable to systems covering the full range of urgent care. However, the NHS England evaluation found a small shift towards self-care with digital triage compared with telephone triage,7 and Madan34 reported that the webGP symptom checkers diverted 18% of patients from requesting a general practice appointment in that consultation. These are findings from the grey literature that have not been peer reviewed and should, therefore, be interpreted with caution.

There is very limited evidence on patients’ reactions to online triage advice and whether they follow the advice or seek further help or information. This would have implications for service use. Preliminary findings from the NHS England evaluation7 suggest that patients may be more likely to seek further advice for more urgent conditions, but further confirmation is required.

Over half of the included studies considered equity and inclusion issues either directly or by comparing users with non-users of digital triage systems. Not surprisingly, studies revealed a clear consensus that younger and more highly educated people are more likely to use these services, whereas older and less educated people are more likely to prefer telephone or face-to-face contact. This could have implications for health equity if urgent-care pathways prioritise (or appear to prioritise) requests originating from digital sources. In primary care, issues have arisen because patients using e-consultation systems to request an appointment following online triage may be seen more quickly than those contacting the practice by telephone.

In summary, the key findings of this systematic review are that considerable uncertainties surround the diagnostic and triage accuracies of digital and online symptom checkers, their overall effects on service use and whether patients are likely to follow their advice or seek further assistance from other sources. Surveys of patients who have used them report high levels of satisfaction with their usability. Increasing reliance on digital and online services generally suggests that a national digital 111 service may be heavily used and its impact on the health-care system, in whatever direction, could be substantial. This review found limited evidence to suggest whether the new service will stimulate or manage demand for urgent and emergency care.

Strengths and limitations

This systematic review has been undertaken to a timescale determined by the review commissioners using a relatively large team of experienced researchers, including both methodological experts and topic experts. In other words, we have sought to accelerate the systematic review process rather than abbreviate it by omitting some processes. The main exception to this was the use of one reviewer for some processes (with checking of a 10% sample carried out by a second reviewer) rather than full, independent duplication. The evaluation of the screening process suggested that there was only a moderate degree of agreement between reviewers. This reflects a degree of learning by the review team during the screening process. Our initial sift of the search results consciously favoured inclusivity, and items that were found not to meet the inclusion criteria on detailed examination were subsequently discarded. We performed a rigorous search of the literature, including reference checking and citation searching. Rather than a conventional, highly sensitive search (which would have resulted in inefficiencies in the screening process), we combined an initial focused search with subsequent rounds of follow-up searching, including searches for named symptom checker systems. Overall, we consider it unlikely that any significant research in this area was overlooked.

We assessed risk of bias in individual studies using a variety of appropriate checklists, as well as summarising the overall strength of evidence for key outcomes (see Table 14). The protocol was developed in association with NHS England and the NIHR HSDR programme team and was registered prospectively on PROSPERO.

The heterogeneous and descriptive nature of the included studies meant that meta-analysis was not feasible for any of the outcomes of interest. Our narrative synthesis approach used a mixture of description and tabulation to summarise the evidence for each of the prespecified outcomes of interest. In addition, we considered the different symptom checker systems as interventions and extracted data on them using the system rather than the study as the unit of analysis (see p. 23 and Appendix 3); we also used an adaptation of an established intervention-reporting checklist to assess them. 8 This process highlighted the lack of information available for some systems and it could be developed to form a template for ongoing evaluations of current and new systems.

This was a review of published, including non-peer-reviewed, literature and the coverage of systems is not exhaustive (e.g. we did not extract data from websites). We also did not carry out any original analyses of raw data even when such data were available. The timing of the review meant that the final results of NHS England’s pilot evaluation were not available to us. We were able to make use of a draft report that was published online,7 but we acknowledge that the findings of the final evaluation report, when available, will supersede those of the 2017 draft.

The review inclusion criteria were relatively broad and findings from symptom checker systems for specific conditions may not be applicable to more general systems and vice versa. We have also included studies of symptom checkers as part of e-consultation systems in general practice, which represent a slightly different setting from a general digital 111 service, and this should be kept in mind when interpreting the results.

Implications for service delivery

The implications of this systematic review for service delivery should be considered in the context that a decision has already been taken to introduce a digital 111 service and that the implementation of the service is in progress. The findings can be used to support the ongoing development and evaluation of the service.

Achieving a high level of diagnostic accuracy will be key to the success of a digital 111 service. Failure to provide an accurate diagnosis may result in outcomes such as patient dissatisfaction and unwillingness to use the service again, increased use of other urgent and emergency care services and possible risks to patient safety (although the cautious approach characteristic of most existing systems that we identified may help to mitigate this).

The studies included in the review suggest a high level of uncertainty about the impact of digital 111 on the urgent-care system and the wider health-care system. Some of these uncertainties can be addressed by research and data collection, as discussed below, but the health service may need to respond to short-term increases (or decreases) in demand and/or shifts from one part of the system to another. This may increase pressure on the system, at least in the short term. In the longer term, if the use of the NHS111 telephone service decreases as planned, there may be opportunities to re-deploy staff to fill other roles within the urgent and emergency care system.

Research on the introduction of the NHS111 telephone service from 2010 was outside the scope of this review. However, the telephone service followed a similar pattern of piloting followed by nationwide implementation to that proposed for digital 111. In brief, the evaluation of the pilot sites (compared with control sites where NHS111 was not available) found that NHS111 did not reduce calls to ambulance services,5 did not improve public perceptions of urgent care provision41 and identified key groups that were less likely than others to use the service (including older people and men). 42 However, the majority of respondents in the study of acceptability were very satisfied (73%) or quite satisfied (19%) with the service overall. 42 A later study of five NHS111 service providers covering 2014–16 also found high levels of patient satisfaction, but views of other stakeholders were more mixed. 43 Decision-makers may be able to use the lessons learnt from the introduction of the NHS111 telephone service alongside the findings of this review to inform their choices around the implementation of digital 111. For example, some respondents had concerns about the relevance of the questions they were asked,42 emphasising the importance of flexibility in the underlying algorithm to prevent users giving up in frustration and seeking help elsewhere in the urgent and emergency care system.

Implications for research

The limitations of the evidence base suggest that further rigorous research is urgently needed. We have identified the following priorities for research (in addition to the ongoing collection of data to monitor the use and safety of the digital 111 service):

• Our research suggests that there is a wide variation in performance between symptom checker systems,6 but we found no studies designed to compare systems directly. Such research will be important for commissioners to make an informed choice on how to implement services at a local level.

• Evidence on costs or cost-effectiveness was classified as very limited; hence, rigorous economic evaluations based on real-world data are needed. A key question for research is whether or not new digital services will reduce overall costs for the health service and whether or not savings are made in one part of the system at the expense of another.

• The introduction of a digital 111 service provides a new point of access for patients needing urgent medical help. Research is needed to investigate the pathways followed by patients using the service and, particularly, if the overall number and level of contacts with the health system can be reduced without affecting the quality and safety of patient care.

• This review located only one study of a service designed to assist parents and other caregivers to assess the acuity of a child’s illness. 23 Research is required to ensure that the systems underlying national digital 111 services are suitable for children’s illnesses as well as those of adults, or to evaluate a separate ‘paediatric digital 111’ service.

• Given the example of the Internet Doctor intervention, with a design based on a specific psychological theory,17,25 the role of behaviour change theory in the development and implementation of symptom checkers should be investigated.

Conclusions

The current evidence base covers a diverse range of interventions, study designs and outcomes. The evidence is generally weak as it is based largely on observational studies and includes some studies that have not been independently peer reviewed. There are major uncertainties around the probable impact of digital 111 services on most of the important outcomes, but evidence on patient satisfaction, together with developments outside the health field, suggests that once introduced the use of these services may increase rapidly. It will be important to monitor and evaluate the services using all available data sources and by commissioning high-quality research.

Contributions of authors

Duncan Chambers (Research Fellow in Public Health) was involved in project co-ordination, study selection, data extraction, quality assessment and report writing.

Anna Cantrell (Research Associate in Health Economics and Decision Science) was involved in information retrieval, study selection, data extraction, quality assessment and report writing.

Maxine Johnson (Research Fellow in Public Health) was involved in study selection, data extraction, quality assessment and report writing.

Louise Preston (Research Fellow in Health Economics and Decision Science) was involved in study selection, data extraction, quality assessment and report writing.

Susan K Baxter (Senior Research Fellow in Public Health) was involved in protocol development, study selection, report writing and co-ordinated the PPI meeting.

Andrew Booth (Reader in Evidence-based Information Practice) was involved in information retrieval, study selection and report writing.

Janette Turner (Reader in Emergency and Urgent Care Research) provided topic expert advice and was involved in report writing.

All authors commented on drafts of the protocol and the report.

Publication

Chambers D, Cantrell AJ, Johnson M, Preston L, Baxter SK, Booth A, Turner J. Digital and online symptom checkers and health assessment/triage services for urgent health problems: systematic review. BMJ Open 2019;9:e027743.

Data-sharing statement

No new data have been created in the preparation of this report and, therefore, there is nothing available for access and further sharing. All queries should be submitted to the corresponding author.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HS&DR programme or the Department of Health and Social Care.

Quality assessment tool for observational/cross-sectional studies

1. Was the research question clearly stated?

2. Was the study population clearly specified and defined?

3. Was the participation rate of eligible persons at least 50%?

4. Were all the subjects selected or recruited from the same or similar populations (including the same time period)? Were inclusion and exclusion criteria prespecified and applied uniformly to all participants?

5. Was a sample size justification, power description, or variance and effect estimate provided?

6. For exposures that can vary in amount or level, did the study examine different levels of the exposure as related to the outcome?

7. Were the exposure measures (independent variables) clearly defined, valid, reliable and implemented consistently across all study participants?

8. Were the outcome measures (dependent variables) clearly defined, valid, reliable and implemented consistently across all study participants?

9. Were the outcome assessors blinded to the exposure status of participants?

10. Were key potential confounding variables measured and adjusted statistically for their impact on the relationship between exposure(s) and outcome(s)?

Quality assessment tool for diagnostic studies

1. Was the spectrum of patients representative of the patients who will receive the test in practice? (Representative spectrum.)

2. Is the reference standard likely to classify the target condition correctly? (Acceptable reference standard.)

3. Is the time period between reference standard and index test short enough to be reasonably sure that the target condition did not change between the two tests? (Acceptable delay between tests.)

4. Did the whole sample or a random selection of the sample receive verification using the intended reference standard? (Partial verification avoided.)

5. Did patients receive the same reference standard irrespective of the index test result? (Differential verification avoided.)

6. Was the reference standard independent of the index test (i.e. the index test did not form part of the reference standard)? (Incorporation avoided.)

7. Were the reference standard results interpreted without knowledge of the results of the index test? (Index test results blinded.)

8. Were the index test results interpreted without knowledge of the results of the reference standard? (Reference standard results blinded.)

9. Were the same clinical data available when test results were interpreted as would be available when the test is used in practice? (Relevant clinical information.)

10. Were uninterpretable/intermediate test results reported? (Uninterpretable results reported.)

11. Were withdrawals from the study explained? (Withdrawals explained.)

12. Comments.

Risk-of-bias results for randomised trials

Risk-of-bias results for cohort/cross-sectional studies

Risk-of-bias results for diagnostic studies