The effects of computerised decision support systems on nursing and allied health professional performance and patient outcomes: a systematic review and user contextualisation

Article history

The contractual start date for this research was in October 2019. This article began editorial review in October 2021 and was accepted for publication in July 2022. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The Health and Social Care Delivery Research editors and publisher have tried to ensure the accuracy of the authors’ article and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this article.

Copyright statement

Copyright © 2024 Thompson et al. This work was produced by Thompson et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This is an Open Access publication distributed under the terms of the Creative Commons Attribution CC BY 4.0 licence, which permits unrestricted use, distribution, reproduction and adaption in any medium and for any purpose provided that it is properly attributed. See: https://creativecommons.org/licenses/by/4.0/. For attribution the title, original author(s), the publication source – NIHR Journals Library, and the DOI of the publication must be cited.

2024 Thompson et al.

The target for CDSS: decision-making by nurses, midwives and allied health professionals

Historically, decisions about the delivery and organisation of healthcare were assumed to be the province of doctors. Whilst medical dominance has proven remarkably resistant to challenge, ‘decision-rich’ areas such as the prescribing of medications,5 the initiation of critical care outreach in acute care, nutritional management and rehabilitation planning offer the chance for professions other than medicine to formally use their decisions to shape the delivery of healthcare, how care processes are experienced and the clinical outcomes that result.

Alongside formal decisions in healthcare such as assigning a diagnosis, prescribing a treatment, or offering a prognosis, the realpolitik of healthcare delivery relies on a range of informal judgements, decisions, and negotiated positions between a range of professionals – often with fluid and overlapping roles. 6 Technology has encroached into healthcare decision-making, purporting to offer support, information and recommendations to help shape professional decisions. 7

In this synopsis we focus on nurses, midwives and allied health professionals (NMAHPs). Why? Because their work, demographic composition, educational levels, and socio-economic positions often differ from medicine and doctors, but they contribute to a complex, fluid, and – crucially – negotiated division of labour in healthcare. 8 Authority within this division of labour stems in part from the power to exercise clinical judgement, clinically reason and make or shape decisions. If we assume that work in healthcare is based on – and reflects – reasoned judgements and decisions, then it follows that different professionals in multidisciplinary teams will face different uncertainties, judgements and decisions. The support needed for tackling differing uncertainties may also be different.

Research into the decisions and decision-making of NMAHPs is relatively scarce compared to studies focusing on doctors and medical reasoning, although eminent decision scientists have studied nursing decisions since the 1960s. 9 Researchers have also described and typologised ‘nursing’ decisions. 3 Some scholars point out that there is no de facto reason why nurses – and by implication, other health professionals – should be treated as possessing their own, unique, decision-making cognition, even if decisions when viewed in context appear different. 10 Others have extended well-established descriptive and prescriptive theories of generic professional decision-making to incorporate forms of knowledge and knowing (such as ‘reflection-in-action’) associated with particular groups – notably nurses. 11

At the heart of attempts to describe, model and theorise clinical decision-making are two core constructs:

• judgements – the weighing-up or evaluation of clinical, research or other information

• decisions – choosing between discrete options.

The most parsimonious models of decision-making bring together judgement, choice and evaluation in a ‘feedback loop’. 12 Consequently, decision-making rarely feels like a discrete event made up separate ‘stages’ to the decision-maker. Despite the difficulties of ‘holistic’ decision-making as experienced by decision-makers, there is value in separating it into component elements; in part because the characteristics of decisions (1) determine the style of clinical reasoning best suited to a decision (intuition vs. rational information-processing), but also (2) shape the likelihood of using different forms of decision support. 3,11 The perceived time available to make choices, the perceived structure of a choice, and the need to show how you got to a judgement or decision (i.e. a choice’s visibility) can increase or decrease the chances of using technology-delivered support. 3,11

Support for decision-making often comes in the form of technologies – ranging from paper-based aids such as printed guidelines or research summaries, to web, app and computer-based decision support systems. More commonly, support can also come from informal resources such as a colleague’s advice, or, at the extreme, a professional’s own internalised resources in the form of experience, knowledge recalled from training, or just gut instinct or ‘intuition’. It is the application of computer technology to judgements and decisions that is our focus in this synopsis.

What are computerised decision support systems?

Computerised decision support systems (CDSS) are software- or computer-based technologies that offer patient-specific recommendations based on either research, expert opinion, machine learning/artificial intelligence or combinations of these, and designed to influence the clinical decision-making of health professionals. 13–15 CDSS access patient information from practitioners, healthcare staff, patients’ manual data entry or queries of electronic medical records before research or expert knowledge is assessed to provide computer-generated recommendations delivered to the clinician via a computer/tablet, mobile-phone screen or electronic medical record. Clinicians can then choose whether to use these recommendations. Examples of decision support used by NMAHPs include: assessing fall risk and preventative behaviours;16 pressure-ulcer management;17 selecting interventions for managing musculoskeletal disorders;18 screening for childhood language disorders;19 depression screening20 and, on a whole-system scale, choices faced in clinical pathways for primary care triage and prioritisation (https://digital.nhs.uk/services/nhs-pathways).

CDSS come in two main forms: (1) knowledge-based and (2) non-knowledge-based. 21 Knowledge-based CDSS use logical ‘IF-THEN-ELSE’ rules to evaluate information provided directly by a clinician or drawn from an electronic health record. These are then matched to a computerised knowledge base (in many cases expert opinion or national/international clinical practice guidelines) to provide assessments/management options/probabilities or actionable recommendations or outputs. 21,22 These forms of CDSS automate information-gathering and provide advice in line with guidelines. Examples of this type of CDSS are drug prescription/alert tools and emergency and out-of-hours telephone calls used for triaging patients. Non-knowledge-based CDSS use machine learning and artificial intelligence rather than flowchart-style rules or logic to support clinicians’ decision-making. 21 Typical examples of this type of CDSS are predictive risk models for assessing the prognosis of a disease outcome. 23 CDSS based around artificial intelligence and/or machine learning are less common than rule-based systems in NMAHP work.

CDSS systems can stand alone, integrated into, or at least capable of interacting with, wider digital infrastructure in health systems such as electronic health records (EHRs) or computerised physician order entry (CPOE – computer-based systems that automate instructions, with standardised, legible, and complete orders). They can be hosted via a computer, tablet or smartphone, and have web-based/local or ‘app’ interfaces. CDSS can present information on host devices or via the integrated EHR/CPOE system.

Why look at CDSS for NMAHPs?

NMAHPs make decisions that could benefit from digital support. Unwarranted variations in practice and outcome, for patients with seemingly similar issues and facing similar decisions and uncertainties, exist. These uncertainties make a synthesis of empirical research timely and useful. 24–26 New ways of working and support for these new roles for NMAHPs feature in many health systems. Opportunity costs associated with digital technology for learners and educators exist: professional preparation for and continuing professional development of digitally competent clinicians able to use new technologies effectively require time. 27

We had three main research-based motivations for the synthesis. First, clinical decision support systems will only be useful if they improve clinicians’ decision performance (for example, more accurate diagnoses and prognoses), improve patient health outcomes (e.g. morbidity, mortality, fewer adverse events), and offer perceived value for money for health services. 28,29 We do not know if any of these are true for NMAHPs.

Second, a previous review of studies on CDSS use by nurses found only limited impact on performance and health outcomes. 30 The review is more than 13 years old and digital technology and the research evidence base has developed significantly. The effect of CDSS on allied health professionals (AHPs) has not been reviewed systematically. Systematic reviews of studies on the impact of CDSS on healthcare delivery generally suggest they can improve practitioner performance in specific areas of decision-making such as diagnosis (4/10 systems), disease management (23/37 systems) and drug-dosing or prescribing systems (19/29 systems). 4 The impact on patient outcomes is more equivocal, with only 13% of systems (7/32) reporting improvements. 4 Reviews focusing on specific areas of clinical practice such as prescribing and drug dosing 31 or clinical subdomains such as neonatal care32 offer very limited conclusions, because the underpinning evidence is either absent,32 low quality and\or narrow in scope. 31

Third, existing reviews often neglect the fact that whilst multi-disciplinary team members may all be involved in delivering healthcare, their decisions reflect their role in the division of labour and so are likely to differ. Extant reviews often contain an implicit rationale that doctors’ decisions alone are the main mechanism for improving healthcare processes and outcomes. 4

How are clinical decision support systems supposed to improve decision-making?

Clinical decision support systems work by providing high-quality relevant useful information delivered when it is required to decision-makers. 13 The main generative mechanism by which CDSS aid NMAHPs decision-making is the combination of CDSS-generated information/suggestions with existing nurse or AHP knowledge. Thus, CDSS augment or supplement clinician decision-making rather than replacing it. CDSS are a key means of encouraging concordance with guideline-based care to reduce unwarranted variations in practice. 33

Examples of decisions supported by CDSS include:34

• Recognising patient deterioration – CDSS can increase situational awareness or incorporation of relevant clinical and research-based information in reasoning, and tailoring of local or national guidance.

• Determining patients with conditions that merit the application of clinical guidelines – CDSS improves the consistency of judgements and adherence to guideline recommendations and reduces (unwarranted) variation.

• Triaging patients, often in the emergency department or primary care, to determine priority cases – the CDSS improves the reliability of judgements and simplifies choices by reducing the ‘noise’ in the situation and amplifying the appropriate ‘signals’ to encourage more appropriate decisions.

As with any health technology, CDSS will only improve care and health if actually used by nurses and AHPs in their decision-making. Whatever the quality of the underlying knowledge base, decision rules, analyses or algorithms, if unimplemented, or implemented badly, will not improve decision quality and patient benefit is less likely. Unfortunately, CDSS implementation and use by NMAHPs is rarely straightforward, and can be suboptimal. 35–37

CDSS can create the potential for harms as well as benefits. 38 These include fragmentation or disruption of work and workflow; alert fatigue; deskilling and the consequences on decisions of poor-quality or incorrect knowledge in the data used for inference or analysis. Additionally, CDSS may rely on a user’s computer literacy – something that is highly variable in nurses and AHPs. Systems can incur opportunity costs for clinicians as well as those charged with maintaining and supporting technology in health systems. 15 CDSS can also widen existing inequalities in access to high-quality care; for example, where effective CDSS are located only in prestigious teaching hospitals and associated with improved access to services, then patients who do not have access to teaching hospitals will be disadvantaged. 39,40

Thus, there are three main mechanisms by which CDSS ‘work’ in the context of decision-making by NMAHPs: successfully combining high-quality or novel CDSS information and clinician knowledge; improving quality of care processes and – by implication – outcomes, by improving the appropriateness of recommendations, management/treatment choices, accuracy of predictions or diagnoses; and successful implementation and use by clinicians.

Literature searching

With an information specialist, we developed a search strategy designed to find studies focusing on CDSS and the healthcare professionals we were interested in: nurses, midwives and allied health professionals.

We ran the search strategy on multiple electronic databases and resources twice: October 2019 and February 2021. Specific databases searched included: MEDLINE (Ovid), Embase Classic+Embase (Ovid), PsycINFO (Ovid), HMIC (Ovid) Health Management Information Consortium, AMED (Allied and Complementary Medicine) (Ovid), CINAHL, Cochrane Central Register of Controlled Trials (Cochrane Database of Systematic Reviews, Wiley), Social Sciences Citation Index Expanded (Clarivate), ProQuest Dissertations & Theses Abstracts & Index, ProQuest ASSIA (Applied Social Science Index and Abstract), ClinicalTrials.gov, World Health Organisation International Clinical Trials Registry (ICTRP), Health Services Research Projects in Progress (HSRProj), OpenClinical (www.OpenClinical.org), OpenGrey (www.opengrey.eu), Health.IT.gov, Agency for Healthcare Research and Quality (www.ahrq.gov).

No date of publication and language restrictions were applied to the search. See Appendix 1 for full strategy and terms.

Deciding which studies to include or exclude

Between them, six of the research team screened all the titles and abstracts retrieved. Two of the team (CT and TM) used Cochrane Collaboration Effective Practice Organisation of Care Review Group criteria51 and the study aims and objectives to decide if studies were relevant. We restricted our review to studies which compared CDSS-use to non-use, evaluated using designs less likely to lead to biased conclusions:

• randomised controlled trials (RCTs)

• non-randomised trials (NRCT)

• controlled before-and-after (CBA) studies

• interrupted time series (ITS) and repeated measures studies.

Data extraction

Data on study characteristics and outcomes were independently extracted by two reviewers (TM, CT) using the Cochrane Collaboration’s EPOC standard data-collection form. 52 A third reviewer (RR) was available to resolve disagreements if needed; none occurred.

We extracted data on:

1. methods: study design, location, study setting, and date of study

2. participants: number, mean age (age range), gender, inclusion criteria, exclusion criteria of patients and providers

3. interventions: intervention components, comparison, presence of characteristics known to increase effectiveness in CDSS generally

4. outcomes: main and other outcomes specified and collected, time points reported

5. study funder.

Quality assessment

Study quality was assessed using the criteria outlined in the Cochrane Handbook for Systematic Reviews of Interventions Section 8.553 and EPOC guide. 54 TM and CT assessed studies for risk of bias. Each potential source was judged as high, low, or unclear. An overall ‘Risk of bias’53 assessment was set: high – a serious bias likely to decrease certainty in the results; moderate – a risk that could plausibly raise doubts about conclusions; low – risks were unlikely to alter the results.

Data synthesis

We explored heterogeneity between CDSS systems and outcomes to determine whether meta-analysis was feasible. Heterogeneity between studies in the nature of the interventions, target groups, and outcomes measures in our initial pre-searches meant a narrative approach to synthesising findings was most appropriate. Studies were grouped and summarised by clinical similarity, for example, topics studied, type of CDSS, types of health professionals involved, patient group, outcomes reported and study design.

Intervention effects were estimated using risk difference for dichotomous data and mean differences for continuous data. We calculated 95% confidence intervals where possible. 53 Where absolute risks were not reported, these were generated from study information. Risk difference values and 95% confidence intervals were then calculated using the absolute risk values of the comparative groups.

Missing data

We contacted investigators of primary studies to verify study characteristics and obtain missing outcome data where only study abstract or results were presented in published manuscripts. Missing summary data were computed from other reported statistics wherever possible.

Investigating the effects of CDSS characteristics on outcomes

For each included study, we abstracted information on 16 system characteristics associated with effectiveness in CDSS for each study. 41 We classified each as present or absent (the predictor variable). A categorical dependent variable of either ‘success’ (CDSS better than usual care in at least one of the outcomes reported in each study) or ‘failure’ (usual care better than CDSS in one of the outcomes reported) was created for each of the 35 included studies. To evaluate whether CDSS-generated outcomes were associated with these characteristics, logistic regression models were constructed using the approach advocated by Firth for generating robust standard errors. 55 We set a 5% significance level and 95% confidence intervals for each CDSS characteristic.

‘Calibration’ interviews

We sought to access the reported experience and perceptions of key staff involved in the implementation and use of CDSS in services to sense-check our synthesis results and aid presentation. We (prior to the COVID-19 pandemic) planned a national online survey of UK NHS Chief (Nursing/AHP) Informatics Officers, but attempts at recruitment using NHSE email-based lists and forums and social media were disappointing. The COVID-19 pandemic and redeployment of key staff meant our original approach was not feasible. We eventually identified six key CDSS leaders in a range of organisations and with links to policy as well as delivery: two acute NHS Trusts (one of which was a large teaching hospital); one mixed acute and community semi-rural NHS Trust; an academic health science network lead with links to a large district general hospital-style Trust; a senior policy-level NHS lead for CDSS; a clinical academic with strategic and operational leadership role in a large urban hospital. Their implementation of CDSS varied from 20 years ago (the large teaching hospital) with most in the last three years.

Whilst two of the leaders highlighted specific system-user professional roles (such as ‘nurses’) or by clinical area (such as ‘renal’) the rest indicated that a wide multidisciplinary staff base were the intended users – including nurses and AHPs. The systems involved were intended to support a wide range of decisions; for example, disease management, detection, diagnosis, generating treatment options, forecasting/prognosis and triage.

We conducted individual virtual interviews, lasting around 40 minutes to one hour, via Zoom or telephone with our six CDSS leaders using an interview schedule developed to address the four main concepts of NPT (coherence, cognitive participation, collective action, reflexive monitoring) and contextualised for CDSS – as the ‘innovation’ or new way of working in NPT – and drawing on our main findings as prompts for discussion/sense-checking. Analysis of transcribed data and notes was abductive56 and thematic57 following the process outlined by Braun and Clarke. 58 An initial codebook was generated based on NPT constructs and sub-constructs and text read and coded. We used matrices59 with NPT constructs as columns, text from each participant as rows before comparison between participants and across columns in a version of metacoding. 60 Two of the team developed sub-themes from the initial codes of the four NPT key concepts. This was a small scale ‘pragmatic’ qualitative analysis aimed at helping understand uncertain review findings; we did not carry out inter-coder reliability checks and other qualitative-analytic techniques.

PPI

In conjunction with our PPI co-applicant (AL) we invited eight members of a single GP practice’s Patient Participation Group to a virtual meeting in early 2021. Our PPI co-applicant (AL) hosted the meeting, supported by one of the research team. Participants were sent a description of the purpose and use of CDSS several days before the meeting and asked to consider issues related to patient care and experience of consultations. The practice had a CDSS system embedded into its EHR system. Advanced practitioners, practice nurses and the practice physiotherapist accessed the EHR and CDSS system both during and outside consultations or treatment. In the meeting participants were presented with some of the uncertainties that the team felt were unaddressed by the included studies and offered the chance to ask new questions. The meeting resulted in 12 frequently asked questions (FAQs) that a patient faced with a nurse or AHP using a CDSS might ask. After the meeting, participants were asked to vote via email on the five FAQs they identified as ‘most important’.

The project team and the primary research had dedicated PPI expertise from our co-applicant and team member Alison Ledward, an experienced partner in health research and with a background in education and social work. The virtual stakeholders were from a semi-rural area of southern England with only small pockets of socio-economic deprivation. Of note is the almost complete absence of PPI and information related to diversity, inclusion and equality in the research study reports synthesised (Figure 1).

Figure 1.

Research Pathway Diagram including PRISMA flow chart of study selection. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi: 10.1136/bmj.n71.

Who are the users of evaluated CDSS?

Overwhelmingly, evaluations focused on single disciplines using CDSS compared to similar professionals making unaided decisions: nurses (n = 25, 71%) and paramedics (6%). Fewer evaluations compared CDSS supported nurses to CDSS unsupported doctors, or a multidisciplinary mix of nurses and physiotherapists in intervention and control groups.

What about the CDSS systems?

Most CDSS come as standalone computer-based systems (89%, n = 31) with less than 10% accessed via mobile technology or the web. All the CDSS were ‘knowledge-based’ (see earlier typology) and whilst single-function systems (such as disease management) were the norm, there were examples of multi-function CDSS (for example, diagnosis and management).

• Triage – five studies in emergency care61–63 and primary care. 64,65

• Disease management – five studies: managing cancer symptoms,66 oral anticoagulation,67 temperature monitoring,68 blood glucose monitoring,69 and optimising shared decision-making for self-management. 70

• Diagnosing and managing disease – four studies; diagnosing and treating diabetes;71,72 recognising and acting on clinical dehydration in acute gastroenteritis;73 and screening, automated diagnosis, and care-planning for people with obesity. 74

• Drug dosing – eight studies: mainly in blood glucose control in intensive and emergency environments75–81 and oral anticoagulant regimens in hospital cardiovascular patients. 82

• Reminder systems – three studies used CDSS for reminders on disease prevention,83 disease diagnosis,84 and disease management. 85 Three others used reminders for multiple functions: disease prevention and management,86 disease diagnosis and management,87 and disease diagnosis reminder/alert along with disease diagnosis and management. 88

Are evaluations of CDSS for nurses, midwives and AHPs biased?

With the exception of three RCTs (classed as ‘unclear’), all the studies’ risks of generating biased conclusions were ‘high’ (Table 2). The threat of bias did not diminish over time. In RCTs, NRCTs, and CBA studies, sources of bias encountered included no randomisation (13%) or unclear randomisation (27%); unclear (38%) or not done (17%) allocation; only a third of studies (n = 10) reported similar baseline measures of outcome and a single study only adjusted their analysis for any differences. Seventeen studies did not specify baseline outcome measurements in their report (57%) and whilst baseline characteristics of providers and patients were similar in around a third of the studies, they differed in another third, and in a further third only patient characteristics were reported – despite the fact that the decisions supported were made primarily by professionals. Half of the 32 randomised studies did not specify missing data and 15 of 32 (47%) studies failed to specify whether CDSS users had knowledge of how they had been allocated to intervention and control groups. In 60% of evaluations (18 studies), ‘contamination’ was likely or could not be ruled out. More positively, there was no evidence of selective reporting of outcomes. For the three evaluations based around an interrupted time series (Table 3), whilst confounding was an issue, there were no issues with selective outcome reporting, missing data or lack of clarity about when the ‘interruption’ happened.

The impact of CDSS on performance and outcomes

A broad range of outcomes are used in evaluations of CDSS: 119 outcomes, with 111 different measures. They can be grouped into five areas: (1) care processes, (2) care outcomes, (3) professional knowledge, beliefs and behaviour, (4) safety and (5) economic costs and consequences.

Do CDSS improve safety in nurse, midwife and AHP performance?

CDSS do not by default make services safer (Table 7). The complex socio-technical systemic location of CDSS and the work required to embed and sustain systems mean unintended consequences are not uncommon. Aside from simply providing ‘ineligible’ or suboptimal advice, unintended consequences can also include (1) errors in information entry and retrieval associated with a poor human-computer interface, and (2) inflexible digital systems leading to errors in coordination and communication. 100 Of course, errors also occur in unsupported practice. CDSS, however, have the potential to hardwire such errors into the socio-technical system of clinical practice, making mistakes systematic and systemic. Iatrogenic harm can also arise from ignoring the complex socio-technical system that the technology must operate in: e-iatrogenesis. 38,101 No studies focused specifically on the effects of CDSS use on patient safety in hospital care. In primary and first-response care, CDSS use by nurses is associated with lower probability of cardiovascular events in people having their diabetes managed by nurses (as opposed to CDSS-unsupported physicians) – an 11% risk difference (95% CI −18 to −4). However, serious adverse incidents and deaths in people who had fallen and were attended by CDSS-informed paramedics were unaffected.

Coherence

Coherence, or the extent to which an intervention is understood as meaningful, achievable and desirable, in an important factor in adoption. Participants emphasised a number of key differences from CDSS-unsupported care and used these to increase understanding and help encourage professional adoption:

• Reducing documentation whilst simultaneously utilising best evidence. One example was encouraging appropriate pain assessment whilst assessing pressure-ulcer risks.

• CDSS as ‘teacher’ – forcing users to engage with new information – such as guidelines – in practice.

• Making decisions ‘visible’ and facilitating appropriate action – an example being greater propensity for escalation in response to NEWS score above the Trust’s threshold for action.

• Technology making information more ‘palatable’ to younger clinicians who were perceived as having preferences for technology-delivered information.

• CDSS facilitating QI and audit and accreditation to a desired standard: something that was much harder prior to the technology being introduced. An example provided was the measurement of ‘care not done’ such as a Waterlow pressure risk assessment within four hours of admission.

• Standardisation – of care planning, collecting and organising data. Examples would be the assessment, recording and suggested actions associated with the North American Nursing Diagnostic assessment system.

The extent to which these perceived advantages were shared by everyday users of the system was unknown. On probing, it was clear that leaders recognised variability in take-up and adoption, implying not all clinicians may feel the same way. Variation was often attributed to user characteristics such as age (‘older users being reluctant’) and clinical role (‘pharmacist reviewing medications invokes less CDSS-use than initiating medications’) rather than the systems themselves or the fit between decision-work and system suitability.

Interviewees highlighted the importance of fostering shared understanding of CDSS functions, stated benefits, limitations and links to concepts such as safety, accountability and (unwarranted) variability as levers for adoption and sustainability. One system developer and evaluator emphasised the understanding required to know when not to use the CDSS:

it’s critical that the users of any of these tools understand the limitations. So, they understand when to use it when not to use it (participant’s emphasis). So, in that patient in front of them, how do they assess how safe and appropriate it is to use that tool?

Participant 4. National Policy Team on Digital Health

Collective action

In terms of the work required to bring the CDSS into use (collective action), it was clear that for all the interviewees that adjustment and pragmatic relaxing of some prerequisites (such as the overriding of alerts) – arising from reflexive monitoring – was required to sustain and encourage use. Implicit in many accounts was a recognition that systems sometimes possessed, and generated, unwanted characteristics that needed active countering. Examples included:

• fine tuning or muting alerts to reduce alert fatigue

• ensuring that over-riding a CDSS recommendation was handled ‘sensitively’ in any post-decision scrutiny – healthcare is uncertain and sometimes over-riding may be necessary or appropriate given the uniqueness of the individual to whom the CDSS is applied

• guarding against ‘garbage in and garbage out’ in both knowledge-based (poor-quality protocols) and machine learning systems (poor-quality ‘training’ data).

The extent to which CDSS developers and proponents expected work to adapt to the CDSS (as opposed to the CDSS having the flexibility to fit into work-as-done) was a strong thread in all the interviews. All the participants reported an awareness of ‘workflow’ (variously referred to as ‘context’, ‘work’, ‘environment’ or ‘practice’) as an important component in the adoption and enactment of CDSS advice.

For example, we had some clinical decision support in a discharge planning assessment … Depending on the boxes [ticked] and the information provided, we gave it some guidance and then if it was greater than a score of 10 it would create a referral to the discharge team, to help with complex discharge planning. Now staff didn’t even use the discharge planning assessment, never mind the associated decision support associated with it.

Participant 1. CNIO large metropolitan teaching hospital

A pharmacist system-developer relayed the importance of understanding the decision tasks and ‘work’ to which the CDSS was to be applied.

Or they may split the work amongst the team so that instead of having one nurse doing a drug round for all 28 patients, they might have four nurses, only giving drugs to six or seven patients. So I think when you’re looking for from my perspective I look at medication safety and I always try to understand, well, if this is happening what are the things that are contributing to that? It’s not just the individual, you know? Yes, there clearly is an individual aspect into this. But is it to do also with them finding and retrieving medicines? Is it to do with interruptions within the environment? Is it to do with the team dynamics the way they work? Is it to do with the trolley, so you know some of the equipment that they use actually isn’t conducive to finding medicines, is not arranged in the right way, so … trying to understand how they interact with the environment around them to do the task at hand.

Participant 2. CNIO large semi-urban NHS acute trust

All the accounts highlighted the need for supportive infrastructure and new ‘roles’ to enable sustained use of the systems.

What we need is more infrastructure and resources to support better implementation of these technologies, and these are tools, these are absolutely our tools, but they need people to use these tools properly. You can’t just give it to the organisations, so we need more specialists who are able to look at the data and analyse who can develop different ways of configuring the system and adapt it to what we need.

Participant 2. CNIO large semi-urban NHS acute trust

Cognitive participation

To encourage cognitive participation, interviewees emphasised formal mechanisms for engagement and increased understanding: training, education, professional committees in areas such as guidance development (rule/knowledge-based systems) and implementation. The accounts revealed a strongly multidisciplinary work environment and decisions that involved multiple professions. However, the training, the committees, and the support infrastructure were almost always mono-professional: all of the respondents’ accounts relayed work in which nurses and AHPs primarily restricted their work scope and communications to other nurses and AHPs.

All the interviewees had roles that involved responsibility for either CDSS introduction or maintaining and/or developing it in services. Aside from the aforementioned mono-professional focus, all discussed strategies that seemingly involved ‘listening’ to users, and encouraging ‘buy-in’, whilst simultaneously emphasising the advantages of ‘forcing’ CDSS use as part of work flow:

Before [CDSS], we were relying on people’s memory, how to manage, literally more than one patient every five minutes arriving at triage for example. So, it did change the way people work because it forced them to assess people in a set way and always ask the same questions.

Participant 4. National Policy Team on Digital Health

Reflexive monitoring

Various accounts relayed this as a valuable mechanism and ‘reality check’ on assumed implementation and the use of formal methods to encourage cognitive participation in clinicians. All the accounts highlighted training and training materials as ‘intended’ mechanisms for supporting initiation and use. One pharmacist’s account, though, revealed both the limitations of self-reported behaviour and the benefits of evaluating efforts:

The thing I would say though, while there’s a lot of these resources available, I don’t know if everybody is accessing it. Certainly, our junior doctors, we did a survey a little while back to try and understand how supported they were in terms of prescribing safely on the electronic system and a lot of them thought they were quite happy. They felt quite confident about using the system to prescribe safely and accurately …. but when we asked them about training types and resources available, a lot of people did not know about them or never used it or didn’t attend the classroom training.

Participant 2. CNIO large semi-urban NHS acute trust

Exogenous factors outside formal strategy were highlighted by a participant CNIO in a large metropolitan teaching hospital with almost 100% adoption of an EHR with decision support capabilities. Strategically, the historic roll-out of a trust-wide EHR system was seen as only partly successful; but the COVID pandemic had catalysed pragmatic adoption and positive feedback. One participant suggested that the CDSS and EHR adoption process was sped up by ‘5 years’. The technology was also being used by the trust to prevent – or at least question – returning to ‘business as usual’ even as the pandemic context changed and more face-to-face care and treatment were possible.

Despite informal reflexive monitoring (primarily communal and individual appraisal) figuring strongly in accounts, the relative absence of formal systematisation-evaluation in accounts was striking. Only one of the interviewees referred to formal evaluation of CDSS effects on work and outcomes: a time series evaluation of a system – they had developed – focused on a relatively rare disease in the emergency department; perhaps tellingly, they were a system developer and a clinical academic. Two CNIOs mentioned ‘informal evaluation’, mainly the collection of anecdotal evidence and committee discussion of first-hand experience of use and adoption. Despite the lack of formal evaluation, participants recognised the need for rapid and relevant evaluation – often in the form of real-time monitoring:

If 80% of nurses are clicking past and just ignoring it, what’s the clinical risk in that? Do we need to go and change practice and process and education to train and encourage people to actually read these things? And I don’t think we do enough of that yet.

Participant 1. CNIO large metropolitan teaching hospital

Where it’s really important like NEWS scoring, they also need to be monitored for compliance. It’s no point having CDSS if you’re not going to monitor if they’re effective.

Participant 3. CAHPIO mixed acute and community, semi-urban NHS trust

Other insights

As with studies in the synthesis, users’ accounts of system development and implementation did not contain references to formal or explicit theory. Phrases such as ‘listening to users’, ‘understanding the work’ and ‘alert fatigue being counter-productive’ implied some awareness of theoretically important concepts, but none of the stakeholders located their points within relevant theories of systems or implementation.

The dangers of ignoring interactional workability and unwanted artefacts of CDSS, such as generating more work for other parts of the healthcare system, were highlighted by one leader. To the extent that adoption itself can be compromised:

You do a nutritional screening tool and then the dietician says ‘stop doing it because actually we’re getting too many referrals’, that’s not the right reason to give it up; and that for me is about how you use the data to inform your future and your resourcing and your requirements and escalating risk where appropriate.

Participant 3. CAHPIO mixed acute and community, semi-urban NHS trust

Standardisation, safety and applying research evidence were the reported primary drivers for initiation, although the extent to which variability, safety and lack of evidence use were historical problems in their organisations – or indeed improved with the advent of CDSS – was not clear.

CDSS as a vehicle for improving knowledge application to practice was highlighted. Whilst two participants referred explicitly to research-based knowledge in the guidance underpinning the CDSS recommendations, for others it was codified, collective (committee) knowledge that was the basis for advice.

Where participants highlighted economic aspects of CDSS use, they generally focused on cost-effectiveness – but in the lay (costs-consequence) rather than formal economic sense (£ per QALY). One participant highlighted how economic context can hinder implementation:

You get paid a set amount for each out-patient appointment that’s face-to-face … let’s say it’s £100 for each patient, and a follow-up is £60 per patient. But there was a lot of, ‘so what is this, it’s not face-to-face physically but it’s face-to-face virtually, is that still classed as a face-to-face appointment, is there a different cost associated. A lesser cost associated with a virtual consultation than a physical one?’ So, it’s not all just about tech it’s about cost as well.

Participant 1. CNIO large metropolitan teaching hospital

Potential recipients of CDSS-mediated advice

As part of the process of calibrating the results with users (a form of stakeholder engagement) we used a voting process to derive the ‘top 5’ (see Box 1) frequently asked questions (FAQs) that might act as prompts for conversations between CDSS professional users and/or those in receipt of, or participating in, CDSS-enabled healthcare decision-making.

What’s changed since the last review of CDSS and NMAHPs?

The volume of studies focusing on CDSS has tripled since the last systematic review undertaken with a similar focus in 2009. The body of comparative research now contains a number of randomised comparisons between CDSS and unsupported practice, which is welcome. However, just less than half of the 35 included studies occur in a single site and all remain at high risk of biased conclusions. The increased number and comparative nature of studies has not led to more information to inform decision-making – whether as a clinical decision-maker or someone with responsibility for commissioning, implementing or purchasing systems in healthcare environments.

The studies included in the review suggest that CDSS have potential to improve both processes and outcomes – especially in nursing, and a more limited range of professions allied to medicine – for specific decision-focused work. However, the limited empirical evidence means that the reproducibility and systematic realisation of this potential are hard to predict. Aside from the limited and contradictory empirical research, systems and evaluations lack an explicit theoretical basis. The net effect is that the evidence base still feels essentially scattergun as a corpus of work on which to base local improvement efforts. When systems work (which, in our review, is approximately half of the time) the reasons why can only be speculated upon. But as importantly, when these systems fail (which, in our review, was approximately half of the time) the often considerable financial and opportunity costs are difficult to justify as – to use an oft quoted idiom in healthcare – lessons cannot be learned. Lists of features known to increase effectiveness of CDSS systems41,42 are only partially useful in a complex socio-technical context in which professional knowledge, work as done (rather than work as imagined by CDSS designers or implementers), interdisciplinary norms and dynamics influence technology use and decision-making itself. Varghese and colleagues in a review of CDSS suggest uncertain and dynamic clinical environments shape the significant differences in impact of CDSS. 109 Many studies – 73% in Varghese et al.’s review – are single-site studies from a single unit, practice or hospital. 109 We might exhort designers, implementers and evaluators to consider the system-level factors that might generate differences in organisational dynamics, but it is only middle-range theories (such as NPT or NASS) that allow technologists, clinicians and researchers to think systematically about the diagnostic, remedial, proactive and evaluative tweaks needed to repair, improve and optimise systems.

For complex interventions such as CDSS, the context surrounding the system is as important as the mechanisms by which they generate outcomes. Whilst not a feature of the shifting research evidence base, the context for technology use in healthcare has changed dramatically since spring 2020 and the COVID-19 pandemic. Technologies that were an exceptional component of service delivery, such as video consultation, have become part of mainstream service provision – rapidly accepted and seemingly sustained. 110 The evidence in our synthesis has not captured this shift, but it is possible that many of the barriers to implementation or positive effects may be less important in the current delivery context.

Study strengths and weaknesses

A strength of our review is our inclusive approach to searching for a broad range of studies of differing research designs to reflect the potential breadth of nursing, midwifery and AHP decision-informed care and treatment. We have also undertaken a tighter – methodologically speaking – systematic review focusing only on randomised controlled comparisons:102 the results are not substantially different. This suggests that risk of bias associated with study design is unlikely to be driving the overall research picture.

A related strength of the synthesis overall was the, limited, use of calibration interviews. These helped us unpack some of the apparent effects and both reinforced the importance of good design and implementation and served to highlight the relative absence of both in the included studies.

Including nurses, midwives and AHPs meant that differences in decision context could be examined. Specifically, we thought that certain uncertainty-related decision types (such as triage) or decision components (recording assessments) might be able to be controlled for and we would be able to look for differences between professions faced with similar uncertainties or tasks and using similar CDSS. To some extent this was realised – for example, in relation to triage. But the dearth of inter-professional studies for similar work using CDSS prevented any meaningful comparison. To the best of our knowledge this attempt to control for similar decisions in different contexts with the same system has only been undertaken once by Pope and colleagues in a qualitative case study of CDSS aimed at nurses in emergency and urgent care. 48 They found that consistency in the use of CDSS (i.e. as the designers intended) was as dependent on the implementation and resourcing to support the work required to enact the system as the technology itself. A theme that was apparent in our stakeholder accounts and also a viable hypothesis arising from the studies included in our synthesis is the lack of explicit theory used to design or evaluate reported implementation.

This was the first large-scale review of CDSS technology for both nurses and AHPs since 2009. We envisaged a larger number of studies, reflecting advances in computing power and CDSS types (e.g. machine learning). In contrast to the rapid development of machine learning and changes to tech-enabled choices in the commercial world (such as predictive analytics from Amazon, Google etc.) the research evidence for CDSS in nursing, midwifery and AHPs seemed somewhat dated and lacking behind wider societal technology adoption. Nonetheless, our findings are based on three-times the number of studies in our earlier review, suggesting further and faster growth in the evidence base may be possible.

Viewing our synthesis through the lens of NPT proved feasible and highlighted issues that may have been missed if we focused only on our results in a pragmatic way. Thinking of implementation using higher-order concepts as part of systems development and implementation planning may be more useful than trying to simply ‘build in’ system characteristics. Whilst these are associated with positive outcomes in the wider CDSS literature they were not indicative of improvement in studies of nursing, midwifery and AHPs – suggesting, it is work/roles/professional lines of demarcation that matter more than the architecture or ingredients of decisions. Interviews with system developers, adopters and leaders suggest that – whilst the language may not map onto NPT terminology exactly – concepts of coherence (understanding/sense-making), cognitive participation (building and sustaining), collective action (enacting) and reflexive monitoring (evaluation) are present in accounts.

As with any review of research evidence, our synthesis is a function of the evidence. Research into CDSS aimed at nurses, midwives and AHPs is overwhelmingly low quality, with an absence of theoretical perspectives on design or implementation that could help explain apparent effects/failure or which others could systematically build on. Empirically, studies overwhelmingly focused on nurses and ‘nursing’ decisions in a professionally demarcated way. This is unfortunate as healthcare work is essentially multi-disciplinary and team-based – especially in acute environments. The evidence on the economics of CDSS aimed at nurses and AHPs is patchy and scarce. Our synthesis is unable to adequately answer policy questions of value-for-money associated with systems.

A key working assumption behind the review was that the comparative research evidence base would have matured sufficiently in quantity and quality to produce a substantive series of ‘effects’ and ‘differences’ between systems with less uncertainty that our last review. We felt describing these effects (and associated uncertainty) in a rigorous and systematic way, and having sufficient volume of similar research – amenable to statistical meta-analysis – would offer useful insights for decision-makers faced with commissioning and choosing CDSS. This assumption did not hold up.

We searched only for comparative research studies and failed to include study designs that – whilst less suited to statistical synthesis – might have yielded a richer insight into the implementation processes and mediating effects, mechanisms and contexts that were clearly influential in explaining the mixed picture of uncertain results. As the evidence base matures and the use of theory and well-conducted process evaluations increases, alternatives to traditional systematic review approaches – for example, realist synthesis – may provide more insight and actionable recommendations for implementation, particularly as a basis for nested evaluation alongside CDSS roll-outs.

Beyond reductionist ideas of ‘satisfaction’ with systems, a sophisticated and nuanced view of user perspective was largely absent in studies. Studies since the 1980s have highlighted the complex way in which people interact and socially create meaning, satisfaction, justifications and reinforcement for their use of technology generally and CDSS specifically. 48,103 This is not a trivial omission; how people feel about systems influences their later behaviours and the consequent effects of systems. Artificially simplifying user perspectives on CDSS use to abstract concepts such as ‘satisfaction’ has limited utility for those seeking targets for optimising and evaluating systems.

The lack of common reporting standards for CDSS made comparison between studies to assess similarities difficult and we cannot rule out the possibility that there was more homogeneity between systems and therefore greater potential for statistical synthesis. In the absence of detailed, structured, reporting we were conservative in our estimates. Out of pragmatic necessity, we abstracted descriptions and system features and effects to higher-order concepts such as ‘knowledge-based’ (features) or ‘care processes’ (outcomes). This meant discarding potentially ‘richer’ information. Similarly, studies rarely addressed issues of context and exogenous factors shaping use, implementation and effects. This meant issues of ‘context’ were hard to unpack and incorporate into our analysis. Other kinds of reviews – for example, realist syntheses – may have produced more information on what works for whom, when and in what circumstances.

A final weakness relates to our limited range of interviews with what can be termed ‘stakeholders’ in CDSS. Our original plans had to be changed as a result of lack of engagement arising from the redeployment of staff (and changed priorities) arising from the COVID-19 pandemic. Consequently, the numbers, extent and richness of the data were less than we would have liked. Nonetheless, the limited work yielded some valuable insights and the approach was promising and we will seek to maximise this technique in future syntheses generally and seek approvals and funds to try again with the communities of practice that surround CDSS design, adoption and implementation.

What could we have done differently?

We started from an optimistic estimate of fit-for-purpose research evidence and sufficient quality for inclusion. Our initial pre-synthesis exploratory work suggested substantially more evaluations than when we undertook our last similar review in the 1990s. This sanguine position was justified on the grounds that we expected the numbers of evaluations of CDSS to mirror wider technological and digital expansion in healthcare; for example, EHRs. The growth – at least rhetorically – of technology in the delivery of nursing, midwifery and AHP decision-informed care has not led to similar growth in evaluations.

Our planned analysis was SECONDARY|primary; a synthesis of a rich and detailed extensive pool of well-conducted comparative evaluations that might even merit meta-analysis of common systems, followed by smaller-scale qualitative interviews with those with a stake in decision support. On reflection, a secondary|PRIMARY balance may have been more productive. Whilst the pool of experimental/quasi-experimental research has more than tripled since the last comparable review in 2009 it is still very limited and heterogeneous and still unsuited to statistical synthesis and aggregation. Our primary data collection was hampered by the effects of the COVID-19 pandemic on accessing services and users of CDSS in healthcare. We struggled to recruit, arrange and conduct interviews remotely with the numbers of CDSS implementers, service supporters and service-users. But our interviews and virtual meetings suggest that there is still much to be gained from understanding the experiences and perceptions of those who develop, implement, use and receive/take part in CDSS-mediated professional advice/choices. The uncertainty in the quantitative evaluations synthesised, the depth of perceptions and experience in the limited calibration undertaken, and the role that theory – and theory revised in the light of empirical and perceptual description – could play in improving systems and their implementation make the need for well-conducted, theory-informed, primary qualitative research more urgent.

Contributions of authors

Carl Thompson (https://orcid.org/0000-0002-9369-1204) drafted this synthesis and was responsible for the initial research idea and bid.

Teumzghi Mebrahtu (https://orcid.org/0000-0003-4821-2304) and Carl Thompson (https://orcid.org/0000-0002-9369-1204) were responsible for the review design, extraction and analysis; all team members at some point helped screen titles and abstracts, informed our analysis, and helped draft work and articles.

Sarah Skyrme (https://orcid.org/0000-0002-6173-0117) and Alison Ledward carried out the virtual meeting of PPG members and production of patient FAQs on CDSS.

Sarah Skyrme (https://orcid.org/0000-0002-6173-0117) also interviewed our system experts.

Ethical approval for the qualitative components of the project was sought and granted by the University of Leeds Faculty of Medicine and Health Ethics Committee HREC 20-002 - Effects of computerised clinical decision support systems (CDSS) on nursing and Allied Health Professional performance and patient outcomes: A systematic review and user contextualisation.

Data-sharing statement

All data requests should be submitted to the corresponding author for consideration. Access to available anonymised data may be granted following review.

Ethics statement

Ethical approval for the study was granted by the School of Healthcare Research Ethics Committee, University of Leeds, on 20 October 2020: approval reference HREC 20-002.

This article

The contractual start date for this research was in October 2019. This article began editorial review in October 2021 and was accepted for publication in July 2022. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The Health and Social Care Delivery Research editors and publisher have tried to ensure the accuracy of the authors’ article and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this article.

Disclaimer

The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care.