Electronic prescribing systems in hospitals to improve medication safety: a multimethods research programme

Article history

The research reported in this issue of the journal was funded by PGfAR as project number RP-PG-1209-10099. The contractual start date was in November 2011. The final report began editorial review in February 2018 and was accepted for publication in September 2020. As the funder, the PGfAR programme agreed the research questions and study designs in advance with the investigators. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The PGfAR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Copyright statement

Copyright © 2022 Sheikh et al. This work was produced by Sheikh et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This is an Open Access publication distributed under the terms of the Creative Commons Attribution CC BY 4.0 licence, which permits unrestricted use, distribution, reproduction and adaption in any medium and for any purpose provided that it is properly attributed. See: https://creativecommons.org/licenses/by/4.0/. For attribution the title, original author(s), the publication source – NIHR Journals Library, and the DOI of the publication must be cited.

2022 Sheikh et al.

Summary of changes to the research programme

The main change from the proposal was that we were unable to employ the stepped-wedge design that was originally planned because of several implementation delays within the hospital sites; instead, we settled on a more pragmatic pre- and post-implementation design. The delays also impaired our ability to assess the longer-term consequences of ePrescribing systems because we were unable to follow developments up over sufficient time periods that were required to understand the longer-term consequences of ePrescribing systems.

One of our original aims was to establish whether or not there is a difference between non-alerting computerised physician order entry (CPOE) systems and more advanced alerting clinical decision support (CDS) systems. However, this proved impossible because both systems implemented some level of CDS within their ePrescribing system. However, given that we measured a fixed set of medication errors, we were able to highlight the different levels of CDS and the impact that this may have had on medication error rates.

Although the original aim of the programme was primarily directed at supporting professionals to implement ePrescribing systems in hospitals, patient and public involvement (PPI) members argued for directly including patients’ views on hospital ePrescribing systems to inform the programme. Therefore, a patient study was introduced into work package (WP)1 that was guided by patient engagement work.

For the health economic work in WP3, an additional step to calculating health gain was used that was not outlined in the original proposal. The health gain was calculated [in quality-adjusted life-years (QALYs)] directly for particular errors chosen by a nominal group method to represent the greatest-impact errors in terms of harm and frequency.

The health economic work was also adapted because we were not able to obtain cost data from the study sites, as hospitals were generally unwilling to provide such information. This resulted in the use of the ‘headroom’ approach, which asks what the maximum price a decision-maker should be willing to pay for a technology on the basis of overall expected benefits. This approach is desirable in this context given that it allows decision-makers to compare the prices that they face with the evidence on effectiveness to make an informed decision.

Development

Prescribing errors are common, costly and often result in considerable harm to patients. 1–3 Work from a handful of predominantly US-based centres of excellence suggests that the risk of patients being inadvertently harmed by erroneous prescribing may be greatly reduced by the introduction of electronic prescribing (ePrescribing) systems. 4–6 Their development, procurement and implementation is, therefore, currently being pursued internationally. 7

Current ePrescribing applications vary significantly in functionality and sophistication. These, in essence, range from basic computerised data entry applications (also sometimes known as CPOE or non-alerting systems) to more sophisticated systems offering advanced CDS systems functionality that generate prescribing and medication administration alerts. 7 CDS systems are active knowledge systems that use real-time patient-specific information to generate individualised prescribing advice. These more sophisticated systems tend to be integrated with existing local electronic health records (EHRs) to produce alerts (e.g. warnings and reminders) tailored to the individual case at hand, for example a prescribing alert suggesting a dose change when a patient’s renal function is deteriorating.

ePrescribing systems are now well established in UK primary care. 7,8 Within secondary care, however, the implementation of ePrescribing systems has been much slower, this in part reflecting the fact that hospital records have until recently tended to be paper based. The widespread adoption of ePrescribing systems had until recently been achieved in very few UK academic hospitals (e.g. the University Hospitals Birmingham NHS Foundation Trust). More recently, there has been an acceleration in the adoption of ePrescribing systems in hospitals, but implementation remains largely unco-ordinated nationally.

Available systems range from the home-grown extensively tailored applications that have been developed over a number of years (as is the case in University Hospitals Birmingham NHS Foundation Trust where the system has been developed for over ≈ 15 years) to the expanding array of ‘commercial off-the-shelf’ (COTS) systems. Although home-grown systems tend to be more easily adopted by hospitals (this reflects the strong sense of local clinical ownership), extensive system customisation over time means that these are not easily transferable to other sites. In addition, very few hospitals have the necessary expertise or budgets to develop their own systems in-house. Moreover, over time, home-grown solutions become very expensive to maintain; as a result, adopters increasingly choose packaged solutions for reasons of cost and dependability. Trusts, therefore, need to decide whether to persist with paper-based prescribing or to move to electronic prescribing. If they choose the latter, as indeed many are, a key decision is to implement a stand-alone basic ePrescribing system or a more integrated packaged solution that is part of a full EHR solution, which often offer additional CDS functionality. This decision is likely to be shaped by a range of factors, including the ease of implementation, training implications, the time and cost investment necessary, and the anticipated benefits, particularly in relation to improving prescribing safety.

Reducing the risk of iatrogenic harm is now firmly established as a priority area throughout much of the economically developed world, including the UK. Given that medication-related harm is associated with a substantial proportion of this potentially avoidable morbidity and mortality, there is now a considerable focus on improving prescribing safety. 9,10

Although real-time prescribing support and electronic alerts/prompts have under some circumstances demonstrated benefit in improving clinicians’ prescribing behaviour and/or reducing error rates, there remains a small, but nonetheless important, risk that the introduction of these systems may also have serious adverse impacts on patient outcomes. 11–15 They can, thus, introduce new areas of clinical risk and unexpected threats to patient safety. Koppel et al. ,13 for example, reported how fragmented CPOE displays prevented a coherent view of patients’ medication and how the separation of functions contributed to accidental double dosing. Another study11 has highlighted how systems’ implementation-generated errors were associated with the process of entering and retrieving information, manifesting as errors in communication and co-ordination processes. Similarly, we have found that many CDS systems can produce clinically spurious or irrelevant alerts that frustrate end users and result in these alerts commonly being over-ridden or ignored. 12 More recent work has highlighted the potentially serious treatment delays that can inadvertently be associated with the introduction of ‘hard stops’ (designed to prevent over-riding of alerts) in the context of potentially serious prescribing errors. 14,15

ePrescribing systems are, therefore, not without associated risks. Unintended consequences of health information technologies (HITs) are strongly linked to poor implementation and adoption strategies. This has been demonstrated not only in the UK, but also in many other parts of the world. 16,17 There is then a need to understand ways of minimising/mitigating the risks associated with the introduction of potentially transformative HITs, such as ePrescribing systems.

We set out to undertake a mixed-methods programme of research involving NHS trusts that were in the process of putting in new systems to study the implementation, adoption, effectiveness and cost-effectiveness of these systems. We also sought to study trusts that had already implemented ePrescribing systems to study more embedded use. Our research team comprised national and international medical and pharmacy experts in ePrescribing/HITs, social scientists, organisational management experts, health economists and PPI experts. Our key aims were to describe the procurement, implementation, adoption and maintenance of basic (i.e. CPOE functionality) and more advanced (i.e. also including CDS) ePrescribing systems; to estimate their effectiveness and cost-effectiveness; and to develop best practice implications for practice and a toolkit for their successful integration into NHS hospitals.

The research was organised into four complementary WPs, namely:

• WP1 – procurement, implementation, adoption and connectivity

• WP2 – assessing impact on prescribing safety

• WP3 – health economics and a value of investment analysis

• WP4 – integration across WPs to develop a toolkit for the NHS incorporating implications for practice.

These WPs are described in detail in the following sections. Figure 1 illustrates the inter-relationships between the WPs.

FIGURE 1.

Overview of the WP relationships. Reproduced with permission, personal communication, 2021.

Reproduced with permission, personal communication, 2021.

Background

The implementation of any new HIT within a health-care system is a complex phenomenon. Previous national schemes, such as the National Programme for Information Technology, demonstrated the considerable complexities of the large-scale adoption of enterprise-wide systems. Some of these difficulties lay in having a sufficient information technology (IT) infrastructure in terms of hardware and wireless connectivity; however, there are many other challenges in implementing systems in a health-care organisation. Some of these challenges are common to all IT system implementations, whereas others are more specific to the hospital environment. Organisations need to be ready in terms of both their infrastructure and the people working within the system (e.g. health-care practitioners and managers) for the successful implementation and adoption of systems. The adoption of an ePrescribing system, therefore, needs to recognise the potential for change not only in the technical aspects of the organisational structure and processes, but also in the complex interactions between the people and the new technology in the workplace.

Good communication and information-sharing within a health-care organisation and across organisational boundaries are critical for continuity of patient care. The Department of Health and Social Care (DHSC) document Discharge from Hospital: Pathway, Process and Practice18 called for effective communication between primary, secondary and social care. Research has indicated the impact of communication breakdown on prescribing and on the monitoring of patients following discharge from hospital,19 and a DHSC-commissioned report20 highlighted how the use of ePrescribing systems in hospitals can give staff quicker access to more reliable information and, thus, has the potential to improve patient safety. General practitioners (GPs) may also require timely information from hospitals to support safer care after a patient has been discharged from hospital. 21 This calls for an extension of complex information systems across hospital settings and beyond.

The consequences of adopting new information systems depend on multiple factors. As well as the design and configuration of the computer system, the implementation strategy, training, systems’ maintenance, work reorganisation and development of new work practices are all influential in systems’ deployment, consequences and optimisation. Given the complex interactions between these various sociotechnical factors (the changing inter-relationships between technology and human/organisational factors), introducing new systems into hospitals may result in both positive and negative consequences. For example, in a hospital the workload of individual staff members, team structures, workflows, communication processes (within and beyond the organisation) and patients’ safety and care outcomes may all be influenced.

We conducted longitudinal case studies to explore the evolution of system implementation and use over time, as well as to identify factors determining the short- and long-term impacts of, and transformations brought about by, these systems. The factors explored included the organisational learning achieved, the identifiable impacts on everyday working practices, the issues relating to communication across organisational boundaries, and the safety and risk arising from systems’ utilisation.

This evolutionary perspective also aimed to throw light on the longer-term processes of capacity development, highlighting the extent to which hospitals and trusts have managed to acquire and retain implementation experience and change management capacity (sometimes described as organisational readiness or ‘maturity’), and on an additional dimension that has hitherto received inadequate attention: the extent of knowledge flow, shared understanding and institutional links between suppliers and adopters.

In WP1, we also sought a better understanding of how different hospital ePrescribing systems influence communication and local connectivity both within the hospital and in the wider health community. Here, the focus was particularly on the perspectives of those responsible for prescribing and administering medication to investigate the consequences of the hospital ePrescribing systems for information flow, as well as on patients and their perspectives. 22 We also sought to understand the implications for the prescribing education of health-care professionals. 23

A national digital maturity assessment has shown that NHS hospitals have a very different rate of adoption of inpatient ePrescribing versus discharge ePrescribing (digital maturity) systems. There was only one supplier (JAC Computer Services Limited, Essex, UK) with more than 10 ePrescribing system roll outs. There was also a growing number of Cerner Millennium (Cerner Limited, London UK) implementations in hospitals. More than 10 other systems were also being procured and implemented by NHS England’s hospitals. Therefore, we also sought to identify specific lessons surrounding the selection and utilisation of different kinds of systems and functionalities in different work settings. To achieve this, we examined the product life cycle of the systems, in which packages emerge from development settings and are further refined through implementation in different settings.

Research aims

Our aims were to investigate the shorter- and longer-term impacts of introducing ePrescribing systems into NHS hospitals by:

• describing the planned and actual implementation and adoption strategies

• understanding key stakeholders’ perspectives by exploring individuals’ expectations, experiences, practices and perceived needs over time.

During the course of this work, it became clear that addressing the main aims (as described above) required us to investigate how these issues are patterned by the broader context, including patients’ perspectives on ePrescribing, the level of ePrescribing maturity within the NHS and the evolving market of COTS ePrescribing applications and its effects on their uptake and implementation. Therefore, two additional research aims were articulated, namely to investigate the:

• evolving market of COTS ePrescribing applications and its effects on their uptake and implementation in English hospitals (which hereafter we refer to as the ‘supplier study’)

• significance of ePrescribing from patients’ perspectives (which hereafter we refer to as the ‘patient study’).

Methods of data collection and analysis

Parts of this text have been reproduced with permission from Mozaffar H, Williams R, Cresswell K, Morrison Z, Bates DW, Sheikh A, The evolution of the market for commercial computerized physician order entry and computerized decision support systems for prescribing, J Am Med Inform Assoc, 2016 vol. 23, pp. 349–55,24 by permission of Oxford University Press.

Key findings

Parts of this text have been reproduced with permission from Mozaffar et al. 46 This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. The text below includes minor additions and formatting changes to the original text.

Parts of this text have also been reproduced with permission form Mozaffar et al. 47 with permission from BMJ Publishing Group Ltd.

The key deliverables for WP1 were to:

• develop a typology of ePrescribing systems

• provide contextual data for the provisional ePrescribing toolkit

• refine the system life-cycle model for ePrescribing systems

• develop a typology of implementation strategies.

Early and widespread dissemination of the findings of this WP (together with WP4) was achieved by a strategically managed programme of publication, including 32 articles in peer-refereed journals and two book chapters. These papers are listed in the Acknowledgements. Lay summaries of these outputs are detailed in Appendix 27. These journal articles, together with the information presented in the ePrescribing toolkit, contribute towards achieving WP1 deliverables. Table 1 summarises the project achievements in relation to these key deliverables.

In the remainder of this section, we organise our findings around six central themes that emerged from our data analyses. The first four themes focus on the implementation site, the hospital: (1) initiation and procurement, (2) implementation process, (3) organisational consequences and (4) medium- to long-term benefits realisation and system optimisation. The remaining two themes refer to findings taking different perspectives: (5) system knowledge exchange mechanisms in the HIT sector and (6) patient view of ePrescribing use.

Conclusions

Based on our findings, we offer a set of key lessons on the different aspects affected by the implementation and use of ePrescribing systems. This is followed by concluding remarks on the work of WP1 and a description of inter-relation with other parts of the programme.

Background

Medications are the most common therapeutic intervention in health care, and over half a million prescriptions are generated in the NHS on a daily basis. Prescribing is a potentially risky process and, although some adverse drug events are unavoidable (e.g. new drug allergies), many adverse events (AEs) are preventable. Preventable medication-related harms are, therefore, classified as medication errors. All health-care practitioners who prescribe have an obligation to be competent to undertake the task, but education and training cannot eliminate all errors. Various strategies have been introduced to tackle medication errors (of which prescriber errors are a subset), one of which is the adoption of ePrescribing systems with CPOE systems plus or minus CDS functionality. The key objective of this WP was to determine whether or not prescribing error rates changed as a result of the implementation of ePrescribing systems.

To evaluate the effectiveness of ePrescribing systems, we focused on COTS systems that are more likely to be implemented in NHS hospitals than ‘bespoke’ systems that have been developed in academic centres. One of the key lessons learnt from previous medication error studies, regardless of the method of prescribing, is that untargeted prescription review can lead to reporting of very large numbers of errors that have minimal clinical impact. Such errors have been shown to be a poor surrogate of more serious errors. 68 Efforts, therefore, need to be directed towards evaluations focusing on errors that are most likely to lead to serious AEs. We, therefore, focused on studying prescribing indicators that are judged by experts to have a significant risk of translating into actual patient harm.

Research aims

The overarching aim of this WP was to estimate the effectiveness and relative effectiveness of different ePrescribing systems in reducing the risk of serious prescribing errors.

We sought to:

• establish whether or not the introduction of ePrescribing results in a reduction in clinically important errors

• assess whether or not the more advanced ‘alerting’ CDS-based systems are more effective than simpler CPOE-based (‘non-alerting’) systems.

Methods

Parts of this text have been reproduced with permission from Thomas SK, McDowell SE, Hodson J, Nwulu U, Howard RL, Avery AJ, et al. 69 Developing consensus on hospital prescribing indicators of potential harms amenable to decision support. Br J Clin Pharmacol 2013, vol. 76, pp. 797–809. © 2013 The Authors. British Journal of Clinical Pharmacology © 2013 The British Pharmacological Society.

Results

The eDelphi study identified 80 prescribing errors as being high or extreme risk by consensus. The most common drugs and drug classes named in the indicators were antibiotics (n = 13), antidepressants (n = 8), non-steroidal anti-inflammatory drugs (NSAIDs) (n = 6), opioid analgesics (n = 6), antiplatelets (n = 5), methotrexate (n = 4), low-molecular-weight heparins (n = 4) and benzodiazepine (and like) drugs (n = 4) (see Appendix 5). These prescribing indicators were translated into the IMPACT tool for the collection of data relating to high-risk prescribing errors, in both paper-based and ePrescribing processes. The full details of the consensus methods and the results of prescribing indicators have been published in full, and can be seen in Acknowledgements, Publications.

Drug charts from a total of 2422 patients were reviewed at the three sites: 1244 before and 1178 after the implementation of system A (integrated CPOE and CDS) at sites J and K, and system B (stand-alone CPOE system with low-level CDS switched on) at site S. These drug charts contained a total of 28,526 medication orders: 14,371 before and 14,155 after implementation. Following implementation, the proportion of medication orders that were technically incomplete, or that showed inappropriate dose units or route of delivery, was reduced to zero at all three sites (from 61.80%, 5.06% and 4.94% at sites J, K and S, respectively). The very large number of incomplete orders at site J on paper resulted because of the omission of a bleep number (contact details for their air pager) for the prescriber, which was regarded as an incomplete order according to this hospital’s protocol; this was not the case at sites K and S. In addition, some improvement was noted in the documentation of allergy status and (especially) of patient weight at sites K and S, but not at site J in which the proportions of patients with documented allergy status and recorded weight fell slightly (see Appendix 8).

Discussion

The outcome for each key deliverable is shown in Table 3.

The implementation of the ePrescribing systems under investigation was associated with a reduction in specific, serious and commonly occurring prescribing errors where CDS was available. Given the before–after design employed in this service-level evaluation, we are not in a position to make clear causal inferences; however, the size of the effects seen and the fact that there were no other major transformations in the hospitals over this time period suggests that the introduction of ePrescribing systems is likely to have been responsible for these effects. Ideally, future studies should select a cluster design with contemporaneous controls, such as the step-wedge cluster approach that we had originally planned.

As expected, there were major improvements in procedural errors at all sites, such as improved legibility, completeness, and allergy and weight reporting. There was also an elimination of documentation of dose unit errors. This is typical of improvements seen in any CPOE implementation regardless of whether or not there is active CDS.

Although we initially set out to examine differences in effectiveness between more advanced ‘alerting’ CDS-based systems and simpler ‘non-alerting’ CPOE-based systems, this dichotomisation of system types did not strictly hold true. In fact, when we tested our prescribing indicators on test patients in the two different systems, there was a varying level of CDS in all of the sites. However, the presence of any active CDS did lead to a reduction in prescribing errors. In particular, where sites implemented a ‘restrictive’ level of CDS, associated errors were eliminated, regardless of the error type. Each site had stated that they were concerned about alert fatigue and, therefore, implemented minimal levels of CDS (approximately only 25% of the opportunities for errors being studied). We identified five different levels of CDS and, although two sites implemented the same system, we found that there were differences in local configuration that led to differences in CDS alerting and obvious differences in error rates. The culture of the organisation and, therefore, how they configure their ePrescribing systems is vitally important,73 because implementing a system that has led to improved safety elsewhere does not in itself guarantee a reduction in error rates owing to the configuration differences. The findings from our study suggest that where it is possible to identify specific institutional problems leading to prescribing errors (e.g. common mistakes or inappropriate drug combinations), targeted CDS could prevent these errors from occurring. 74

The errors that were measured were chosen because they were potentially amenable to CDS if implemented within the scope of current systems. However, there are other significant errors or error types for which CDS might improve safety that are dependent on information from other systems (e.g. electronic patient noting of comorbidities or pathology test results). In our research, the sites focused on getting ePrescribing installed and widely used as part of the immediate implementation strategy; therefore, additional integration of other data had not yet been achieved or was not possible. The research team were careful not to influence the decision-making at hospitals and have, therefore, reported on real-world ePrescribing implementation. Additional system integration and optimisation may lead to further improvement in quality and safety, and the hospitals can continue to measure these prescribing errors in this context.

Computerised provider order entry and CDS have been associated with new error types. Identifying these errors is important in the ongoing monitoring and optimisation of systems to minimise the risks of the technology to patient safety. However, in view of our approach to capture data on the occurrence of defined high-risk prescribing errors, investigating the specifics of any new errors was outside the scope of the study.

Implications

Previous studies75,76 have demonstrated that alert fatigue is problematic in practice and that alert over-rides can be inappropriate. This study has shown that varying levels of CDS have different impacts on medication error rates. Previous work that has focused on CDS acceptance and human factors highlighted the differences in acceptance of CDS alerts dependent on the level of acknowledgement required,77 and Phansalkar et al. 78 highlighted the importance of having different levels of alerts relating to the severity of the error. Rather than switching on all possible alerts, sites implemented CDS selectively, focusing on the most serious drug interactions and allergies, which meant that 75% of the targeted errors had no CDS. The errors that were chosen to be on at a ‘restrictive’ level of CDS resulted in a complete elimination of those error types occurring. This strategy may have reduced the number of inappropriate over-rides that the implementers were keen to avoid.

The indicators (and the IMPACT tool developed as a result) used in this study were agreed by an expert panel to be commonly occurring and to be serious errors. The top 25 indicators accounted for 81% of the error opportunities, which suggests that these were indeed the appropriate focus. In addition, using this audit tool provided an objective view of the potential for serious errors in hospitals, as incident-reporting data (including near misses) in hospitals can be unreliable. 79

Conclusions

In this first real-world evaluation of commercial ePrescribing systems on clinically important prescribing errors across three hospitals in NHS England, prescribing error rates post implementation fell significantly, suggesting that CPOE systems when implemented with even relatively low levels of CDS can improve medication safety. We have been unable to specifically determine whether or not increasing the number of CDS warnings has a differing impact on prescribing error rates, but can propose that higher levels of CDS alerting (e.g. restrictive alerts) can eliminate or reduce clinically meaningful errors.

Hospitals considering ePrescribing may find it helpful to consider how their system will interface with existing infrastructure (e.g. laboratory records), as this may have an impact on the types of error that can be targeted. In addition, there may be opportunities arising from the secondary uses of ePrescribing data to improve medication processes to continuously improve patient safety. The implementation of ePrescribing CPOE systems with CDS can reduce medication errors, but with further system optimisation, integration and use of secondary data there is an even greater potential to target adverse drug events and improve patient safety.

Inter-relationship with other parts of the programme

This WP has provided important quantitative data for the health economic WP (WP3). It has further provided content for the toolkit (WP4) by providing access to a bespoke data collection tool that has been made available to other hospitals implementing systems.

Background

Decision-makers in hospitals need to decide whether or not to invest in new medical technologies. However, the health-care system has limited resources, requiring decision-makers to operate in a framework that maximises the returns produced by the limited inputs. The methods for the economic evaluation of health technologies are well developed and are incorporated into assessments by bodies, such as the National Institute for Health and Care Excellence. However, the evaluation of service delivery interventions, such as ePrescribing systems, is more complicated. Unlike for standard health technology assessment for which the effects of the intervention on the end points relevant to economic evaluation, such as mortality or QALYs, are (or at least can be) measured directly, the evaluation of service delivery interventions requires us to extrapolate from more ‘upstream’ outcomes, such as clinical errors. This is because an important signal (e.g. improved quality of life or fewer deaths because of the ePrescribing system) is likely to be lost in the noise. Consider a scenario in which preventable adverse drug events could be reduced by a worthwhile 25%, a large effect. This would reduce the incidence of these events from approximately 1% to 0.75% in the patient population. Such an absolute reduction of 0.25 percentage points is unlikely to show up in a comparative evaluative study. The logic behind this ‘inconvenient truth’, that worthwhile effects from generic service improvements are likely to be lost in the noise (type II error), is explicated in more detail elsewhere. 80 The cost-effectiveness of large-scale service interventions is further complicated in situations in which there are no fixed costs, but in which costs are locally negotiated as, for example, in the case of an ambulance service or an IT system. Thus, we have to deal with two uncertain aspects of cost-effectiveness: the cost and the effectiveness. We deal with our approach to these challenges below.

Research aims

It is unlikely that an ePrescribing system will have a clinically significant effect on any one patient, on average. However, when measured and aggregated across an entire hospital, the benefits may outweigh the health service costs. This WP, therefore, had the following aims:

• to further develop the methodology of service delivery intervention evaluation and to develop the methods of evaluating ePrescribing systems

• to estimate cost-effectiveness parameters based on multiple data sources

• to estimate the expected net benefits of an ePrescribing system to inform decision-makers.

Overall approach

The model

The model we chose for this evaluation, which links ePrescribing to costs and QALYs, is conceptually simple. The benefits of the system are mediated by medication errors and consequently adverse drug events. A reduction in adverse drug events both reduces patient costs and improves quality of life. The (direct) costs of the ePrescribing system are a fixed capital cost (amortised over the useful life of the system), along with annual maintenance and training costs. The model structure is shown in Figure 2.

FIGURE 2.

The decision model. a, Assumes all types of error affected equally. b, Assumes proportionality across errors and AEs, and across category of AEs. c, Distributions of quality-of-life loss and life expectancy/duration across category. This figure is reproduced with permission from Yao et al. 85 This is an Open Access article distributed in accordance with the Creative Commons Attribution-Noncommercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt and build upon this work, for non-commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by-nc/3.0/. The figure includes minor additions and formatting changes to the original figure.

This figure is based on Yao 85 This is an Open Access article distributed in accordance with the Creative Commons Attribution-Noncommercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt and build upon this work, for non-commercial use, provided the original work is properly cited. See https://creativecommons.org/licenses/by-nc/3.0/. The figure includes minor additions and formatting changes to the original figure.

It might be argued that this model is not sufficiently complex to take into account the necessary contextual variables to understand the impact of an ePrescribing system. Health-care systems are inherently complex systems. They comprise many interacting causal processes across multiple levels. However, it is important to distinguish between the data used to make inferences about the underlying processes and the processes themselves. These processes are constant and stable across different health-care contexts. The effect of an ePrescribing system is targeted towards the same phenomena across all health-care contexts, namely medication errors. Differential use of the system between hospitals may result in heterogeneous treatment effects, which may be reflected in the statistical modelling, but the underlying causal phenomena are the same. Further analysis, whether qualitative or quantitative, would be required to investigate the heterogeneity of treatment effects.

The simple model of ePrescribing effects implies the following estimator of the incremental effect of ePrescribing on patient costs and QALYs arising from averting adverse drug events. The estimated relative risk (RR), which we describe below, serves as an input into the larger model of the effects of an ePrescribing system on patient QALYs in the following way:

⁽¹⁾ ( 1 – RR ) × baseline risk of ADE × incremental effect of ADE on QALYs.

The effect on patient costs attributable to reduced adverse drug events (ADEs) is similarly calculated. Further information is, therefore, required to estimate the baseline risk of an ADE and the incremental effect of an ADE on patient costs and QALYs. The health economic evaluation method we describe here takes a health system perspective, including costs incurred by the hospital only and not losses accruing to the patient owing to, for example, lost employment. Although a societal perspective would be preferred, to the best of our knowledge, there is currently no research into the effects of preventable AEs on outcomes occurring outside hospital. This is an important future area of research. We note some key assumptions: first, that there is no other factor that causes a change in the risk that a medication error causes harm and, second, that the implementation of ePrescribing is independent of patient case-mix and risk of medication error.

Estimating health benefits in the form of quality-adjusted life-years

Estimating cost

Health economic analysis: headroom estimation (threshold analysis)

Conclusions

The outcome for each key deliverable is shown in Table 5.

In this section, we have estimated the RR of a patient experiencing a medication error with an ePrescribing system compared with that of a paper-based prescribing system. The RR is a key parameter for the larger decision model. The next stages incorporate the estimated RR with other sources of data. The RRs that we have estimated are all smaller in magnitude than that previously estimated. A recent meta-analysis comparing CPOE with paper-based prescribing estimated a pooled RR of 0.47 [95% confidence interval (CI) 0.31 to 0.71] compared with the estimates presented here of between 0.81 and 0.97. 83 However, our estimates were supported by both the prior and the data. The structured discussion that took place during the expert elicitation workshop suggested that the meta-analysis estimate may be too large for a number of reasons: the baseline rate of medication errors may have been higher in the predominantly US-based studies included in the review; the ePrescribing system may cause errors as well as prevent them; and the effectiveness of the systems may be lower in the UK. The posterior distribution differed little from the prior distribution, which suggests that the data did not contain much information to inform the estimate compared with the prior. As far as we are aware, this is the first study to use expert elicitation and Bayesian methods to estimate the effects of ePrescribing systems in hospitals.

This WP has provided an estimate of the maximum price that a decision-maker is likely to be willing to pay for an ePrescribing system at three study sites on the basis of the data collected here, previous studies and expert opinion. The costs of an ePrescribing system are unknown and will vary by hospital site on the basis of supplier, required features of the system and any negotiation. However, the model shows that systems would need to cost less than our reference system or be more effective to justify their costs. We utilised a cost per QALY threshold of £20,000 per QALY; however, guidance suggests an upper limit of £30,000 per QALY. Local decision-makers may wish to select an alternative threshold that may be influenced by other budgetary concerns.

Inter-relationship with other parts of the programme

This WP aimed to integrate and synthesise the findings from the other WPs and external findings to support decision-makers. The findings from this WP are not intended to be proscriptive, but to be informative. The key question is whether or not the evidence supports implementation given the costs and the societal willingness to pay for health benefits. The evidence reported by the other WPs cannot be used to directly address this question, hence the requirement for this further step. This WP has not addressed the question of the optimal implementation of an ePrescribing system or the factors explaining treatment effect heterogeneity. The evidence from the other WPs support these questions of implementation.

Background

This WP integrates findings from the qualitative and quantitative strands of the programme to identify the overall transformative contributions of ePrescribing systems in terms of procurement, implementation, adoption and use (see Work package 1), as well as of impact and costs (see Work package 2 and Work package 3). 112 Moreover, it integrates our empirical findings with contextual and existing research to develop an analytical framework within which implementation strategies may be evaluated in terms of their impact and transferability to make recommendations for future deployments and research efforts in this area.

Research aims

1. To provide contextual information for WPs 1–3.

2. To provide timely and tangible outputs that may be used by sites implementing and sites yet to implement systems, including a toolkit.

3. To map out potential future scenarios of procurement and implementation of these systems internationally, and make recommendations to ensure maximum returns for all stakeholders.

Methods for data collection and analysis

This integrating WP was informed by a sociotechnical approach, thereby enabling us to interlink the qualitative and quantitative components of our work. 112–116 In doing so, we have drawn on our ‘systems life cycle’ model of the ePrescribing implementation process as a theoretical framework. 22,23

This WP has involved organising a range of conferences, workshops and focus groups with relevant audiences, including those outside case study sites who had an interest in implementing, adopting and optimising ePrescribing systems.

Key findings

The key deliverables for WP4 were to:

• set up a helpline for NHS trusts that would provide support and answers to implementation-related questions

• organise stakeholder conferences and annual information-sharing events

• develop a detailed typology of existing systems and an implementation and adoption toolkit to inform and support trusts implementing or yet to implement ePrescribing systems

• provide formative feedback to participating sites

• integrate findings and recommendations for procurement and implementation, and identify future research priorities.

Conclusions

Inter-relation with other parts of the programme

This WP has provided the overall summative element of the programme, integrating findings across qualitative and quantitative WPs, engaging a wide range of stakeholders and distilling overall implications for policy-making and implications for future research.

Background

Our PPI strategy was designed to be an important, cross-cutting component of our work. We had lay representation at every level of the research to contribute views, questions and priorities from the perspective of health service users and the wider public. Our lead lay member, Antony Chuter, was a co-applicant on the funding application. His early input contributed to formulating and refining the research proposal and to developing the PPI strategy. He was instrumental in recruiting the members of the six-strong PPI group (patients living with long-term conditions and carers) who worked with us to deliver the programme. He remained a key PPI member throughout, having primary responsibility for managing the PPI budget, in addition to attending the programme’s steering group and programme management group meetings and contributing to the lay advisory group’s work days.

Of the recruited volunteers, two experienced lay representatives had a programme oversight role on the independent Programme Steering Committee (iPSC), which convened twice per year. In our original PPI strategy (see Appendix 24), each of the remaining four volunteers was attached to one of the four inter-related WPs that made up the programme. In response to feedback in the first evaluation of the PPI strategy in 2012, we amended the processes to give our lay representatives greater involvement in the programme and more opportunity to work collaboratively with each other as a lay advisory group. The amended processes aimed to address some lay feedback of sometimes feeling isolated as the single lay representative attached to one WP and the challenge of appreciating the work of the programme as a whole beyond that particular WP. We also found that some researchers reported a level of uncertainty among the WPs’ academic teams as to how best to make full use of their individual lay representative. Subsequently, it was agreed that the PPI volunteers would participate in the programme’s quarterly steering group meetings, usually in pairs, and also all meet as a lay advisory group twice per year to work together on specific tasks for the programme.

For example, working as a lay advisory group, PPI representatives contributed significantly to the development of the protocol and documents for a patient study that was undertaken as part of WP1. Although the main work of the programme was primarily aimed at supporting professionals to implement ePrescribing systems in hospitals, PPI members argued for directly including patients’ views on hospital ePrescribing systems to inform the programme. This PPI input led the academic research team to embed a new patient study within WP1. The study included interviews with hospital ward patients and with several patient organisations (i.e. Diabetes UK, British Heart Foundation, Anticoagulation Europe, Carers Trust, Neurological Alliance, Lifeblood, National Kidney Federation, National Childbirth Trust and Healthcare Quality Improvement). Key outputs from the lay advisory group over the course of the programme also included producing a ‘jargon buster’ or lay glossary of ePrescribing-related terms and a list of top 10 tips for researchers for successful PPI in HIT research (see Appendices 25 and 26). In addition, the lay advisory group requested and then worked on a set of lay summaries of all of the programme’s academic publications (see Appendix 27). These lay summaries were then circulated to the patient organisations of the hospital trusts that had been participating in the programme and, along with the lay glossary and PPI tips, they are publicly available at www.ed.ac.uk/usher/eprescribing-antimicrobial/publications-and-research-outputs/publications-from-previous-eprescribing-programme (accessed 28 January 2019).

A retained component of the original PPI strategy was to have a designated, part-time staff member who had the role of supporting the lay co-applicant and the PPI volunteers; liaising between lay and staff team members; and, throughout the programme, undertaking a series of interviews and reviews of meetings’ minutes to continually assess the PPI processes and outcomes. Evaluation of the programme’s PPI was considered an important element of the PPI strategy (see Report Supplementary Material 1).

In 2013, a second round of PPI evaluation interviews found that the revised PPI processes were deemed more successful by both staff and lay team members. Therefore, we continued to have lay participation in steering group meetings and lay advisory group workdays between 2013 and the close of the programme.

We enjoyed a high level of commitment from and retention of our lay volunteers, despite the programme lasting for 5 years. Only one of the recruited PPI members left (owing to ill health) and was replaced in that time. We believe that this retention record was, among other factors, in large part attributable to strong commitment to PPI from the principal investigators heading up the programme.

All of the programme’s PPI members were invited and funded to attend the conferences and workshops that were organised by the programme. Hence, PPI was integrated with all of the programme’s activities, at every level, from the inception of the research to its completion.

Aims

The aims of the PPI strategy, which remained unchanged for the duration of the programme, were to:

• Conduct our academic research in the NHS with awareness of what is important from a lay perspective – whether or not what we talk about and plan to do makes sense, what is missing and whether or not the way that we present/disseminate what we have undertaken and why is comprehensible and meaningful to health service users and members of the public. This might relate in particular to the patient-/carer-facing aspects of our work, but not exclusively.

• Undertake an ongoing evaluation of the PPI strategy to (1) address any problems and suggestions at an early stage to enhance the potential benefits of co-operation between the public and the researchers in our programme; and (2) enable us to share what we learn about PPI in health-related research and thereby contribute to current debates about PPI.

Methods

The initial PPI co-investigator was an experienced lay representative in health-related research, with whom we had worked previously. We invited him to be a co-applicant on the funding application and, thereby, have input in the development of the research proposal. The subsequent recruitment of additional PPI members drew on our lay co-applicant’s network of patient and carer contacts. Following an induction day for all of the programme’s PPI recruits (see Appendices 28 and 29), we integrated PPI in the day-to-day work of the programme by including lay representation at the grant holder and management level (n = 1), iPSC level (n = 2) and steering group level (n = 4), with, from 2013 onward, additional separate meetings of the lay advisory group that consisted of all of the lay representatives other than the iPSC lay members. We evaluated the PPI strategy by a staff member reviewing the minutes of the programme meetings and by conducting repeated rounds of semistructured, thematically analysed, audio-taped interviews between 2012 and 2016. These interviews were conducted with both lay and staff team members to assess how well or otherwise the PPI strategy was perceived to be working to pinpoint and address any problems with the strategy and to identify outputs of the programme’s PPI.

Patient and public involvement outputs

Patient and public involvement representatives saw their principal role in attending steering group meetings to be asking questions and continually bringing the academic team members’ focus back to what they perceived to be the most important reasons for doing the research, that is patient safety and the welfare of individuals who were or would become NHS patients or carers. In addition to such ‘soft’ benefits of PPI in terms of having lay perspectives constantly brought to the programme table, we were also able to identify specific outputs derived from the PPI strategy and from the lay advisory group’s work days. The key specific outcomes of PPI are summarised below:

1. instigating a patient study to be embedded in WP1 –

   • input in the design of the study protocol and documents

   • reviewing the findings of the patient study

2. lay summaries of the National Institute for Health and Care Research (NIHR) ePrescribing programme’s written academic outputs

3. ‘10 top tips’ for researchers for successful PPI in HIT-related research

4. a ‘jargon buster’ (lay glossary) of ePrescribing-related terms

5. two submissions to the 2016 conference of the Society of Academic Primary Care, which was held in Dublin –

   • an award-winning ‘Dangerous Ideas’ submission – ‘Reorganising community pharmacy services to meet the patients’ needs’

   • poster presentation – ‘Getting PPI right: Who’s a lay patient?’

6. a PPI plenary, ‘Maximising the opportunity for partnership with patients, carers and members of the public’, at the NIHR ePrescribing programme’s 2016 London conference

7. an ePrescribing programme PPI-led workshop on PPI, in collaboration with the Greater Manchester Patient Safety Translational Research Centre, for delegates at the programme’s 2014 London conference

8. oral presentation – ‘The Why(s) and How(s) of PPI in the NHS’ (Wellcome Trust Symposium for the NHS, Edinburgh, 2013)

9. oral presentation – ‘Health Research PPI. Comparing Some Different Approaches’ (Society of Academic Primary Care, 2013)

10. poster presentation – ‘A Lay Partner’s Experience of Patient and Public Involvement in Health Research’ (ePrescribing Symposium, London, 2013)

11. PPI input to the development of the ePrescribing implementation toolkit and of the programme’s website –

    • overall ‘sense checking’ of the implementation toolkit

    • preparation of a PPI section in the toolkit

    • review of the frequently asked questions section

12. preparation of a journal article drawing on lessons learned from this Programme – ‘How to do HIT-related PPI’ (for submission to BMJ Open)

13. preparation of a journal article – ‘What is lay?’ (for submission to Res Involv Engagem).

Evaluation

Two major reviews of the available evidence on PPI in research122,123 reported that evaluating PPI was methodologically challenging and hampered by the complexity and wide variation in what constituted PPI. It has been argued that any evaluation of PPI could take account of the particular context and processes of that PPI activity. 124,125 The PPI literature indicates that, although PPI has become strongly advocated by the government and major research funding bodies, its evaluation often relies on anecdotal evidence of perceived benefits. 126–128 However, there is relatively scant reporting of health research PPI in the literature, and this is particularly so for HIT-related research. Our proposed approach was to base the evaluation of our programme’s PPI on PPI purpose + process + context = outcome(s) (Box 1).

Key findings

The first semistructured evaluation interviews (see Report Supplementary Material 1) in 2012 (lay interviewees, n = 5; staff, n = 5) provided very useful feedback on aspects of the processes of the original PPI strategy and indicated areas in which some improvements could be made (Box 2). Specifically, the initial evaluation highlighted opportunities for modifying PPI processes to help clarify the lay representatives’ roles for lay and staff team members alike.

The initial evaluation also showed early, perceived outcomes of the PPI strategy, with the interesting dimension that those outcomes were identified by programme staff rather than by PPI members. Instances where PPI input led to reported outcomes, that is specific positive or negative consequences of PPI, were identified under the headings of ‘helping staff to reflect and reconsider’ (e.g. changing document wording), ‘raising staff team members’ awareness of the way they communicate’ (e.g. modification of academics’ language in meetings) and ‘opening researchers’ eyes to a lay perspective’ (e.g. challenging staff perspectives). These might all be viewed as positive outcomes. Some staff interviewees were also able to identify specific consequences with respect to PPI creating more work for them, which was viewed as a negative outcome. The key findings then were:

1. Getting the right processes in place for optimum PPI in a large-scale, mixed-methods investigation into health-related technology takes time and requires flexibility in the PPI strategy.

2. The programme’s lay volunteers and staff team members recognise that PPI is a learning process and remain actively committed to making PPI as beneficial to the ongoing work as possible.

3. Academic investigators perceive more benefits from having PPI to date than the lay representatives. Currently, the ‘reward’ for the PPI members lies in personally finding it interesting to learn about the programme, but what they want is to know that they are being genuinely useful.

4. The reported PPI benefits can come at a personal cost to academics in terms of additional work and effort from them.

The following round of evaluation interviews that took place in 2013 (lay interviewees, n = 5; staff, n = 5) found that, overall, the revised PPI strategy was working far better for both staff and lay team members (Box 3). Based on the higher levels of satisfaction reported in 2013 and in subsequent years, we made no further changes to the PPI processes during the programme. Lay members increasingly felt that they were a useful part of the team, and staff members increasingly valued the influence and input of the PPI members at steering meetings and at the programme’s conferences and workshops.

By the time of the final round of evaluation interviews in 2016 (lay interviewees, n = 5; staff, n = 6), we were able to trace attitude shifts in some but not all of the interviewees. Changes in team members meant that it was not always possible to re-interview the same individuals throughout the whole programme, but we did so as far as was possible. Among the repeatedly interviewed research staff, some started and remained enthusiastically committed to PPI and to learning from this programme to ‘do’ PPI even better in the future. Others were somewhat ambivalent about the usefulness of PPI at the beginning and clearly shifted towards greater recognition of its worth as the programme progressed. A few members of the staff team appeared to start out and to remain somewhat ambivalent about PPI.

The lay representatives, by contrast, became universally convinced of the importance of involving patients and carers in HIT-related research after some initial, personal uncertainty about their usefulness on the ePrescribing team. This was because they saw beyond the technology itself to the potential impacts of using HIT systems in hospitals. They envisaged implementation of the ePrescribing system leading to improved medication safety for patients, to more convenient and efficient care processes for patients, their carers and hospital staff, and to cost savings to the NHS, all of which were deemed highly relevant to NHS patients, carers and taxpayers.

Evaluation interviewees in 2015 were asked to identify the most notable impact of PPI over the course of the programme. Staff interviewees repeatedly referred to the ‘soft’ but valued influence of the lay members in keeping the researchers ‘grounded’ and aware of NHS patients’ perspectives (Box 4) and, more specifically, to the introduction of an additional patient study to the original programme’s scope. However, it was acknowledged that, although our PPI strategy was deemed successful and well resourced, PPI had sometimes felt challenging. Interviewees suggested that this was because of the nature of the research topic, which was not specific to any particular health condition and, therefore, not closely aligned to a clearly identifiable patient group. Other perceived PPI challenges reported by interviewees related to the very different types of work involved in a complex, long research programme and, particularly, to the large size of the academic team and its geographical dispersal.

A further challenge highlighted in the final interviews was how well the university sector in general was geared up to deal with the bureaucratic administration necessary to support research PPI, for instance organisations’ processes for making honoraria payments to PPI representatives attached to academic research teams.

Staff and lay interviewees cited good interpersonal communication and genuine commitment to PPI from the very top of the research team as the most critical factors underpinning successful PPI; overall, this programme’s PPI was considered successful (Box 5). An improvement for the future would be to increase the emphasis on the PPI inductions and to include the academics, not only the PPI recruits. A joint induction would allow staff and lay team members to meet face to face and jointly to discuss expectations and hopes, and to refine roles at the start of a project, and might also help foster open, constructive dialogue for its duration.

Discussion

Other authors have noted the importance of attending to process and context in attempting to evaluate PPI. 122 Previous experiences, personal attitudes and individual academics’ skills and capabilities with respect to engaging with PPI form an important part of the context in which research PPI can function well or less well. Beyond individual factors, organisational structures and processes contribute to the context. The various contextual factors will influence a range of PPI activities taking place within a given institution over time, and potentially negatively if problems are not identified and addressed in a timely manner. We have highlighted two areas in the context of our PPI that might perhaps be improved in future PPI undertakings. When the academic research team is large and often physically working long distances apart, greater thought could be given to creating an induction at the start of the work to include both staff and lay representatives. Both may have misunderstandings of what is and what is not possible for PPI to achieve, and both could benefit from candid debate, greater support and possibly some training if wanted. At the institutional level, universities in general could consider reviewing their PPI-related processes. Embedding simpler and streamlined channels for servicing the needs of PPI representatives recruited to their organisations would help to support a positive context for PPI work.

We also identified some problems with the processes of our early PPI strategy, which we addressed in response to evaluation feedback at the end of the first year. This highlighted the need to approach planned PPI strategies with flexibility and for evaluation if reasonably nimble adjustments are to be made if the need arises.

We found that HIT-related PPI in most respects proved very similar to other health-related PPI. Lay representatives for HIT research do not need to be technology experts: they understand that HIT systems, and how these are deployed by the NHS, will effect patients and carers and, for this reason, clearly see themselves as stakeholders in HIT-related research. On the other hand, researchers may be slower to see patients and carers as HIT stakeholders, for example, compared with patients with a particular health condition who are more readily recognised as stakeholders in research programmes related to that condition. Nonetheless, it was our experience in a large and complex programme that aspects of the work were more amenable to PPI than others. For instance, the patient study benefited more obviously from PPI input than the health economics work. With hindsight, we could perhaps have increased the potential benefits of having PPI by ensuring that all of the study protocols within the programme included PPI from the very beginning; however, practical implications in terms of finding additional time to do so and of not overburdening part-time volunteers needs to be considered.

The other main findings from evaluating the programme’s PPI strategy included the importance to lay representatives of enthusiastic engagement with PPI by the team leader, the value of recruiting a mix of more experienced and less experienced PPI members (who bring different perspectives and learn from each other), the need to bring in PPI right at the start and the usefulness of having a named staff member specifically for PPI. These all are common to PPI findings reported in other areas. 123 For this reason, and given the very different contexts and processes that are happening under the umbrella of PPI, HIT-related PPI does not appear to be clearly distinguishable.

Patient and public involvement was found to be a valuable asset in this instance of HIT-related research.

Implementation and effectiveness research

There are many areas of research that are within the scope of our programme of work, but that we have been unable to explore in sufficient detail:

• We discovered that there is limited knowledge transfer not just within England, but more widely within the UK and internationally. We would, therefore, suggest that more systematic cross-national research is required into increasingly globalised system development and implementation to give insight into market adaptation and systems adoption. This could, for instance, include comparative work relating to particular ePrescribing systems.

• In terms of prescribing error research, wider-scale use of error reporting tools, such as IMPACT, is recommended. IMPACT has recently been updated to include newer errors commonly found in hospital settings to reflect the changing nature of drug use, and it can provide a useful benchmark for hospitals implementing ePrescribing and wider HIT interventions. We are keen to make IMPACT available via our toolkit for large-scale real-world comparative error data collection.

• The design of future research programmes for service delivery interventions are recommended to ensure a sampling design that (1) is consistent between study sites, (2) identifies admission type and ward for each patient and (3) includes enough hospitals to reliably estimate between hospital variance in treatment effects.

System optimisation issues

Our research has proposed a system life-cycle approach to ePrescribing adoption, with a clear steer that implementation is only the starting point for further cycles of system enhancement and optimisation. Even our analysis of embedded sites has been limited; therefore, we would propose careful longitudinal study of system optimisation issues:

• In view of our observation that there is a continued dynamism of HIT applications, including extension in technical functionalities and increasing integration between applications, there is a need to continue to independently evaluate this rapidly evolving landscape that may further shape the field. These evaluations are ideally longitudinal in nature and responsive to changing environments.

• We are aware that effective system configuration and optimisation is important in achieving benefits in terms of quality, safety and efficiency of medicines use. It remains unclear what is the best strategy towards systems optimisation to deliver the digital maturity of systems and medicines optimisation and related activities within the NHS. A systematic evaluation of ePrescribing system optimisation approaches (including those that are looking at the usability and safety of systems in practice) is required. This could begin with systematic review activities and most usefully focus on secondary uses of data.

• Given the differences in effectiveness that we and others have observed even when the same system is implemented in different sites, comparative analysis of systems use in practice is warranted. We would promote the use of tools similar to Leapfrog (The Leapfrog Group, Washington, DC, USA) (used in the USA) as a simulation tool to ensure that systems are optimally configured to address known usability and safety issues in ePrescribing.

Secondary use of data to improve population use of medicines

The secondary use of data to drive quality improvements is in some ways part of systems optimisation; however, this is such an important issue that we believe that it would benefit from a research stream in its own right:

• We noted the early impacts of trust’s secondary uses of ePrescribing data to drive quality and safety initiatives at an institutional level, particularly within our embedded case study sites. The evidence behind effectiveness of medicines optimisation approaches at a population level using hospital ePrescribing data deserves further evaluation. Particular areas that are relevant in the current climate are strategies to improve efficiency and reduce waste in medicines use in hospitals.

• In addition, we are keen to explore the key data science strategies in terms of medicines (getting information out and using it) in more detail to drive evidence-based informatics approaches. This needs to include exploring artificial intelligence and machine learning based approaches and can be promoted by closer collaboration with industry.

Other issues

Inevitably, there are a few areas that do not fall neatly within the above areas, but are nevertheless important potential areas for future research:

• We noted from our patient study that the use of systems has the potential to change the way that we care for patients and how patients interact with their medication-related data. Further work may focus on how patients access information about their medication and how systems can not only provide CDS for practitioners but also allow patients to provide input for their decisions alongside the technical infrastructure of the ePrescribing systems. In the era of person-centred care, such study is important to avoid cutting the patient out of decisions about their medicines.

• We found it very difficult to extract data on the costs of these systems from hospitals owing to commercial sensitivities. We considered resorting to freedom-of-information requests, but were concerned that this was likely to have alienated our case study sites. Now that this work has been completed, there is scope to see if information on costs can be extracted in this way.

• It was clear that there was reluctance to deploy or fully deploy the computer decision support functions on ePrescribing systems. Such error-trapping solutions are reported to be effective in the literature and have the potential to reduce medication error, arguably to a greater extent than any other particular intervention; the barrier to use seems to lie in a fear of information overload and alert fatigue. More research is, thus, urgently required in how to optimise feedback in real time from IT systems.

• There is a need to formally evaluate the impact of the ePrescribing toolkit that we have produced and, if necessary, to improve this. It also needs to continue to be populated as systems are further evolved (e.g. with the anticipated move towards closed-loop prescribing systems).

• Finally, our research has recognised that ePrescribing and CDS systems are only a relatively small part of a much larger technical infrastructure that involve medication use in the hospital environment. As the NHS moves towards a more integrated view of digital medicines management, it is imperative that the integration of ePrescribing with other supporting technologies (e.g. smart infusion pumps) and data processes (e.g. clinical dashboards and personal health records) is also subject to further investigation.

Health-care professionals/personnel

Health-care managers often underestimate the effect of system implementation on short-, medium- and long-term care processes. Our work shows that, although the introduction of systems can enhance care co-ordination and improve safety, ePrescribing system implementation can change workflows, redistribute roles and have unintended and undesired outcomes. The unintended and undesired outcomes include local elaboration of new informal processes (workarounds) to manage constraints in the implemented system. Actively addressing such usability issues with clinical and system staff to eliminate any novel errors and identify opportunities for system improvement can facilitate implementation and improve safety.

We found that hospitals lacked knowledge about different aspects of the implementation journey, often including overambitious expectations of system deliverables leading to delays with procurement and challenges with implementation. Small groups of informatics specialists and seconded clinical staff played a key role in system procurement and roll-out. Staff turnover owing, for example, to their employment on contracts/secondments that were often shorter than the lengthy time scales of system implementation impeded progress. There were also longer-term consequences because system optimisation and secondary uses often rely heavily on initial implementation knowledge. At the end of a project, experienced staff often reverted to their clinical roles or general informatics functions.

In the past, doctors, nurses and pharmacists spent their education using paper-based systems to later work with EHRs, which led to consequences for their understanding of, confidence in using and acceptance of ePrescribing systems. This work has shown that the education of health-care professionals using the electronic systems that are used in clinical practice will enhance the quality of training and ultimately the safety of their future patients. Prior training in electronic systems could enhance the acceptance of incoming cohorts of health professionals and allow them to effectively exploit the full functionality of IT systems.

We noted the particular influence of senior clinical managers in the successful implementation strategies in some of the case study sites. By contrast, we saw that the scale and long time frames of implementation decisions, especially with the joint procurement, made the chain of ownership and engagement too attenuated to achieve effective engagement of senior clinical and other staff. Increasing skills to align IT with business and clinical needs to use systems to drive safety, quality and efficiency was found to be key.

Hardware

We noted difficulties in the availability and deployment of hardware and wireless connectivity in hospital practice. Although this seemed to be one of the more apparently remediable challenges, we found it to be a prerequisite of system implementation. This included enabling capabilities in terms of hardware and wireless infrastructure to cope with the demands of system use during peak times of clinical activity. As well as capital investment, we have identified the importance of revenue funding streams to provide for the upkeep, maintenance and planned replacement of hardware, as well as additional IT support.

Other considerations, such as business continuity, downtime procedures and planned maintenance, were also found to be important. Novel issues, such as cyber security, were only beginning to be discussed during our research programme, but are likely to become more important.

Software

The implementation and adoption of ePrescribing was extremely challenging and time-consuming. This was, to a large part, reflected by the relative immaturity of currently available ePrescribing system offerings, and the lack of local expertise and experience within individual hospitals in their implementation and exploitation. We have supported hospitals with system deployment through the use of the ePrescribing toolkit, but sharing of existing expertise for the deployment and optimisation of systems in tandem with the establishment of Global Digital Exemplar sites in the NHS may support the national digital agenda.

The programme has illustrated that promoting centralised support for procurement and optimisation strategies (e.g. interoperability standards, transferring data between primary care and hospitals) is key.

Given that the ePrescribing system market in the UK is still immature (with a limited number of systems and established vendors), with hospitals having limited choice of systems that are tailored to the needs of the NHS, our work has shown that hospitals need to be aware that packaged solutions may vary in maturity and are not likely to fulfil all demands of different specialties. Investing in local expertise that allows creating innovative approaches and interfacing solutions, as well as relationship building with vendors, may mitigate this risk.

We found the principle of procuring technology based on an identified need within the NHS is important throughout, and that an adequate base of suppliers with implementation experience before promoting wide adoption of new generations of technology could facilitate wider rollout. Pursuing generification strategies to create economic and robust generic solutions for the UK market, while incorporating the flexibility to cater for local specific functional requirements, has been shown to enable vendors to match the expectations of current health-care providers. We also found that packages need to incorporate a variety of care processes made available by configuring the software rather than by slow and expensive processes of user customisation requests. This is in concord with the next steps on the NHS Five Year Forward View,24,121 which suggests that hospitals will not choose an IT vendor in the future, they will choose a hospital that they want to partner with and implement the same system, keeping the IT 80% the same and making only the 20% of changes that are absolutely necessary to meet local needs.

We note that, at a hospital level, increasing financial support, efforts to study existing paper-based processes before the introduction of systems to inform whole-system (re-)design, continued stakeholder engagement, and evaluation efforts throughout the system life cycle can act as important enablers. Although desired efficiency savings may not materialise within specified time frames, we have found that in case study sites organisational vision was an important driver for change. The programme indicates that factors that are particularly important to consider going forward include the role of developing business cases considering strategic contexts, cases for change and objectives; future service requirements and options appraisal; capital and revenue implications; time scales and deliverability; and risk analysis and management.

Throughout implementation, we note that it is important to acknowledge and accept that an initial decline in organisational performance is likely, and that costs may exceed anticipated benefits. Our work shows that to ensure realisation of benefits, tracking of potentially undesired workaround and unanticipated risks is a crucial step that organisations need to take. This may include wide dissemination of risks, particularly across hospitals that are implementing the same system and also nationally.

Evidence suggests that delays in implementation timelines are par for the course. These can be owing to a range of tactical, unintended, internal (i.e. the adopting hospital) and external (i.e. suppliers, other hospitals, policy-makers) factors. User engagement can be promoted through tactically introducing clinical benefits as early in the implementation process as possible.

Our work also shows that system optimisation activity is likely to be ongoing. It consists of resolving continuity usability issues, risk mitigation strategy and refining systems for more effective use/re-uses of information. At a national level, it is likely to also include integration of care settings and refinement/updating of standards.

We have further found that integration of ePrescribing with ancillary information (contextualised demographic data) and specialty systems, and across settings (hospital, primary and social care), is an activity that is likely to result in significant benefits.

Within organisations, this work indicates that leadership is important in ensuring that continued resources are available to optimisation activities and that a dedicated optimisation team drives development forward.

Evidence from case studies illustrates that unrealistic user expectations about ease of adoption and the ability of complex packaged software to match particular organisational processes can inspire a sense of disengagement with systems that are not yet adopted. Improving the culture of use by organising appropriate training, as well as by assessing processes offered by the systems and redesigning work practices where adopted systems’ processes are more advantageous, can facilitate these developments. Our research found that the implementation of ePrescribing systems (stand-alone CPOE or integrated CDS) was associated with overall reductions in the risk of clinically important prescribing errors. We have also found that effective system configuration and optimisation is important in achieving these benefits. Given that we also found no clear advantage for the more advanced and expensive alerting system over-and-above the more basic ePrescribing system, there may be implications for how hospitals consider starting the journey to ePrescribing even if fiscal constraints do not allow for a full EHR implementation.

Policy/practice

We have shown that hospitals find that wider support in the procurement implementation and optimisation of ePrescribing systems is beneficial. The ePrescribing toolkit and the associated frameworks and tools that we have produced to help overcome some of the challenges, maximise the benefits of ePrescribing systems and minimise potential harms and associated costs may have benefits beyond the programme.

Data science

This work has illustrated that effective transfer of data within and between systems is vital for the efficient and effective operation of HIT systems. We identified that a lack of availability of timely information and duplicate data entry compounded these problems, even in integrated systems in which there was often a lack of data coherence in accessing information across modules. We note that interoperability is a key enabler of data sharing within the NHS to support the life sciences research strategy and encourage the full-scale adoption of common standards, such as Dictionary of Medicines and Devices129 and SNOMED-CT (SNOMED International, London, UK) within systems.

Data on medicines use are particularly important in relation to efficiency of health-care delivery and reduction in variation within the health service. Chronological analysis and content analysis of 207 interviews conducted in six sites over the period 2011–16130 suggests that policy-makers often failed to appreciate the length of time associated with enabling a meaningful utilisation of secondary uses of data. Most of the users reported that, although secondary uses of data were conceivable, they lacked appropriate skills and tools to extract and process data accordingly. This may suggest that there is a need to continue to track emerging benefits through longitudinal evaluations of systems and processes, as well as upskill data processing skills within the NHS workforce (including health, clinical and pharmacy informaticists).

Cross-cutting themes

Infrastructure, digital maturity and constantly evolving markets are factors that are likely to influence local and national developments over the coming years. Our work shows that anticipating to what extent effects of systems on the organisation and delivery of care can be anticipated and potential risks can be mitigated can facilitate these developments. This may include monitoring new sources of error, developing system resilience, being alert to cybersecurity threats and developing capacity in the NHS (monetary, analytic, change management) to maintain and optimise systems.

An overall shared vision of a learning health system at individual, organisational and national levels may facilitate such efforts and ensure that different stakeholders are working towards a common goal.

Concluding comments

The overarching implications that we highlight are that there is now an increasing experience around the implementation of ePrescribing and an accumulating knowledge base within sites that is captured within the toolkit. However, moving systems and organisations beyond the implementation stage towards ePrescribing system maturity and optimisation strategies will be key to maximising the benefits in terms of quality, safety and efficiency of medicines optimisation and related activities in the NHS.

Contributions of authors

Professor Aziz Sheikh (https://orcid.org/0000-0001-7022-3056) (Chairperson of Primary Care Research and Development) contributed to the overall design of the programme and specific design of individual WPs and protocols; contributed to writing the Scientific summary and Abstract; and contributed to writing the sections What was successful and not successful in the programme?, Recommendations for future research and Implications for practice.

Professor Jamie Coleman (https://orcid.org/0000-0002-7512-5153) (Professor in Clinical Pharmacology and Medical Education) contributed to the overall design of the programme and specific design of individual WPs and protocols; contributed to writing the Scientific summary and Abstract; contributed to writing Work package 2; and contributed to writing the sections What was successful and not successful in the programme?, Recommendations for future research and Implications for practice.

Mr Antony Chuter (https://orcid.org/0000-0003-0751-6758) (Lay Member) contributed to the overall design of the programme and specific design of individual WPs and protocols; contributed to writing the Plain English summary; and contributed to writing the Patient and public involvement section.

Professor Robin Williams (https://orcid.org/0000-0002-9044-4611) (Director, Institute for the Study of Science, Technology and Innovation Studies) contributed to the overall design of the programme and specific design of individual WPs and protocols; contributed to writing the Scientific summary and Abstract; contributed to writing Work package 1; and contributed to writing the sections What was successful and not successful in the programme?, Recommendations for future research and Implications for practice.

Professor Richard Lilford (https://orcid.org/0000-0002-0634-984X) (Pro-Dean, Research) contributed to the overall design of the programme and specific design of individual WPs and protocols; contributed to writing the Scientific summary and Abstract; contributed to writing Work package 3; and contributed to writing the sections What was successful and not successful in the programme?, Recommendations for future research and Implications for practice.

Ms Ann Slee (https://orcid.org/0000-0002-9731-7041) (ePrescribing Lead) contributed to the overall design of the programme and specific design of individual WPs and protocols; and contributed to writing the sections What was successful and not successful in the programme?, Recommendations for future research and Implications for practice.

Dr Zoe Morrison (https://orcid.org/0000-0003-4655-2268) (Senior Lecturer) contributed to the overall design of the programme and specific design of individual WPs and protocols; and contributed to data collection and analysis.

Dr Kathrin Cresswell (https://orcid.org/0000-0001-6634-9537) (Senior Research Fellow) contributed to the overall design of the programme and specific design of individual WPs and protocols; contributed to data collection and analysis; contributed to writing Work package 1; and wrote Work package 4.

Dr Ann Robertson (https://orcid.org/0000-0003-1077-0343) (Research Fellow) contributed to the overall design of the programme and specific design of individual WPs and protocols, and wrote the Patient and public involvement section.

Dr Sarah Slight (https://orcid.org/0000-0002-0339-846X) (Reader in Pharmacy Practice) contributed to the overall design of the programme and the specific design of the individual WPs and protocols.

Dr Hajar Mozaffar (https://orcid.org/0000-0002-6400-5260) (Lecturer in Innovation) contributed to data collection and analysis, and contributed to writing Work package 1.

Dr Lisa Lee (https://orcid.org/0000-0002-9735-9989) (Research Fellow) contributed to data collection and analysis.

Ms Sonal Shah (https://orcid.org/0000-0003-2465-9531) (Research Fellow) contributed to data collection and analysis, and contributed to writing Work package 2.

Dr Sarah Pontefract (https://orcid.org/0000-0002-0344-4075) (NIHR Doctoral Research Fellow) contributed to data collection and analysis, and contributed to writing Work package 2.

Ms Abby King (https://orcid.org/0000-0002-2212-5221) (Research Fellow) contributed to data collection and analysis, and contributed to writing Work package 1.

Dr Valeri Wiegel (https://orcid.org/0000-0002-9597-0645) (Research Fellow) contributed to data collection and analysis, and contributed to writing Work package 1.

Dr Samuel Watson (https://orcid.org/0000-0002-8972-769X) (Research Fellow) contributed to data collection and analysis, and contributed to writing Work package 3.

Dr Nde-Eshimuni Salema (https://orcid.org/0000-0002-0981-7661) (Senior Research Fellow) contributed to data collection and analysis.

Dr David Bates (https://orcid.org/0000-0001-6268-1540) (Chief Innovation Officer) contributed to the overall design of the programme.

Professor Anthony Avery (https://orcid.org/0000-0001-7591-4438) (Professor of Primary Health Care) contributed to the overall design of the programme.

Mr Alan Girling (https://orcid.org/0000-0003-2022-045X) (Reader in Medical Statistics) contributed to data collection and analysis, and contributed to writing Work package 2.

Dr Lucy McCloughan (https://orcid.org/0000-0003-4099-6183) (Research Manager) contributed to writing the Plain English summary.

Mr Neil Watson (https://orcid.org/0000-0002-5024-8456) (Director of Pharmacy) contributed to the interpretation of the results.

Publications

Yao GL, Novielli N, Manaseki-Holland S, Chen YF, van der Klink M, Barach P, et al. Evaluation of a predevelopment service delivery intervention: an application to improve clinical handovers. BMJ Qual Saf 2012;21(Suppl. 1):i29–38. https://doi.org/10.1136/bmjqs-2012-001210

Coleman JJ, van der Sijs H, Haefeli WE, Slight SP, McDowell SE, Seidling HM, et al. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop. BMC Med Inform Decis Mak 2013;13:111. https://doi.org/10.1186/1472-6947-13-111

Cresswell K, Coleman J, Slee A, Williams R, Sheikh A, ePrescribing Programme Team. Investigating and learning lessons from early experiences of implementing ePrescribing systems into NHS hospitals: a questionnaire study. PLOS ONE 2013;8:e53369. https://doi.org/10.1371/journal.pone.0053369

Cresswell KM, Bates DW, Sheikh A. Ten key considerations for the successful implementation and adoption of large-scale health information technology. J Am Med Inform Assoc 2013;20:e9–e13. https://doi.org/10.1136/amiajnl-2013-001684

Cresswell KM, Slee A, Coleman J, Williams R, Bates DW, Sheikh A. Qualitative analysis of round-table discussions on the business case and procurement challenges for hospital electronic prescribing systems. PLOS ONE 2013;8:e79394. https://doi.org/10.1371/journal.pone.0079394

Thomas SK, McDowell SE, Hodson J, Nwulu U, Howard RL, Avery AJ, et al. Developing consensus on hospital prescribing indicators of potential harms amenable to decision support. Br J Clin Pharmacol 2013;76:797–809. https://doi.org/10.1111/bcp.12087

Cresswell K, Coleman J, Slee A, Morrison Z, Sheikh A. A toolkit to support the implementation of electronic prescribing systems into UK hospitals: preliminary recommendations. J R Soc Med 2014;107:8–13. https://doi.org/10.1177/0141076813502955

Cresswell KM, Bates DW, Williams R, Morrison Z, Slee A, Coleman JJ, et al. Evaluation of medium-term consequences of implementing commercial computerized physician order entry and clinical decision support prescribing systems in two ‘early adopter’ hospitals. J Am Med Inform Assoc 2014;21:e194–202. https://doi.org/10.14236/jhi.v23i2.178

Cresswell KM, Sheikh A. Undertaking sociotechnical evaluations of health information technologies. Inform Prim Care 2014;21:78–83. https://doi.org/10.14236/jhi.v21i2.54

Lee L, Cresswell K, Slee A, Slight SP, Coleman J, Sheikh A. Using stakeholder perspectives to develop an ePrescribing toolkit for NHS Hospitals: a questionnaire study. JRSM Open 2014;5:2054270414551658. https://doi.org/10.1177/2054270414551658

Lilford RJ, Girling AJ, Sheikh A, Coleman JJ, Chilton PJ, Burn SL, et al. Protocol for evaluation of the cost-effectiveness of ePrescribing systems and candidate prototype for other related health information technologies. BMC Health Serv Res 2014;14:314. https://doi.org/10.1186/1472-6963-14-314

Mozaffar H, Williams R, Cresswell K, Morison Z, Slee A, Sheikh A, ePrescribing Programme. Product diversity and spectrum of choice in hospital ePrescribing Systems in England. PLOS ONE 2014;9:e92516. https://doi.org/10.1371/journal.pone.0092516

Cresswell KM, Lee L, Slee A, Coleman J, Bates DW, Sheikh A. Qualitative analysis of vendor discussions on the procurement of Computerised Physician Order Entry and Clinical Decision Support systems in hospitals. BMJ Open 2015;5:e008313. https://doi.org/10.1136/bmjopen-2015-008313

Cresswell KM, Sheikh A. Health information technology in hospitals: current issues and future trends. Future Hosp J 2015;2:50–6. https://doi.org/10.7861/futurehosp.2-1-50

Lee L, Williams R, Sheikh A. What does ePrescribing mean for patients? A case study of the perspectives of hospital renal patients. J Innov Health Inform 2015;22:391–8. https://doi.org/10.14236/jhi.v22i4.176

Schofield B, Cresswell K, Westbrook J, Slee A, Girling A, Shah S, et al. The impact of electronic prescribing systems on pharmacists’ time and workflow: protocol for a time-and-motion study in English NHS hospitals. BMJ Open 2015;5:e008785. https://doi.org/10.1136/bmjopen-2015-008785

Cresswell K. Evaluation of implementation of health IT. Stud Health Technol Inform 2016;222:206–19.

Cresswell K, Bates DW, Sheikh A. Six ways for governments to get value from health IT. Lancet 2016;387:2074–5. https://doi.org/10.1016/S0140-6736(16)30519-0

Cresswell K, Coleman J, Smith P, Swainson C, Slee A, Sheikh A. Qualitative analysis of multi-disciplinary round-table discussions on the acceleration of benefits and data analytics through hospital electronic prescribing (ePrescribing) systems. J Innov Health Inform 2016;23:178. https://doi.org/10.14236/jhi.v23i2.178

Lee L, Sheikh A. Understanding stakeholder interests and perspectives in evaluations of health IT. Stud Health Technol Inform 2016;222:53–62.

Lee L, Williams R, Sheikh A. How does joint procurement affect the design, customisation and usability of a hospital ePrescribing system? Health Inform J 2016;22:828–38. https://doi.org/10.1177/1460458215592915

Mozaffar H, Cresswell KM, Lee L, Williams R, Sheikh A, NIHR ePrescribing Programme Team. Taxonomy of delays in the implementation of hospital computerized physician order entry and clinical decision support systems for prescribing: a longitudinal qualitative study. BMC Med Inform Decis Mak 2016;16:25. https://doi.org/10.1186/s12911-016-0263-x

Mozaffar H, Williams R, Cresswell K, Morrison Z, Bates DW, Sheikh A. The evolution of the market for commercial computerized physician order entry and computerized decision support systems for prescribing. J Am Med Inform Assoc 2016;23:349–55. https://doi.org/10.1093/jamia/ocv095

Cresswell KM, Bates DW, Sheikh A. Ten key considerations for the successful optimization of large-scale health information technology. J Am Med Inform Assoc 2017;24:182–7. https://doi.org/10.1093/jamia/ocw037

Cresswell K, Mozaffar H, Shah S, Sheikh A. Approaches to promoting the appropriate use of antibiotics through hospital electronic prescribing systems: a scoping review. Int J Pharm Pract 2017;25:5–17. https://doi.org/10.1111/ijpp.12274

Cresswell K, Blandford A, Sheikh A. Drawing on human factors engineering to reconsider paradigms for evaluating the effectiveness of health information technology. J R Soc Med 2017;110:309–15. https://doi.org/10.1177/0141076817712252

Cresswell KM, Lee L, Mozaffar H, Williams R, Sheikh A, NIHR ePrescribing Programme Team. Sustained user engagement in health information technology: the long road from implementation to system optimization of computerized physician order entry and clinical decision support systems for prescribing in hospitals in England. Health Serv Res 2017;52:1928–57. https://doi.org/10.1111/1475-6773.12581

Cresswell KM, Mozaffar H, Lee L, Williams R, Sheikh A. Safety risks associated with the lack of integration and interfacing of hospital health information technologies: a qualitative study of hospital electronic prescribing systems in England. BMJ Qual Saf 2017;26:530–41. https://doi.org/10.1136/bmjqs-2015-004925

Cresswell KM, Mozaffar H, Lee L, Williams R, Sheikh A. Workarounds to hospital electronic prescribing systems: a qualitative study in English hospitals. BMJ Qual Saf 2017;26:542–51. https://doi.org/10.1136/bmjqs-2015-005149

Cresswell K, Slee A, Sheikh A. Five key recommendations for the implementation of Hospital Electronic Prescribing and Medicines Administration systems in Scotland. J Innov Health Inform 2017;23:904. https://doi.org/10.14236/jhi.v23i4.904

Mozaffar H, Cresswell KM, Williams R, Bates DW, Sheikh A. Exploring the roots of unintended safety threats associated with the introduction of hospital ePrescribing systems and candidate avoidance and/or mitigation strategies: a qualitative study. BMJ Qual Saf 2017;26:722–33. https://doi.org/10.1136/bmjqs-2016-005879

Mozaffar H, Williams R, Cresswell K, Sheikh A. Anglicization of hospital information systems: managing diversity alongside particularity. Int J Med Inform 2018;119:88–93. https://doi.org/10.1016/j.ijmedinf.2018.09.014

Wiegel V, King A, Mozaffar H, Cresswell K, Williams R, Sheikh A. A systematic analysis of the optimization of computerized physician order entry and clinical decision support systems: a qualitative study in English hospitals. Health Inform J 2019;26:1118–32. https://doi.org/10.1177/1460458219868650

Data-sharing statement

All data requests should be submitted to the corresponding author for consideration. Access to anonymised data may be granted following review.

Patient data

This work uses data provided by patients and collected by the NHS as part of their care and support. Using patient data is vital to improve health and care for everyone. There is huge potential to make better use of information from people’s patient records, to understand more about disease, develop new treatments, monitor safety, and plan NHS services. Patient data should be kept safe and secure, to protect everyone’s privacy, and it’s important that there are safeguards to make sure that it is stored and used responsibly. Everyone should be able to find out about how patient data are used. #datasaveslives You can find out more about the background to this citation here: https://understandingpatientdata.org.uk/data-citation.

Disclaimers

This report presents independent research funded by the National Institute for Health and Care Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, CCF, PGfAR or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, the PGfAR programme or the Department of Health and Social Care.

Cresswell et al.55

Electronic prescribing (ePrescribing) systems show considerable promise for making health care more safe, efficient and cost-effective. ePrescribing systems have been shown to lower prescribing errors and costs. While these systems are not yet in many English hospitals, there are plans to introduce them into most hospitals over the next 5 years.

Choosing an ePrescribing system is not straightforward. There are many systems available, coming with different capabilities and at varying costs. While one system might work better for one hospital, a different system could work better for another. Currently, there are no standardised guidelines to help hospitals decide which ePrescribing system might be best for them.

This paper looks at the perspectives of health professionals who have a prescribing role in hospitals that already use an ePrescribing system or in organisations that are planning to do so in the near future. We asked such health professionals from all over England to fill out a questionnaire to provide us with their views.

We found that the majority of our study participants felt unclear about which ePrescribing system should be chosen. They wanted guidance on how to choose a system, as well as guidance on setting up the systems in their hospitals. They also wanted good communication between an ePrescribing system and the other, local health-care computer systems.

We conclude that hospitals in England need more guidance on how to choose and to set up a new ePrescribing system. Lessons that have been learned from hospitals that have already set up systems should be shared with hospitals that are soon to go through this process. More research is needed to investigate the outcomes for hospitals of using ePrescribing systems.

Cresswell et al.50

Health-care environments are complex. Trying to integrate a new technology, such as an electronic prescribing (ePrescribing) system, is difficult and often comes with many frustrations. Health-care workers and health-care organisations need time to adapt to using a new technology. The benefits of having the new system may not be seen until after quite some time.

In this paper we offer important points for hospitals to consider before and during the process of setting up a new health-care technology. We use our experiences of studying various health-related technologies to give 10 pointers that should help health-care professionals and organisations to set up these new technologies.

1. Before selecting a technology:

   1. Agree on the goals for setting up the technology. In other words, what problems should the new technology fix and what improvements should the technology support?

   2. Will the technology affect the whole organisation or will it target a specific task? Agreement on this point needs input from management teams, as well as from the people who will be using the technology on a day-to-day basis (such as doctors and nurses).

   3. Consider all of the different options available. It is helpful to observe other health-care organisations that have already set up new technologies. It is important to consider how well a system will work in your own setting and how much the new technology will cost.

2. Choosing a system:

   1. Choose a system that works well for the organisation as a whole, as well as for individual patients and health-care workers. It is difficult to find a system that pleases everyone so, usually, compromises are needed.

3. Setting up the system:

   1. Aim to prioritise getting benefits for patients and health-care workers from the new technology as early as possible. The setting up process should support communication between those managing the setting up process and those using the technology.

   2. Consider the hardware that supports the new technology. For example, the system will need a strong wireless connection to work properly.

   3. Train staff just before the new technology is introduced. Training should be hands-on and simulate how the technology would actually work day to day. Some individuals will need more training than others and different types of training modules should be available for different people.

4. Once the technology is in use:

   1. Keep evaluating the technology and get feedback from the users. This feedback can help organisations use the technology in better ways in the future.

   2. Allow for the technology to change as the organisation changes its goals.

   3. Leave enough time for the organisation to see benefits from a new technology. Once these benefits are noticed, they should be assessed to make sure the technology is being used in the best way.

These 10 pointers will, it is hoped help health-care organisations to avoid common mistakes in the setting up of new technologies.

Thomas et al.69

Prescription errors made in hospitals can cause unnecessary inconvenience, illnesses and deaths. Monitoring the types, frequency and severity of these errors is crucial to preventing such errors. ‘Prescribing indicators’ – viewable effects of prescription errors – are generally seen by medical professionals as a sound way to record, measure and help stop errors being made. However, prescribing indicators are rarely very exact and they point more to matters of quality rather than safety. A standard opinion on the importance of different prescription errors does not exist. Therefore, as yet, there is no validated list of important prescribing errors to help prescribers in hospitals to work more safely. This study aimed to provide such a list.

We carried out two rounds of surveys of hospital pharmacists and doctors, who reported severe and/or highly frequent prescribing errors from both paper-based and electronic prescription systems. In the first round of the survey, participants identified 210 prescribing indicators. For the second round of the survey, the full list of reported indicators was returned to study participants in the hope of creating a more focused list. This second round produced a new and smaller list of 89 prescribing indicators, but with an additional 71 new indicators being suggested by our participants. The authors of the study then selected 20 of those new indicators for inclusion in the revised list, which brought the list total to 109. Of the prescribing errors that our survey had now identified, 80 of the 109 were assessed to be of high or extreme risk when the scores for likelihood and severity were considered.

Compiling and standardising prescribing indicators can be a very useful tool for understanding and preventing prescription errors in hospitals. It is hoped that our new validated list of important prescription indicators will help to refine and tailor the alerts issued in ePrescribing systems with clinical decision support, and so help hospital doctors and pharmacists to prescribe more safely for patients.

Yao et al.85

This study aimed to understand the cost-effectiveness of a service intervention to improve the transfer of patient care from hospitals to community organisations (clinical handovers).

Health professionals are undecided whether or not new services for understanding emergency preparedness (i.e. the ability to respond to health-care emergencies) represent a sound investment. In this paper, the authors describe creating a way to estimate the expected cost-effectiveness of proposed new services, which they tested by evaluating a potential service intervention to improve clinical handovers.

The new method focuses on economic evaluation at the pre-implementation stage before a new service or intervention has actually been introduced. Applying the method to a service intervention to improve clinical handovers required the authors to guess the number of potential adverse events and which adverse events were potentially preventable. Adverse events were classified by severity and duration, but not frequency. Afterwards, the authors estimated the level of success of the proposed intervention, by asking experts their opinion on the effectiveness of the intervention. After incorporating the experts’ opinions into the evaluation, the authors then estimated the costs associated with categories of adverse events and the savings resulting from the intervention.

Health policy reforms for emergency preparedness and response are being hampered by poor understanding of the health economics of proposed service interventions. This study offers a method that could be applied to evaluations of service interventions in general, while the interventions are still at the design stage. For this reason, the study might help with projected health economic evaluations of health service emergency preparedness interventions.

Cresswell et al.48

Electronic prescribing (ePrescribing) may help to improve the quality, cost and efficiency of health care. There is much interest in setting up computer systems across the UK for hospital prescribing, but progress with getting ePrescribing systems into hospitals has been very slow. We are working to develop a toolkit as an online resource for hospitals to provide them with help and advice for setting up such systems.

This paper shares our work so far in putting together this toolkit, which we are developing using information from the literature on ePrescribing, our own experiences and findings from early studies in this area. Importantly, we encourage users of ePrescribing systems to view setting up their system as a journey because much work is required before and after the initial phase of getting a new system.

In the toolkit, we give recommendations for each stage of implementing an ePrescribing system. Recommendations for hospitals include, but are not limited to:

• identify a hospital’s needs for a system

• put together a team to help with the setting up process

• look to other hospitals for help in understanding what different types of systems can offer

• involve as many of the people who will be using the system, from as many different professions as possible, when choosing a system

• build strong relationships with the people who supply the system

• have different types of training for the different types of people who will be using the system

• put all necessary hardware and software in place before setting up the system so it can run efficiently and effectively

• recognise that the process is ongoing and will continually change.

The toolkit recommendations will, we hope, be useful to hospitals as they judge their needs for an ePrescribing system, choose a system, plan the process of setting it up, set up the system and maintain the system over time.

Coleman et al.134

Clinical decision support for computerised provider order entry is an online system that helps health professionals to prescribe medicines in the right amounts at the right times for the right patient. Medicines can have damaging effects if they are administered wrongly. For this reason, systems can issue ‘alerts’ to health professionals who are prescribing for patients. Prescribing is often complex. Therefore, a clinical decision support system might issue many alerts, which can distract the prescriber and can lead to ‘alert fatigue’, where they the alerts start to be ignored.

Some clinical decision support systems might also issue too few alerts, which could result in a doctor or pharmacist making a prescribing error and causing an ‘adverse drug event’.

Today, there is no agreement or standard for the alerts in clinical decision support systems. The purpose of this study is to explore how clinical decision support alerts should look. The study authors wanted to understand what experts thought about the alerts in clinical decision support and, further, wanted to come to some agreement on the gaps in current clinical decision support capabilities. European experts in clinical decision support and computerised provider order entry were invited to participate in the study, and their contributions were vital to the study’s findings.

The authors list what they believe to be the most important knowledge gaps in clinical decision support alerts. A perfect system would be ‘100% sensitive and 100% specific’. Current systems can be very sensitive, but they are often lacking in specificity. To correct this, clinical decision support systems need to be made more sensitive and more specific than they are today. The authors also found that allowing prescribers to personalise the screens on their clinical decision support systems, as a person would personalise their smart phone, could be a good improvement. They thought that if doctors and pharmacists could customise their alert systems, they would be more likely to pay attention to them. The authors said that there were differences of opinion on the best timing and frequency of prescribing alerts. The authors, in conjunction with the experts, suggested that an alert should be displayed as early as possible in the prescription process and, ideally, the alert should be shown only once.

While the authors identified a number of problems with current clinical decision support, they agreed that much more research is needed to better understand the best ways to set up clinical decision support alerts.

They conclude that the use of clinical decision support systems within computerised provider order entry is increasing, especially in the USA and, therefore, this study is an important reminder to medical professionals in the UK and Europe of the need to assess prescribing alerts within computerised provider order entry systems.

Cresswell et al.117

Electronic prescribing (ePrescribing) systems could help to improve the quality of health care by making it more safe, efficient and cost-effective. However, setting up an ePrescribing system in a health-care organisation comes with challenges. ePrescribing systems can make big changes to the day-to-day work of the people who use the system. In addition, the benefits of using an ePrescribing system might take a while to be seen. Organisations that want to set up an ePrescribing system need to make a strong business case to show why it is worthwhile investing in the new system.

Business cases describe the investment that is needed, the expected benefits and the time frames for setting up an ePrescribing system, and that developing a business case is usually one of the first steps in the process of getting an ePrescribing system. Given that there are a lot of unknowns before the new system is set up, it can often be difficult to write a strong business case.

This paper looks at ways to deal with the difficulties of writing a business case. We did this by discussing the matter of business cases with groups of people from all over England who are interested in or who have experience of setting up ePrescribing systems.

We found that business cases should recognise that an ePrescribing system would introduce changes to the whole organisation, not to only some parts of the organisation. To make a good business case, it was important for hospitals to create strong relationships with the supplier of the ePrescribing system. Hospitals should also look to other hospitals with more experience of ePrescribing systems for help. As the setting up process went on, there would probably be changes in the finances of the health-care organisation, so a business case should not be ‘set in stone’. In addition, health-care organisations should distinguish between what they really needed from their ePrescribing system and what they would like from it.

We conclude that it is important to have a business case before introducing an ePrescribing system. Given that many English hospitals are thinking about setting up these systems, how to write good business cases is an important topic to discuss more. Hospitals in England should share lessons with each other, as well as looking to learn from experiences in other countries, when developing business cases for implementing ePrescribing systems.

Cresswell et al.58

Electronic prescribing (ePrescribing) systems may help to improve the quality and cost of health care. Many studies have investigated the short-term outcomes of setting up ePrescribing systems in hospitals, but there is a lack of research looking into the longer-term effects of ePrescribing systems.

This paper looks at differences and similarities between two English hospitals in which ePrescribing systems have been used for at least 2 years. The two hospitals are using different systems. One set up an ePrescribing system to help with ordering patients’ medications, called a computerised provider order entry system. The other set up an ePrescribing system that also includes clinical decision support. This second system is more integrated with that hospital’s other computer systems and it provides more advice for health professionals while they are prescribing. We used a combination of interviews, observations and analysis of ePrescribing system documents to carry out our study.

We found that there were not many differences between the two different types of ePrescribing systems that we looked at. We found that both of the hospitals reported benefits as well as negative effects from setting up an ePrescribing system. The reported benefits mostly came from using the data collected from the systems to improve quality of care. Staff members in both hospitals were frustrated that not enough computers were available for accessing the ePrescribing systems. In both hospitals, staff members were also frustrated by the amount of time it took to log in whenever they wanted to look at a patient’s medications or to write a prescription.

We conclude that these two different types of systems are similar in the sorts of challenges and benefits that they had produced in a 2-year period after being set up. We believe that 2 years is too short a time fully to understand all the benefits that an ePrescribing system could bring and that the information held in these systems is not yet being used as much or as well as it could be to benefit the hospitals. We recommend that hospital staff and policy-makers have more realistic expectations about the costs, benefits and timelines for benefits from ePrescribing systems. We also recommend that future research tracks the benefits from ePrescribing systems after a longer period of the systems being in place. It is important that future research into the longer-term effects of having ePrescribing systems should study a wider range of hospitals than we looked at in this study.

Mozaffar et al.37

More and more hospitals in England are moving from paper-based to computer-based systems for prescribing and managing the administration of medicines to improve patient safety. At a basic level, these computer systems, often referred to as ePrescribing systems, are replacing the old paper-based drug charts.

This paper investigated the different ePrescribing systems that are available. We searched published articles and data available online (such as news articles, hospitals’ own websites, suppliers’ websites and NHS England data sources) to identify the various ePrescribing systems in use or in the process of being set up. We also spoke to experts to ask about additional systems and their status in UK hospitals.

We found that there is a wide range of ePrescribing systems available to English hospitals, including 13 hospital-wide applications and a range of specialty systems (e.g. for chemotherapy). These were either stand-alone systems or integrated into the hospital’s other computer systems. More than half of the ePrescribing systems were from international suppliers. We found very few hospitals were using the same systems.

We conclude that the market for ePrescribing systems in England is still in its early days, and ongoing review of system suppliers and their products is required to assist hospitals going digital.

Lee et al.49

Electronic prescribing (ePrescribing) systems have the potential to increase the safety and efficiency of health care. Currently, there is limited experience or knowledge of how best to set up these systems in the UK. Our team is developing an online toolkit that will support hospitals to go through the process of setting up an ePrescribing system.

This paper looks at how the toolkit might provide support for those setting up ePrescribing systems in UK hospitals. Creating a user-friendly technology and a user-friendly toolkit require input from the users themselves. Therefore, we designed a questionnaire and gave it out to pharmacists, NHS managers, information technology staff, doctors and suppliers of ePrescribing systems who were at a national conference on ePrescribing. The questionnaire asked what people needed at different stages in the process of setting up an ePrescribing system. We used the answers to help determine how useful it might be to have an ePrescribing toolkit for hospitals and what content it should have to make it most helpful.

We found that there was great interest in having a toolkit that could give more help with setting up ePrescribing systems. We found that the toolkit would be most useful if it provided guidance for each stage of setting up a system. It should have online tools for planning and overseeing the system, as well as useful information from hospitals that had already been through the process. People were aware of the challenges of developing a toolkit that was general enough to help different hospitals across the country but at the same time could be specific enough to help particular groups of people in particular settings.

We conclude that there is clearly a need for more support for hospitals setting up and using ePrescribing systems. An online toolkit can offer reliable and up-to-date help. A toolkit also allows its users to share their experiences and ideas. We realise that the value of a toolkit will become more evident only once it is being widely used and people can share their knowledge and experience. This questionnaire study was only the first step in creating an ePrescribing toolkit. Future research should seek views from a wider range of users and especially from hospital patients whose care will be affected by ePrescribing systems.

Lilford et al.81

Health information technology is an area of computer technology concerned with the design and development of systems to generate and store health-care-related information and make use of that information. Health-care organisations that use health information technology, for instance, such as an ePrescribing system, are expected to have lower costs, improved efficiency and reduced errors while also providing better services and patient care.

To know if health information technology systems are really as good as might be expected, we need methods to study the effects, or end points, of using health information technology systems in health-care organisations. Here, the authors seek to develop a protocol for evaluating the cost-effectiveness of ePrescribing systems in four hospitals in the UK. In two of the hospitals, ePrescribing systems are to include extensive clinical decision support. In addition to ordering medications, systems with clinical decision support offer health professionals advice and onscreen alerts to aid decision-making while they are prescribing. In the other two hospitals, the ePrescribing systems are more basic systems.

The method that the authors describe for evaluating the ePrescribing systems is a statistical method called Bayesian probability logic. This is a type of statistic that uses knowledge of prior events to help to predict future events. For a cost-effectiveness evaluation of ePrescribing systems, we need to consider the costs and the benefits of limiting the number of adverse drug events. To do that requires identifying adverse events, the expected health benefits, the cost of the health information technology system and the cost consequences of avoided adverse events. No single event can totally capture the effectiveness of an ePrescribing system, and these authors describe observing changes in error rates and adverse events in four error categories (i.e. of errors resulting in death, permanent disability, moderate disability and minimal effect). The authors suggest using both quantitative and qualitative data from different sources to find evidence for or against the cost-effectiveness of ePrescribing systems.

This protocol is designed to help address the criticism that evaluations of health information technology systems do not capture all of the relevant end points. The authors argue that evaluations of ePrescribing systems cannot ignore multiple end points merely because they are difficult to observe and to analyse. This protocol seeks to include as many variables as possible in the cost-effectiveness analysis. The authors conclude that there is still room to better understand the full impact of ePrescribing systems on preventing adverse events in hospitals. The protocol they describe offers a method for evaluating the cost-effectiveness of ePrescribing systems, which could also be useful for evaluating other health information technology systems.

Mozaffar et al.24

Electronic prescribing (ePrescribing) systems could improve the quality, efficiency and costs of health care. Standardised ePrescribing systems are popular because they can communicate more easily with other computer systems in hospitals. They are also more affordable than a system that is especially designed for one health-care setting. Although many ePrescribing systems are available to buy now, and setting up these systems is a top priority for the NHS, so far, their uptake in English hospitals has been very slow.

This paper looks at why the uptake of ePrescribing systems has been so slow. To do this, we carried out interviews with companies that sell these systems, the vendors, and with people who are using ePrescribing systems. We wanted to understand the viewpoints of these two different groups. We also observed meetings between vendors and users of ePrescribing systems to help us understand the relationship between these two groups. In addition, we hoped to understand how the market for ePrescribing systems had developed over time.

We found that there were big differences in the range of capabilities offered by standardised ePrescribing systems and in the particular requests made to vendors by hospitals that had taken standardised systems. Most ePrescribing systems had been designed in other countries, usually the USA. This caused some difficulties because English hospitals differ greatly from American hospitals. Vendors and buyers went through many phases of redeveloping a system after it had been installed to try to make it fit the hospital better. That process was time-consuming. In addition, if the hospital requested changes that were too specific to that hospital, the vendor could not make the changes that the hospital wanted.

We conclude that the development of standardised ePrescribing systems for English hospitals is still at an early stage. We recommend that vendors take more time to decide on how to respond to change requests from hospitals before trying to set up ePrescribing systems throughout the UK. We also recommend that these systems are evaluated continually. The issues brought up in this paper will be important for other countries too, as they begin to adopt ePrescribing systems.

Lee et al.54

In England, the former National Programme for Information Technology was introduced with the aim of providing an electronic health record for every NHS patient in the country. At the time, the National Programme for Information Technology was the largest, public, health sector health information technology programme ever undertaken. Post National Programme for Information Technology, the UK government now wants the NHS to go paperless by 2018. However, some worry that the goal of going paperless is setting too difficult a task too soon.

This qualitative study aims to explore how the joint procurement model that was in place during the National Programme for Information Technology has affected the design, customisation and usability of electronic prescribing (ePrescribing) systems in use today. To explore the impact of the National Programme for Information Technology procurement process, we studied two sites in which an ePrescribing system had been jointly procured within one of the National Programme for Information Technology’s geographical areas.

We found that the procurement process, relationships with the system supplier and the strict contractual arrangements under the National Programme for Information Technology had contributed to reported usability problems with the ePrescribing system and to difficulties with implementing and customising the ePrescribing system. The National Programme for Information Technology arrangements had limited the sites’ ability to make changes to the ePrescribing system, which affected the system’s usability and the users’ perception of its usability.

This study illustrates how important the early stage of procurement is in the process of implementing an ePrescribing system. The joint procurement model, as used during the National Programme for Information Technology, appeared to make setting up an ePrescribing system more difficult and less efficient in the two sites we studied.

Cresswell et al.25

Health-care workers now argue that any assessment of HIT needs to account for its technical uses and its social uses. International observers of health care are also interested in this ‘sociotechnical’ perspective, a view that assumes that both organisational and human (socio) factors and information technology systems (technical) factors will influence the success of HIT systems. Sociotechnical evaluations of HIT, therefore, attempt to study the processes associated with using a new technology in its social and organisational setting. This approach to HIT evaluations could lead to learning lessons relevant for assessing information technology systems in other industries too.

Sociotechnical evaluations that use multiple methods are becoming more popular and are likely to offer a new research approach for some scholars. There have been clear benefits of sociotechnical analyses of the efficiency of HIT. However, there are issues with this approach, mainly that a sociotechnical analysis offers subjective qualifications rather than objective answers. There are, therefore, opportunities for scholars to decide on the role and scope of the sociotechnical investigation and analysis of HIT, and to understand if it demands further research in its own right.

The authors hope that there will be more sociotechnical evaluations of HIT systems in the future. Given the interests of financial and policy groups in developing HIT to improve health outcomes (and to reduce health-care costs), the authors argue that greater use of sociotechnical evaluations will help to illustrate the big idea of the relationships between technology, health-care organisations and patients. They believe that such understanding will improve patient well-being, reduce health-care costs and offer lessons for other industries.

Schofield et al.135

NHS hospitals in England are putting ePrescribing systems into place. ePrescribing systems have been successfully used in primary care settings.

Introducing such systems in hospitals can be problematic because they force pharmacists to change the way that they work. We do not know yet if such changes make hospital pharmacists work more or less efficiently or more or less safely. To understand such changes and the effects they bring, we need to develop a protocol to measure and analyse the impact of ePrescribing in hospitals. It is very important to identify and assess the impact if we wish to design HIT systems that aid (rather than hinder) the work of pharmacists.

We conducted a time-and-motion study in six wards at one NHS trust in England, with two periods of observations (one before and one after introducing the ePrescribing system). Researchers followed the hospital pharmacists, and recorded and time-stamped the details of all the work tasks that they carried out. They made a record of all the tasks that the pharmacists completed, how often they completed each task and the time that they took for each task. Then the researchers analysed changes in these measures before and after getting an ePrescribing system.

We also interviewed pharmacists to ask their opinions of the impact of ePrescribing on their work. Pharmacists’ expectations of the likely impact of ePrescribing were also explored.

This study has limitations. The results of the study relate to two English NHS hospitals. The pharmacists who took part may have altered their behaviour during the study, perhaps because of fears of being seen as ineffective. However, this is the first and only time-and-motion study of hospital pharmacists before and after using an ePrescribing system and, for this reason, it provides a useful baseline and guide for future studies.

Cresswell et al.136

Doctors, pharmacists and others are interested in improving current health care. For example, many hope to move from a doctor-centred model of care to a more patient-centred approach. Using HIT is seen as a very important part of making these changes happen in our health-care services. However, English hospitals are being slower to take up HIT than might be expected, and the authors of this paper seek to explore why this is so.

The authors review a number of factors that have contributed to the successes of HIT in English hospitals in recent years, and point out the complex challenges to hospitals of setting up HIT systems. They review possible future developments from HIT, such as increased patient involvement; the sharing of medical records; more patient-centred and integrated care for patients; using smart devices; sharing of health-related data sets; and new uses for those data. They argue that HIT systems should be viewed as aids to doctors and also as an important source of data to support re-designing health-care services. This is known as a Learning Health Systems approach. In the Learning Health Systems approach, different computer systems are linked with each other and share databases of information to make improvements to medical practice and to patient care. According to the authors, implementing HIT is a ‘work in progress’, which will aim to tackle new challenges as well as possible through taking a Learning Health Systems approach.

The authors conclude that health-care professionals need to understand that HIT systems are not final products but rather they are tools to help with a never-ending task of improving health outcomes.

Cresswell et al.118

Electronic prescribing (ePrescribing) systems may increase the safety, efficiency and quality of health care. There are many different types of these systems available and hospitals are responsible for choosing the system that will work best for them. There is a great need to collect knowledge and experience to help hospitals to choose a new system and to set it up. The companies that sell these systems, the vendors, have the most experience of setting up ePrescribing systems and could be an important resource for other hospitals, but the vendors’ knowledge is rarely used more widely.

This paper looks at some of the challenges of setting up ePrescribing systems from the point of view of the vendors. It also considers how those challenges might be addressed, using insights gained from vendors. We held group discussions with nine participants, who represented six different vendors.

We found that there were currently no guidelines for hospitals that were choosing and setting up an ePrescribing system. Vendors recommended that hospitals should employ people with a lot of experience in that particular hospital to help out with decision-making and setting up the new system. Vendors suggested that hospitals often moved too quickly through the process of setting up an ePrescribing system. They recommended that hospitals should begin by setting up a simple system and give the staff time to get used to those changes first. Vendors also mentioned the importance of strong relationships between vendors and hospitals. Good relationships would help hospitals to choose the system that would best meet their needs.

We conclude that vendors of ePrescribing systems can provide important insights to hospitals on how best to choose and set up a system. Guidelines for hospitals are needed to support them through the decision-making and setting up phases. These findings may be useful to English hospitals introducing ePrescribing systems, as well to health-care organisations in other countries.

Lee et al.45

Parts of this section have been reproduced with permission from Lee et al. 45 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by/4.0/. The text below includes minor additions and formatting changes to the original text.

Electronic prescribing (ePrescribing) in hospitals is expected to improve the quality of patient care. Patients themselves, however, have not had the opportunity to voice their views on ePrescribing systems. Anecdotal evidence suggests that many patients may understand the potential benefits of electronic health-care systems. For instance, it is thought that patients recognise that ePrescribing could improve information-sharing between health professionals and patients. This study looked at the significance of ePrescribing systems to patients.

We interviewed representatives from national patient groups as well as hospital patients in a renal ward in which an ePrescribing system was in use. We also spent time on that ward and took notes about what we observed. We organised the information collected in these three ways by keeping a focus on perceived benefits of ePrescribing; patients’ awareness and understanding of their prescribed medications; and patients’ views on medications prescribed (or advised) on admission and on discharge from hospital.

We found that, overall, patients seemed to view ePrescribing in a positive light. However, the ePrescribing system in the site we studied did not seem to have improved patients’ involvement in the prescribing process or to have improved medication counselling at discharge. We found that ePrescribing systems did not make patients feel that they had more information about their medications. Indeed, by removing the paper charts at the end of hospital beds, patients seemed to have less access to information about their medications and care than before. We did not identify technical reasons to do with the ePrescribing system itself to explain why patients’ involvement in hospital prescribing had not apparently changed for the better.

We conclude that ePrescribing could bring new opportunities for patients and health-care professionals to improve patient involvement. Having an ePrescribing system is not enough on its own to promote better patient involvement. To achieve that also needs other changes to take place, such as policy changes and changes in health-care professionals’ approach to prescribing for hospital patients.

Cresswell et al.52

This section has been reproduced with permission from Cresswell et al. 52 Published by John Wiley & Sons. © 2016 Health Research and Educational Trust.

The implementation of health information technologies, including electronic health records, computerised provider order entry and clinical decision support systems, promises health organisations to achieve a range of benefits. However, these benefits can be achieved only if the systems are used as intended. Implementing health information technologies is not without challenges and risks, and not all obstacles can be resolved during implementation. To overcome challenges and to realise benefits in the long run, implementers need to pay more attention to the needs of prescribers and other users of these systems after implementation. However, given that most research focuses on the challenges prior and during implementation, we have little understanding of issues that users experience in later periods of system use. For this reason, this study aims to explore a long-term perspective on user engagement through investigating the range of ways in which health-care workers and organisations familiarise themselves with new technology and how they adopt them in their workplaces over time.

To begin, we define that user engagement is a process with three elements: (1) the need to involve different types of users in the selection and design of a technology, for example by giving them the opportunity to be heard; (2) the need to make sure that the workforce supports using a new technology; and (3) the need to give users the opportunity to change the technology as they discover its features and capabilities over time.

During the period of 2011 and 2015, we approached six hospitals in England to conduct 173 interviews, but also to collect 17 relevant documents and to observe health-care professionals in practice on 24 different occasions. Observations in the field were noted down on paper and the interviews were audio-recorded and later transcribed. We analysed the data by looking for recurring themes related to our interest on user engagement. As a result of analysing and discussing preliminary results among researchers in the team, we identified three dominant themes and a set of subthemes within each.

The first theme indicates that users were more willing to support a new technology if they perceived it as beneficial in terms of improving patient safety or everyday work. The second theme identifies a variety of negative responses, including resistance and workarounds, to the introduction of a new technology. Finally, the third theme points to an increase of sustained use over time as users’ competence improved and changes in software eased the application of the new technology.

Our study shows that new computerised provider order entry and clinical decision support systems were eventually successfully adopted in our case study hospitals. However, our work indicates that the adoption process could be smoother and quicker if more attention would have been paid to user concerns to sustain user engagement in contrast to system installation. In conclusion, we suggest hospitals take a long-term perspective when considering the implementation of new technologies and computerised provider order entry/clinical decision support systems in particular.

Cresswell et al.119

Electronic systems that facilitate prescribing, administration and dispensing of medicines (ePrescribing systems) are at the heart of international efforts to improve the safety, quality and efficiency of medicine management. Considering the initial costs of procuring and maintaining ePrescribing systems, there is a need to better understand how to generate more financial benefits and how to generate them sooner.

In this study, we sought to find out what representatives of different sectors think and what experiences they made about the return of investment from ePrescribing systems in UK hospitals and what lessons can be learned for future developments and implementation strategies within health-care settings.

We conducted international round-table discussions with 21 participants from different backgrounds, including policy-makers, health-care organisations, academic researchers, vendors and patient representatives. We audio-recorded the discussions and transcribed them for a thematic analysis. For this, we identified recurring and emerging themes addressed during these discussions using the qualitative analysis software NVivo10.

There was an over-riding concern that it is difficult to realise direct financial benefits from ePrescribing systems. The underlying reasons included substantial fixed costs of care provision, the difficulties in radically changing the medicines management process and the lack of capacity within NHS hospitals to analyse and exploit the digital data being generated. Any future data strategy should take into account the need to collect and analyse local and national data (i.e. within and across hospitals), agreeing on qualities for comparison to measure progress (i.e. baseline measurements) and clear standards guiding data management so that data are comparable across different health-care organisations.

Cresswell et al.56

Electronic prescribing systems in hospitals might be able to make health care safer and more efficient. However, the health-care professionals in hospital are still trying to find the best ways to use these systems. Sometimes, the people who use these systems (doctors, pharmacists and nurses) find it difficult to use the system the way that it was designed to be used. In these cases, health-care professionals might create a ‘workaround’. Workarounds are new behaviours or new ways of using the electronic prescribing system that allow someone to accomplish their work in a way that was not planned for by the designers of the system. It is important to be aware of the different types of workarounds that may exist to decrease any safety risks and redesign any systems or hospital procedures to better fit the system and the health-care professionals who use it.

This research had three goals: (1) to identify the workarounds that were created and used by health-care professionals when an electronic prescribing system was set up in hospital; (2) to understand why the workarounds happened; and (3) to understand how workarounds might create safety risks. To accomplish these goals, this research studied workarounds at six hospitals in England that had recently set up electronic prescribing systems. To study these workarounds, this research involved interviews with health-care professionals who used the systems, as well as observation of the systems being used in hospital.

This research identified different types of workarounds: informal workarounds (practices that had not been approved by hospital management) and formal workarounds (practices that were understood by leaders in the hospital to be necessary to use the system). The main reason for creating and using a workaround was that certain practices within a hospital are very complicated and diverse. This makes it difficult for an IT system to control all aspects and all possible situations in the prescription process. Sometimes, doctors, pharmacists and nurses would use another electronic system or paper and pen to record information that should have been entered into the electronic prescribing system instead.

This work found that workarounds can cause new safety risks and, at the same time, decrease other risks to safety that may have been introduced by electronic prescribing systems. Instead of seeing workarounds as a bad thing, workarounds should be seen as helpful, for example helpful in the design of future systems that will fit better with the work of the people who use them. This paper emphasises that workarounds might change over time. Understanding the difference between informal and formal workarounds is important in planning for potential risks causes by the use of these systems.

Overall, workarounds should not be seen as a bad thing. They need to be better understood to make sure that the quality of health care does not decrease when an electronic prescribing system is set up in hospital.

Cresswell et al.137

Resistance to antibiotics is a huge threat to health across the world. Resistance to antibiotics means that the medicines used to treat infections (antibiotics) are no longer strong enough to fight against the germs that cause infections. This results in infections that last longer and are more dangerous to patients. As the resistance to antibiotics keeps increasing, it may be unsafe in the future to get routine surgeries. For example, caesarean sections or hip replacements might become very risky procedures.

One way to prevent the increase in resistance is for doctors, pharmacists and nurses to make sure that the antibiotics that they prescribe and give to patients are the most effective antibiotics for the particular patient. This means that the antibiotic targets the specific germ that is causing a patient’s infection. It also means that the antibiotic is prescribed at the correct dose and for the correct length of time. However, choosing the most effective antibiotic is difficult and not straight forward. To help select the most effective antibiotic, some hospitals are turning to information technology systems to provide support and guidance for doctors, pharmacists and nurses.

This paper looked at research from 1997 to 2015 to understand how a certain type of information technology system, electronic prescribing systems, can help doctors, pharmacists and nurses to use antibiotics correctly and effectively in hospital. Electronic prescribing systems help to bring together all of the information available about a particular patient to guide decisions about which medicines to prescribe. These systems also connect doctors, pharmacists and nurses with guidelines for prescribing antibiotics and information about medicines.

This paper provides an overview of electronic prescribing systems that help doctors, pharmacists and nurses to prescribe and give the correct antibiotic. This paper found that many of the electronic prescribing systems that helped prescribe the correct antibiotic are used on particular, specialty wards (such as intensive care), rather than in an entire hospital. Most of the systems were very specific and designed to work in one particular hospital, rather than designed more generally to be able to work in many different hospitals. This paper also provides ideas about how information technology systems can help prescribe correct antibiotics in ways that have not yet been used. This paper suggests that the designers of these electronic prescribing systems will need to be more active in changing the systems to help with the correct prescribing of antibiotics. In addition, hospitals will have to help to tailor their system to meet local antibiotic prescribing guidelines.

There still is a lot that is unknown about how information technology can best support antibiotic prescribing in hospital. This paper calls for more research into understanding how electronic prescribing systems can help antibiotic prescribing. This paper also recommends that, alongside the technical aspects of electronic prescribing systems, people working in the hospital also need to help encourage antibiotics to be prescribed correctly.

Cresswell138

Across the world, there are large investments in health information technologies to try to decrease the number of medical errors and costs, and to increase the quality of care. However, there is still a lot of difficulty when setting up health information technology systems and getting them to work in hospitals. These difficulties can come from the technical aspects of the systems but also from the interactions between the system and the people who use the systems (such as doctors, pharmacists and nurses).

Health information technology evaluations follow the process of a system being set up in hospital and can help to better understand what works when setting up a health information technology system. A good understanding of what works when setting up these systems is helpful for other hospitals that are trying to set up these systems, so they can know what to do and what to avoid when getting their system up and running. Given that evaluations can provide hospitals with information about how to best set up an information technology system, evaluations can also help to avoid any safety risks.

Although these evaluations are really important, they are also complex. This paper describes the different types of evaluations that could be used to better understand how health information technology systems are set up. By describing the different types of evaluation, this paper outlines the necessary steps of an evaluation and also provides advice to people who carry out these evaluations.

During an evaluation is it important to pay attention to these different aspects:

1. the characteristics of the health information technology system

2. the characteristics of the hospital that the system is being set up in

3. how the health information technology system is first brought into the hospital and introduced to the people who use it, as well as what happens when it is being set up

4. any changes that have happened as a result of setting up the system.

It is important that evaluations should always happen during the same period of time that the health information technology system is being set up. Evaluations that happen only after an information technology system has been set up will miss everything that happened during the set-up of the system. Given that there are a lot of challenges during the system set-up, it is important to understand these challenges as well as any challenges or changes after the set-up of the system. Therefore, evaluations should span the entire process of the system set up, rather than just a short period of time at the end when a set-up is complete.

Evaluations should also include different types of understandings, such as overarching statistical understandings as well as more specific and detailed understandings from people in the hospitals who use the system. Evaluations can focus on different parts of the set-up of information technology systems. The most common focus for evaluations are the technical parts of the system and the interaction between these technical parts and the people who use them. Another focus of evaluations is the hospital as whole and understanding how the set-up of a system makes changes on a larger scale, for the whole hospital. It is important for evaluations to focus on a few different aspects, from the technical components of a system to the people who use the system and to how the whole hospital is affected by the system. Evaluations also help to learn about what happens during the set-up of a system within one hospital and then to compare what happens within other hospitals that are also setting up systems.

Mozaffar et al.47

Parts of this text have also been reproduced with permission from Mozaffar et al. 47 with permission from BMJ Publishing Group Ltd.

Hospital ePrescribing systems offer a wide range of benefits to improve patient safety. However, like other attempts to implement new hospital health information technology, they may also introduce new areas of risk. Despite efforts made by many other researchers in identifying these risks, the development and use of ePrescribing systems is still leading to numerous unintended consequences, which may undermine improvement and threaten patient safety. These negative consequences need to be analysed in the many stages a system goes through including design, implementation and use. This study, therefore, aimed to understand the roots of these reported threats and identify possible approaches to avoid or mitigate such threats.

We analysed six English hospitals over an extended period of time to learn about the implementation and adoption of electronic prescribing systems. To this end, we collected data that included interviews with people affected by the system, observations of implementation meetings and system use, and a collection of relevant documents. Our data set included 214 interviews, 24 observations and 18 documents. After analysing each hospital individually, we compared the findings across hospitals to find recurring patterns.

Based on our data, we found a range of reasons that can cause unintended safety threats. We built a taxonomy of factors, which means that we sorted these reasons into groups of different classes, underlying these safety threats. The first class of factors encompasses poor system design, including lack of support for complex medication administration regimens (such as infusions, lack of effective sharing of data between different systems, and lack of effective automated decision support tools, such as automated alerts for drug intolerances). Another class of factors covers the inappropriate use of systems, including too much reliance on the system and introduction of workarounds. Finally, a third class of factors causing unintended safety threats include inadequate implementation strategies that result from partial roll-outs, simultaneously running new electronic systems and old paper-based systems, and a lack of appropriate training. We have identified a number of system and organisational strategies that could potentially avoid or reduce these risks.

Imperfections in the design, implementation and use of ePrescribing systems can give rise to unintended consequences, including safety threats. Hospitals and suppliers need to implement short- and long-term strategies in terms of the technology and organisation to minimise the unintended safety risks.

Cresswell et al.139

Electronic health records and other health information technologies are used across the world in hospital care and primary care. However, it is still unclear how these information technology systems directly help patient care and the health of populations. It is important to outline strategies for the future about how these systems will be used in health care and, in outlining these strategies, is it important for national governments to be involved. This paper offers six suggestions for governments to draw the most value and financial benefit from these systems.

Role guide: National Institute for Health and Care Research ePrescribing patient public involvement, updated: January 2014

National Institute for Health and Care Research ePrescribing patient and public involvement group: general guidance for expenses claims