Investigating innovations in outpatient services: a mixed-methods rapid evaluation

Introduction

The 2019 NHS Long Term Plan set out an ambitious programme to fundamentally redesign outpatient services in England. 1 This was in part a response to the steady rise in outpatient attendances in the UK over the previous decade, which outstripped population growth and was not matched with a commensurate increase in workforce or system capacity. 2 The number of outpatient appointments increased by two-thirds between 2008–09 and 2019–20 to 125 million a year. 3 This was the largest increase in activity of any hospital service and, given the context of chronic financial and workforce constraints and system pressures, long wait times, delayed appointments, and rushed consultations, have all become increasingly common, frustrating patients and staff alike. 4

After a drop during the COVID-19 (coronavirus disease discovered in 2019) pandemic, total volumes of outpatient activity have recovered to approximately where they were beforehand,5 but challenges have been exacerbated with the number of patients waiting for consultant-led elective care rising to 7.6 million by June 2023. 6,7

Traditional outpatient service models have relied on face-to-face consultations, which can require repeat hospital visits that prolong uncertainty and potentially waste patient and staff time. 2 As the number of appointments has grown, so too has the number of those that are unattended without prior cancellation [‘did not attend’ (DNA)] which account for nearly 8 million booked appointments each year, which is 6% of all appointments. 8 The high rate of missed appointments has led clinics to overbook, exacerbating problems of poor patient experience.

These inefficiencies have made improving the value of outpatient care a key priority for the NHS. In 2018, the Royal College of Physicians declared that the ‘traditional model of outpatient care is no longer fit for purpose’ and that the NHS must change how it commissions and delivers the service if it is to be sustainable over the long term. 2 It also aims to create a more ‘holistic’ service for patients who do not need to attend hospital, by making better use of primary, secondary and community services, and helping people to manage their own health. 9 As part of its outpatient redesign programme, the NHS Long Term Plan sought to avoid one-third of face-to-face outpatient appointments by 2024 – estimating that this would save the NHS £1.1bn per year (and patients 30 million visits to hospital) by streamlining service delivery through expanded technology at each stage of the pathway. Similar strategies have also been rolled out in Wales and Scotland to expand digital access and shift more care to community settings. 10,11 More recently, NHS England’s (NHSE) 2023–24 Operational Planning Guidance asked health systems to reduce follow-up outpatient attendances by 25% from 2019 to 2020 levels by March 2024. 12

Across the outpatient care pathway, a broad range of innovations have and are being pursued to better manage outpatient care and reduce unnecessary appointments.

Such innovations include referral optimisation (specialist triage of outpatient referrals), Advice and Guidance, Patient-Initiated Follow-Up (PIFU) and video consultation.

Through Advice and Guidance, general practitioners (GPs) can access specialist advice about a patient, thereby allowing a timely response to patients and reducing the need for referral to outpatient services. Expanding primary care access to specialist Advice and Guidance is a central part of NHSE’s delivery plan for tackling elective care backlog. Referral optimisation of outpatient referrals (e.g. rapid triage) involves GPs accessing specialist advice to discuss the most appropriate care with the aim to triage outpatient appointments more effectively, get the right support and reduce ‘unnecessary’ appointments. PIFU aims to give more flexibility and choice to patients over the timing of their care and allow them to book appointments as and when they need them, rather than follow a fixed or standard schedule (e.g. every 6 months).

Understanding which interventions are most effective and the factors that contribute to their success is limited by several factors: (1) innovations being delivered in conjunction with other service changes; (2) differences in delivery across specialties and conditions; (3) the precise components of the interventions often not being reported and (4) variation in the outcomes studied to judge effectiveness.

However, available evidence shows that digital innovations such as teleconsultations and virtual appointments can be effective in reducing missed appointments and lead to shorter consultation times. 13–15 Remote monitoring services have been found to be related to fewer outpatient visits across several specialties. 16–18 Innovations such as rapid triage services (e.g. telephone assessment clinics) have been found to reduce referral waiting times. 19,20 Similarly, direct or rapid access clinics have been shown to reduce waiting times and can alleviate requirement for face-to-face clinic attendances. 21–23 The introduction of non-specialist clinics and expanding of clinical support roles (e.g. advanced care practitioners, nurse practitioners) has also been linked to reduced waiting times24 and increased appointment availability. 25 Other innovations, such as clinic administration and scheduling systems, show variable outcomes, as these tend to be linked to the context in which they have been implemented.

The aims of outpatient transformation efforts have been varied but coalesce around several common themes, including:

• Making better use of clinical space and staff time.

• Making better use of wider professional teams.

• Integrating primary, secondary and tertiary care.

• Co-ordinating care for people with multiple health conditions.

• Increasing patient satisfaction, empowerment and convenience.

• Reducing unnecessary in-person appointments.

• Increasing savings for the NHS and improving cost effectiveness.

• Reducing greenhouse gases and other pollution through reduced travel.

• Decreasing waiting times for patients and their carers.

Study aims

This study aimed to identify innovations in outpatient services implemented in recent years in the English NHS and to carry out a rapid evaluation of one of the most important but little studied recent innovations: PIFU. The identification of innovations included a review of published literature to understand the breadth of system innovations and their potential impacts. This was followed by detailed data analysis of national outpatient activity to identify hospital trusts or clinical specialties where there were notable and recent changes in activity. The evaluation of PIFU was a rapid mixed-methods study investigating its implementation and impact on the use of hospital services and on patients and staff.

The specific aims were:

1. To understand the scope and breadth of interventions being pursued to improve efficiency in outpatient service delivery, and to understand key evidence gaps and future research needs.

2. To identify trusts and/or specialties where there is quantitative evidence of a notable change to outpatient activity, for example, a reduction in the numbers of attendances, or a substitution between different modes of attendances (e.g. from face-to-face consultation to teleconsultation).

3. To undertake interviews of selected trusts and specialties thus identified, to investigate whether changes in their outpatient activity were the result of specific innovations in care management.

4. To conduct a mixed-methods evaluation of PIFU, considering its implementation, impact and the experiences of patients and staff.

Research questions

The evaluation included four workstreams associated with each of the four aims with the following research questions (RQs):

1. RQ1. What interventions to improve efficiency and effectiveness of outpatient care have been studied? Where are the greatest gaps in evidence? (workstream 1)

2. RQ2. Can we identify from routine NHS data any healthcare organisations or specialties that have exhibited significant and sustained changes in outpatient activity that might suggest the influence of service innovations? (workstream 2)

3. RQ3. Can any identified changes in outpatient activity be linked to specific service innovations? (workstream 3)

4. RQ4. What is the current evidence relating to the implementation and impact of PIFU? What is the impact of PIFU on measures of outpatient activity? What are the experiences of patients and staff? (workstream 4)

Further, more detailed RQs were developed for the PIFU evaluation in workstream 4, which are stated separately in Aims and objectives.

Motivation for the study

This study was carried out by members of the National Institute for Health and Care Research (NIHR)-funded Rapid Service Evaluation Team (RSET) and arose following recommendations from the RSET Stakeholder Advisory Board (SAB). The initial aim for workstream 4 (RQ4) was to evaluate one of the promising innovations identified in workstreams 2 and 3. However, as the project progressed, we were approached independently by the Outpatient Transformation and Recovery Programme team at NHSE to undertake an evaluation of PIFU. Moreover, it was already becoming clear from the earlier workstreams that PIFU was the most prominent innovation in outpatient care within the English NHS. For these reasons we devoted workstream 4 to an evaluation of PIFU.

Report structure

This report has four main chapters. Chapter 1 is an overview of the clinical and policy context for outpatient services in the English NHS together with the project aims and a broad overview of the methods employed, including use of patient involvement and the approach to equality and diversity. Chapter 2 combines workstreams 1, 2 and 3, describing the work to identify innovations in outpatient care. Chapter 3 focuses on the methods and findings from the evaluation of PIFU (workstream 4), with an evaluation guide described in Chapter 4 and a discussion in Chapter 5. In Chapter 6 we outline recommendations for decision-makers and further research arising from all workstreams.

Study design

This study employed mixed methods and was divided into four workstreams (see Background and context). Workstream 1 was designed as a literature review to provide context for the rest of the study, workstream 2 was designed in parallel as a quantitative analysis of routine patient data. Findings from workstream 1 informed the final analysis in workstream 2 leading to engagement with healthcare providers in workstream 3. Workstream 4 arose from a combination of findings from workstreams 1 and 3, alongside engagement with external stakeholders. These links between the different workstreams are illustrated in Figure 1.

FIGURE 1.

How the different workstreams interact. The arrows represent the links between the different workstreams.

Workstream 4 was designed separately as a mixed-methods evaluation with an evidence review and two phases, with phase 1 providing interim findings that informed how we approached phase 2. For further details on the designs and methods for workstream 1, see Rapid scoping review of outpatient innovations. For workstreams 2 and 3, see The development and use of a methodology to detect, rank, and investigate positive changes in outpatient activity, and for workstream 4 see Methods for the evaluation of Patient-Initiated Follow-Up.

Contact with stakeholders and advisers

The initial idea for evaluating outpatient services was put forward by the NIHR RSET SAB. This group comprised health and social care professionals, patient representatives, data analysts, voluntary service representatives and service managers. The SAB reviewed progress at our annual board meetings and made suggestions for taking forward.

Distinct from the SAB, a formal Project Advisory Group (PAG) was established to oversee progress. This group reviewed the protocol (including amendments) and provided constructive feedback on all the outputs during the course of the project. They met by video conference once at the start of the project and then at intervals as appropriate, including at key points where decisions needed to be taken by the research team. Where meetings were not possible due to other commitments, we engaged with the group by e-mail.

The group included members with various types of experience and responsibilities, including clinical and academic expertise. The group also included one patient representative (see Patient and public involvement and engagement).

The responsibilities of the group included:

• providing context, advice, and challenges to the research team, based on knowledge of outpatient service policy and delivery

• helping the team to prioritise in addressing the RQs

• advising on service issues around outpatient activity and related data collection

• guiding the research team in seeking further expert advice

• helping to interpret outputs, for example, activity patterns by specialty or the long list of trusts, with potential innovations from workstream 2

• supporting the research team in scoping workstream 4.

Workstream 4 was shaped in partnership with the Outpatient Recovery and Transformation Policy and Strategy team at NHSE, with whom we had regular meetings to report ongoing findings. The team also provided advice, particularly about the data and engagement with the service.

The identity of the study sites was not revealed to any of the stakeholders or advisors.

Patient and public involvement and engagement

Patients and the public were actively involved in:

• project design

• project management

• developing participant information resources

• contributing to writing up and sharing findings.

Patient and public involvement and engagement (PPIE) benefited the project in the following ways:

• ensuring the study focused on issues of importance to service users

• ensuring that this focus was reflected in the aims, objectives and RQs

• ensuring that these were operationalised suitably in the approach to data collection and analysis, and

• ensuring that the findings were disseminated effectively and in a manner that is meaningful to patients, carers and the public.

Patient and public representatives reviewed the protocol and contributed to the design of this study. For example, two patient representatives from the NIHR RSET (PPIE) panel reviewed the initial protocol in November 2020 following a virtual meeting in which they were given an overview of the project and the opportunity to ask questions.

We recruited a patient representative with lived experience of outpatient care to be a member of the PAG to provide perspectives on their experiences of care and different system innovations to guide the work.

The patient representative [Jenny Negus (JN)] participated in all PAG meetings and commented on study documents, such as the protocol’s Plain language summary. They were offered appropriate training and support, for example, on how to effectively participate in meetings. The team budgeted to support them in all these activities. To support effective participation, the team ensured that documents relating to meetings and events were distributed in a timely fashion (e.g. a week in advance). Also, a member of the team [Pei Li Ng (PLN)] was identified as the primary contact with whom the patient representative could raise any issues or concerns.

Patient and public involvement and engagement feedback contributed significantly to several aspects of the project. This included the development of the patient interview materials (including the participant information sheet, consent form, and topic guide) which were submitted as part of the ethics application. We also conducted a mock patient interview with a member of the PPIE group, which informed the approach. Prior to publication, the PPIE representative reviewed and provided feedback on project outputs (e.g. PIFU evidence review and explainer,26 slide packs for NHSE).

Equality, diversity and inclusion

Many outpatient innovations would have important implications for equality, diversity and inclusion (EDI), for example, if they rely on engaging with technology or patient activation.

For the analysis in workstream 2 (see The development and use of a methodology to detect, rank, and investigate positive changes in outpatient activity), we were analysing changes in outcomes within providers over time, so we decided that corrections for patient characteristics between or within different populations did not need to be applied. Where notable changes in outcomes are driven by changes in population characteristics it is not appropriate to adjust for different patient groups, otherwise they would remain undetected.

The PIFU evidence review (see Rapid Patient-Initiated Follow-Up scoping review) sought to understand outcomes and experiences of PIFU across different patient groups, such as age, gender, ethnicity, and socioeconomic status; however, many studies did not report these findings. We identified further understanding of the impact of PIFU across different population groups as a crucial area for further research.

In phases 1 and 2 of the qualitative work in workstream 4, we aimed to sample sites based on several criteria in order to get a diverse sample. This included the levels of ethnic diversity and deprivation relative to the average population and other demographic factors such as age.

We identified staff interviewees primarily by role and aimed to include a variety of perspectives including clinical, operational, and administrative at different levels of the organisation (specialty and trust-wide). We considered the EDI implications for PIFU in the design of the topic guides and included questions which focused on understanding the opportunities and barriers for PIFU in access for different patient groups.

For the patient interviews, we intended to sample patients from a range of backgrounds to ensure we had a representative sample and to identify whether there were differences in experience depending on patient demographics. Initially we aimed to sample patients by age, gender, ethnicity, and socioeconomic status. We included optional demographic questions at the end of the patient interviews. Due to numerous challenges in setting up the patient interviews, the approach became more pragmatic and we sampled primarily based on specialty. Although we did capture demographic details of the interviewees we spoke to, given the low number of patient interviews (four) we were unable to comment on differences by patient background.

Methods for the evaluation of Patient-Initiated Follow-Up contains further details of how EDI influenced the selection of study sites and patient interviewees.

To evaluate EDI implications of PIFU in the quantitative analysis, we asked for specific data from local sites since the national data were not going to allow us to look at these aspects. Further details are provided in What is the impact of Patient-Initiated Follow-Up on health inequalities and how is this being measured?.

The research team consisted of researchers from two organisations [the Nuffield Trust and University College London (UCL)]. The team was multidisciplinary in nature and team members differed in seniority. The team comprised a mix of backgrounds in relation to gender and ethnicity. When members of the team had to leave, due to a variety of commitments, we ensured, as far as possible, they overlapped with their replacements so they could receive the necessary support during the handover.

Differences from the project protocol

Any changes from the original protocol were discussed with the advisory group.

The final conduct of workstream 1 differed from the project protocol in the following ways:

• We stated that we would review studies published between 2010 and 2022. However, to manage scope and ensure we were focusing on more current innovations, we changed the start year to 2015. This is explained further in Methods overview.

• We also stated we would summarise existing evidence of the impact of outpatient interventions, identify factors that might support or hinder the implementation and identify gaps in evidence. The final scope of the review did not include this. The revised aims and RQ are described in more detail in Rapid scoping review of outpatient innovations.

The final conduct of workstream 4 differed in the following ways:

• We stated we would analyse local patient-level data from hospitals that can identify patients on PIFU pathways to compare their outcomes with patients from hospitals that have not applied PIFU within the same specialties. None of the hospitals we engaged with were able to supply such data, either because they could not identify PIFU patients within their electronic patient administration systems (PAS) or because of the local practicalities of preparing such data. We considered that a more practical way of using local data was to analyse differences in the use of and engagement with PIFU between different individual characteristics where we could use aggregated data that could be easily prepared by the local sites.

• We stated that we would hold a workshop with three to four ‘late-adopter’ sites that show limited evidence of moving patients on to PIFU pathways. Due to challenges with engagement, the focus of the workshop was broadened to focus more generally on the barriers and facilitators to PIFU implementation. Sites represented at the workshop were therefore at varying stages of PIFU implementation. We considered this an advantage since focusing on late adopters assumes that these issues are specific to such organisations.

• We stated that we would interview patient advocacy groups, such as Healthwatch and National Voices, but this did not take place as we thought it best, given the time available, to focus on interviewing individual patients who had a relationship with the services we had selected for staff interviews.

What this section adds

This section describes a rapid scoping review of the published literature on innovations in outpatient care that was carried out in the early stages of the project. Adopting an evidence-mapping approach, the review identified which innovations have been studied, their purposes and potential benefits. This has helped us identify where there were gaps in the evidence and inform the choice of innovation to study in workstream 4.

Introduction

In this scoping review we sought to address RQ1:

• What interventions to improve efficiency and effectiveness of outpatient care have been studied? Where are the greatest gaps in evidence?

Across the outpatient care pathway, a broad range of innovations have and are being pursued to better manage outpatient care and reduce unnecessary appointments, but there is limited understanding of which interventions are most effective and what factors contribute to their success.

The purpose of this review was to provide context for the wider study of outpatient services and, in particular, help us to identify innovations that would be suitable to evaluate in workstream 4. Assessing the effectiveness of innovations in improving outpatient service delivery or examining outcomes were beyond its scope.

Methods

We adopted an evidence-mapping approach,27–29 followed a pre-defined protocol and, where applicable, have reported the methods and findings according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. 30

Findings

The initial searches identified 5249 papers, of which 149 were included for review (Figure 2). A summary of the characteristics of these studies is provided in Appendix 1, Table 13. A full list of studies is provided in Report Supplementary Material 1.

FIGURE 2.

Flowchart of study selection.

Discussion

What this section adds

• Indicator saturation (IS) methods can be successfully applied to large numbers of multiyear, clinical service-level time series derived from national-scale hospital data sets.

• These methods simplify complex time series into relatively few best-fit lines and can identify change or break points: points in time where performance on an activity measure appeared to change.

• The identification and prioritisation of exceptional potentially positive changes was more challenging, in part due to the wide scope of the aim.

• We nevertheless found a small number of units where potentially positive changes in performance may have been tied to specific service changes.

• Investigation with trusts raised questions about data quality and highlighted the importance of local data validation.

• With more tightly focused study aims, IS methods could be more widely used to detect signals of changed performance within large health service data sets.

Introduction

National appointment-level outpatient data have been collected in England since 200344 and is made available for research and other secondary uses as Hospital Episode Statistics (HES) and Secondary Uses Service data (typically the latter for NHS organisations, the former for researchers and other users). After 20 years, while there are concerns about aspects of data quality,45,46 the data set has proven to be a substantial resource for service evaluation and research, although one which could be used more extensively.

One challenge comes from its size. In the most recent financial year 2022–3, it reported on 123.8 million appointments for 23.3 million people, in 170 clinical specialties, across 295 acute care providers (mostly NHS hospital trusts).

Rapid scoping review of outpatient innovations outlined approaches to reduce rates of unnecessary attendances, make better use of virtual consultations, and reduce the number of DNAs among other strategies to improve outpatient services. In 2019, the SAB recommended that we might prioritise evaluating innovations in outpatient service delivery as an important policy area with which secondary services were grappling.

One innovation of NIHR RSET has been the approved access to HES outpatient (HES OP) data collection for NIHR RSET programme work. This offered an opportunity to use data-led techniques to identify potential innovations that might benefit from evaluation.

In theory, where an innovative service change had been implemented, and the innovation had a direct or indirect positive effect on a measure of outpatient activity, it is possible that we might be able to detect the change retrospectively in HES OP data with an appropriate analytical methodology.

In this workstream we aimed to develop such a methodology. The anticipated outcome was a small list of clinical units with exceptional positive recent performance. We would then attempt to contact the relevant units to investigate potential reasons for the detected changes. Where innovative service changes were plausibly suggested as having caused or contributed to the detected changes, we would consider one or two units for more detailed mixed-methods evaluation work.

Hence, we aimed to implement the ‘positive deviance’ approach to improving quality of health care. This methodology, as described by Bradley et al. ,47 aims to identify exceptional performance that already exists within healthcare organisations, and follows this with detailed investigation, testing, and finally dissemination of evidence about best practice.

The aim was that the methodology should be able to prioritise between a potentially very large number of signals of positive change in HES OP data, covering different outpatient activity measures, and multiple clinical specialties within English NHS hospital trusts.

However, methods to achieve the above are not well defined. So, as well as being a potential prelude to evaluation, we hoped that this workstream would help to define new methods that might be applied perhaps not just to outpatient data, but more generally.

We took as a starting point the approach of Walker and colleagues. 48 In their 2019 paper, they outlined a novel approach to automatically detect the timing and magnitude of changes apparent in prescribing data. Noting that previous work to quantitatively assess implementation of new practices ‘has typically relied on measuring change at the level of a whole population, using techniques such as interrupted time series analysis or static measures of variation in care at one point in time’, they presented indicator saturation (IS) techniques as robust, systematic methods to find meaningful changes in data over many institutions, over any general time. These methods – adapted from econometric modelling – were able to detect ‘structural changes’ or ‘breaks’ in time series data ‘without imposing an intervention or change date a priori.’

Ultimately, in part due to the decision to focus the evaluation efforts on PIFU, we did not achieve all the aims. The analysis involved was complex, and we were not fully able to test the consequences of choices made within the context of a rapid evaluation study. However, we learnt several valuable lessons and discuss these at the conclusion of this section.

Aims

We aimed to develop a methodology to produce a ranked list of units (a unit being a clinical specialty at an English NHS hospital trust) where there was evidence of apparent exceptional positive improvement made in performance on an outpatient activity measure. We also aimed to identify whether positive changes identified may have been linked to specific service changes in trusts.

We addressed RQs 2 and 3 (see Background and context):

• Can we identify from routine data any trusts or trust specialties that have exhibited significant and sustained changes in outpatient activity that might be indicative of the impact of innovations in service design? (RQ2)

• Can any of these identified changes in outpatient activity be linked to specific innovations in care? (RQ3)

Methods

General approach

Figure 3 provides a summary of the approach. The analysis aimed to act as a filter and prioritisation tool to select approximately ten time series from upward of 20,000.

FIGURE 3.

Stages of analysis.

Results

Summary of implementation of approach

The flow chart in Figure 4 summarises the implementation approach of the analysis, through numbers excluded at each stage.

FIGURE 4.

Flow chart summarising numbers of time series and exclusions. The stages are grouped under three categories, depending on which outpatient activity measures were relevant (left-hand side).

We extracted outpatient activity attendance time series data for 10,017 units covering 190 trusts and 153 treatment specialties (Table 2). We excluded 4329 (43.2%) units with low activity volumes (fewer than 100 mean attendances per 28-day period in the most recent calendar year 2019). A further 354 units (3.5%) were excluded as they had one or more 28-day periods (between 2017 and 2019) with zero attendances.

TABLE 2 - Summary of outpatient activity measure time series constructed and exclusions made pre-modelling

Summary of stage of time series and exclusions	Units excluded	%	Units remaining	%
Total units initially extracted	–	–	10,017	(100%)
Primary exclusions (based on attendances)
Low recent volumes (< 100 per 28-day period)	4329	(43.2%)	5688	(56.8%)
Zero attendance 28-day period	354	(3.5%)	5334	(53.2%)
Total after primary exclusions	4683	(46.8%)	5334	(53.2%)
Secondary exclusions (measure-specific)
Attendances: (no further exclusions)	0	(0%)	5334	(100%)
DNAs: < 5 per 28-day period	205	(3.8%)	5129	(96.2%)
Tele-attendance: < 5 per 28-day period	3128	(58.6%)	2206	(41.4%)
Follow-up to first attendance ratio: <5 per 28-day period	140	(2.6%)	5194	(97.4%)
After primary and secondary exclusions: sum of four measures for modelling	–	–	17,863	–

There were 5334 units (53.2%) remaining for potential modelling. For each of these units we constructed three additional time series, one for each of the other outpatient activity measures (DNA, tele-attendance, follow-up to first attendance ratio).

Secondary exclusions were made, such that 17,863 time series were finally taken forward for modelling. The tele-attendance time series were the most likely to be removed at this point: only 2206 time series were taken forward for modelling (vs. over 5000 each for each of the other three time series).

Application of indicator saturation to time series – detecting break points and outliers

Indicator saturation modelling was successfully applied to 16,897 of 17,863 time series. This was the case for 100% of attendance, 95.7% of DNA, 97.4% tele-attendance and 86.8% of the follow-up ratio time series.

Table 3 summarises characteristics of the resulting IS-modelled time series. DNA rates generally fell over the 7-year period. The opposite was the case for tele-attendances, and follow-up to first attendance ratios were stable.

TABLE 3 - Summary of IS-modelled time series

Note

Percentages and means are of times series successfully modelled, unless specified.

		Activity measure
		Attendances	DNA	Tele-attendance	Follow-up to first attendance ratio
Number time series, modelling attempted		5334	5129	2206	5194
Number time series, successfully modelled (% of attempted)		5334 (100%)	4908 (95.7%)	2148 (97.4%)	4507 (86.8%)
Mean value start baseline period (12 December 2012) (interquartile range)		746 (133–919)	8.6% (5.3–11.1%)	2.5% (0–9.0%)	3.7 (1.3–3.8)
Mean value start search period (7 January 2015) (interquartile range)		1145 (246–1370)	7.9% (4.9–10.0%)	3.6% (0.0–3.3%)	3.5 (1.3–3.7)
Mean value end search period (31 December 2019) (interquartile range)		1077 (244–1300)	7.2% (4.4–9.0%)	10.1% (1.8–14.7%)	3.6 (1.2–3.7)
Mean number of changes (break points) during baseline and search period (12 December 2012 to 31 December 2019)		2.14	1.83	1.68	1.72
Mean number of changes (break points) during search period (7 January 2015 to 31 December 2019)		1.46	1.27	1.46	1.18
Number of series (%) with N changes (break points) during search period (7 January 2015 to 31 December 2019)	N = 0	1955 (36.7%)	2160 (44.0%)	696 (32.4%)	2185 (48.5%)
	N = 1	1010 (18.9%)	683 (13.9%)	433 (20.2%)	638 (14.2%)
	N = 2	1440 (27.0%)	1330 (27.1%)	676 (31.5%)	1072 (23.8%)
	N = 3	452 (8.5%)	426 (8.7%)	174 (8.1%)	288 (6.4%)
	N = 4	200 (3.7%)	137 (2.8%)	98 (4.6%)	157 (3.5%)
	N = 5 +	277 (5.2%)	172 (3.5%)	71 (3.3%)	167 (3.7%)
Number with any outliers during search period (2015–9) (%)		168 (3.1%)	177 (3.6%)	329 (15.3%)	345 (7.7%)
Mean number of outliers during search period (2015–9) for time series with any outliers		4.0	2.9	3.5	2.8

The IS modelling resulted in 1.7–2.1 break points on average (depending on activity measure) during the 7 years of the entire period, and 1.2-1.5 break points during the 5 years of the search period.

During the 5-year search period many time series (up to almost half, depending on activity measure) had no modelled breaks (and so were modelled as a single trend line). The next most likely outcome was two breaks (three trend lines). A minority of time series (3.3―5.2%) had five or more break points (six or more trend lines).

Modelled outlying points were common for tele-attendance time series (15.3% had at least one), but less so for the other activity measures (e.g. only 3.1% of attendance time series had at least one).

Figure 5 shows trend lines for two example time series (one for attendances, the other DNAs), overlaid on the data points, including a brief description of the key features of each. Appendix 2, Figure 24 includes further examples (two for each activity measure).

FIGURE 5.

Two examples of IS-modelled time series, with key features noted. Points are 28-day period values during the baseline and search periods (late 2012 to end-2019). Red lines are IS-modelled trend lines. Outlying data points are shown in orange, attached to a hashed vertical line (attendances example).

Classifying units via their primary positive change periods – creating summary metrics and ranking

Summary ranking metrics

There were 3316 time series to rank, based on each series’ PPCP. Figure 6 summarises the ranking metrics we developed. The figure includes for each metric a description of transformations made and the aims for each.

FIGURE 6.

Summary of ranking metrics developed. Examples are given with respect to a measure where a fall in value is ‘positive’. PPCPs are shown as bold lines.

Many of the metrics we created included linear transformations (i.e. linearly scaling minima to maxima – or vice versa – on to a 0-to-1 scale), but for the length metric we used an inverse function and this risked overpromoting very short length changes. The implications of this are visible in the following section but is mentioned here to make the point that the metrics outlined in this section were a work in progress, and may have been changed in subsequent iterations.

Combining metrics: ranking

We ranked time series by combining sets of metrics. Appendix 2, Section 6 outlines a series of iterations of ranking attempts we made. At the time the study’s focus moved to the evaluation of PIFU, we were testing a relatively simple ranking approach, combining four metrics (magnitude, length, persistence, and end percentile) as a simple sum, with equal weights. Development of the general approach to ranking stopped at this point. Figure 7, however, shows a representation of the ranking outcome at this point. It shows 400 (of 3316) units as ranked by the third approach: 200 from the top of the list, 100 from the middle, and 100 from the very bottom.

FIGURE 7.

Representation of a subset (400 of 3316) unit time series from top, middle and bottom of ranked list (based on third-attempt method). Key included for activity measure and ranking and for metric score colour scales (deeper colours being better). The first four metrics listed were those used in the final approach ranking.

One observation is that there was not an equal distribution of the four activity measures across the ranking. For example, while making up 27% of all time series being ranked, only 14% of the top 200 were follow-up to first attendance ratio time series, and 42% in the top 200 were tele-attendance time series (which made up 29.5% overall). At the bottom of the ranking, 91 out of 100 time series were for tele-attendance measures. We would not necessarily expect a completely even distribution of activity measures throughout the ranking, but we might have investigated these discrepancies before the next iteration had there been one.

Briefly, other observations include:

• The impact of having a non-linear length metric transformation is clear. Length contributed relatively little to the ranking for the majority of time series, except for the very shortest lengths (these were one or two 28-day periods), which were typically to be found promoted to the top of the ranking.

• The correlations between recency and persistence are visible, with similar patterns of light/dark occurring in each.

Focus on Patient-Initiated Follow-Up

Selecting time series for a single activity measure: follow-up to first attendance ratio

The starting point was 906 time series describing the follow-up to first attendance ratio activity measure and their metrics, developed for the general ranking approach. The focus on a single activity measure greatly simplified the process of finding a small number of specially performing units.

We decided to implement a specific set of filters to reduce the number of time series further. These are outlined in the flow chart Figure 8. The reason for these filters was to prioritise more sustained improvements, better modelling performance and better relative performance versus other trusts in the clinical specialty. Applying these filters left us with 262 follow-up to first attendance ratio time series.

FIGURE 8.

Numbers of follow-up to first attendance ratio time series available and excluded in filtering and final time series selection.

With the number reduced so considerably, we manually ranked the subjective quality of the identified change on a scale of 1–10, and were left with 36 time series that scored 10.

Three researchers (JS, TG and JA) further considered the top 36, and manually selected 9 of these that we considered to be most promising as potentially being the result of a service change or innovation. We considered, among other things:

• The pattern of activity before the PPCP (whether the time series had been flat, rising or jumped upwards).

• The volume of activity in the service, compared to others.

• The context in terms of the other activity measures for the unit.

• How sustained the changes had been into 2020 and 2021.

Figure 9 shows the time series of the nine services that we identified. All services were from different NHS trusts.

FIGURE 9.

The selected nine services with significant falls in follow-up to first attendance ratios [each from a different NHS trust, labelled (a)–(i)]. Red lines are IS-modelled trend lines. The PPCP is shown in bold.

Service interviews

We received five responses from nine hospital trusts contacted. One trust was unable to participate due to pandemic-related service pressures. A second – an anticoagulation service – responded that the finding had been recognised and understood locally and was likely due to a change to anticoagulant drugs which no longer needed regular blood test monitoring. This change had taken place in 2017, broadly matching the start of the changes identified but was out of scope as an innovation for further study. Note that we had been made aware of this change by one of the advisory group members, and we had made specific attempts to reduce the relative ranking of anticoagulant services in the earlier general ranking methodology (see Appendix 2, Section 6).

Three trusts welcomed a call and suggested one or more relevant members of staff who would be able to talk with us.

The outcome of the interviews was mixed. Table 5 summarises key reflections from the three unit interviews.

TABLE 5 - Summary of findings of interviews

	Unit 1	Unit 2	Unit 3
Unit clinical specialty	Gastroenterology	Obstetrics	General internal medicine
Organisation type, location	Teaching hospital trust, Midlands	Multi-hospital trust, South East	Teaching hospital trust, London
Interviewee role	Consultant gastroenterologist (in position > 8 years)	Operational Lead Women’s Health, and Clinical Director of Women’s Health (both in position < 18 months)	Head of transformation (in position < 3 years)
Change in follow-up to first attendance ratio found and discussed	Reduction from 2.3 to 1, from 2017 to 2020 (Figure 9e)	Reduction from 3.1 to 1, from 2015 to 2020 (with accelerated fall in 2018) (Figure 9f)	Reduction from 5.5 to 2, from 2016 to 2020 (Figure 9g)
Trend recognised at trust?	Yes, and reflects a change in practice	Not picked up at the trust	No. Did not reflect trust’s own data; no such large falls known in any medical service
Suggested main cause of change	Clinical assessment service, introduced in 2017. Aimed to refer patients directly to tests without initial consultation. A modest trial had shown a large fall in consultations before discharge to general practice	No clear explanation identified. Suggestion of possible recording artefact, especially given declining birth rate (and yet first attendances more than doubling according to HES OP)	N/A
Additional notes	Other initiatives also noted: - consultant peer review of patient lists - increase in telephone follow-ups - increase in use of clinical nurse specialists

In only one of the three units, a gastroenterology service, did the interviewee recognise the finding we presented of falling numbers of follow-up attendances compared to first attendances over time. This was cautiously attributed to a clinical assessment service that started in 2017 (and which a local trial had shown to reduce numbers of consultations before discharge).

Interviewees of the other two units were more surprised by the findings; indeed one (from a general internal medicine service) consulted internal trust data during the call and could find no such large fall in ratios in any medical specialty during the relevant period. This remains unexplained. The other unit interviewees (from an obstetric service) suggested that there were data recording problems, given that locally declining birth rates were at odds with rising numbers of first attendances as recorded in HES OP data.

No unit discussed PIFU as an intervention, and the investigation into these sites ended.

Discussion

The aim of this section’s work was twofold. Firstly, to develop a quantitative methodology to apply to national hospital outpatient data, the output of which would be a small, ranked list of clinical units where there was evidence of outstanding improvements in an aspect of outpatient activity data, potentially indicating the impact of a service change. Secondly, to investigate a small number of the detected improvements, with a view to identifying one or two potentially innovative service changes that we might evaluate.

The wider study changed course as the analytical work was in progress, and as a result we did not achieve all the aims. The work changed to support an evaluation of the implementation of PIFU, and this halted work on a generalised quantitative methodology for identifying positive changes in outpatient data. The final process had more manual – and subjective – input than was envisaged, and arguably might have been better supported with more well-established quantitative and statistical methods, given the increased simplicity of the analytical aim.

Of a small number of clinical units, we identified as having unusually large falls in numbers of follow-up attendances versus first attendances (an impact we expected to be connected to successful implementation of PIFU), we were able to interview three. Two did not recognise the data findings and had no explanation for the apparent improvements we found. The third recognised the changes and attributed them to a non-PIFU-related service change. A fourth (not interviewed) identified the fall as having been connected to a change in medication use. While these results are a potential measure of success for half of the units from which we received further information, numbers are too small to generalise, and we cannot exclude the possibility that there was no direct connection between data findings and the suggested causes.

Nevertheless, this work did not therefore feed into the PIFU evaluation.

We reflect below on the analytical work in the context of other studies, and its strengths and limitations within the setting of a rapid evaluation study.

Background

This section of the report covers workstream 4 of the overall outpatient study and is a mixed-methods evaluation of Patient-Initiated Follow-Up (PIFU) in the English NHS.

In Background and context we described some of the issues and challenges facing outpatient care within the English NHS. These include the large increase in volumes over recent years, long waiting times exacerbated by the COVID-19 pandemic and the potential overuse of patient and staff time. Part of the problem is that under standard pathways, patients with long-term conditions or following surgery are automatically called back for outpatient appointments at regular intervals (e.g. every 6 months). These timings are not necessarily decided by clinical need or when a patient wants extra support. This means that when follow-ups do occur, they can fail to lead to further investigation or any meaningful change in patient management. 63 Conversely, when a patient’s symptoms or circumstances do change, they may experience a long wait for an appointment as capacity has been devoted to routine follow-up.

NHS leaders have called for an ‘industrial’ drive to cut the number of unnecessary outpatient appointments and better prioritise clinical time where it adds the most value – setting an ambition to reduce outpatient follow-ups by 25% against 2019–20 activity levels by March 2024. 12 To deliver this, NHSE has been encouraging hospitals to move some of their outpatients to PIFU pathways. 64

PIFU aims to give more flexibility and choice to patients over the timing of their care and allow them to book appointments as and when they need them, rather than follow a fixed or standard schedule (e.g. every 6 months). Figure 10 shows how the process might change under a typical PIFU pathway when follow-up appointments are initiated by the patient or carer compared to a traditional fixed or standard schedule pathway. In some specialties, similar services have been operating for some years, referred to by terms, such as ‘open access’. One of the key goals of PIFU is to better match clinical resources with patient need – an ever-growing challenge in the NHS as the number of patients waiting for specialist care continues to grow. NHSE has produced guidance to support trusts to implement PIFU, including advice on which specialties and cohorts PIFU is suitable for. 53 This also outlines three quality standards that must be met (reproduced from public sector information 40 licensed under the Open Government Licence v3.0. www.nationalarchives.gov.uk/doc/open-government-licence/version/3/):

FIGURE 10.

Fixed follow-up outpatient schedule (standard pathway) vs. personalised follow-up schedule (PIFU pathway). A typical outpatient pathway.

1. All patients and/or carers should have PIFU explained to them and the opportunity to ask questions and raise concerns. If they do not understand how or when to trigger an appointment, PIFU should not be used (see information on shared decision-making).

2. A standard operating procedure (SOP) that includes patient safety nets should be in place.

3. All patients moved to a PIFU pathway should be logged and tracked on the organisation’s information technology (IT) system, and the service able to report on key metrics including the number of patients who are on a PIFU pathway.

The requirement to meet these standards aims to distinguish PIFU from other less-formalised approaches to personalised follow-up (such as open access). Additional guidance has been produced to support PIFU implementation in particular specialties, such as adult Trauma and Orthopaedic pathways. 65 These consider specific factors relevant to implementation in these specialties, such as which patients are clinically suitable and the duration of a PIFU pathway for different conditions or injuries.

Although the guidance provides support, it recognises that implementing PIFU is a decision for individual services who should ‘develop their own guidance, criteria and protocols on when to use PIFU and be clear that it should not be used as a substitute for the appropriate timely discharge of patients’. Several factors are stated as influencing how long it will take to implement PIFU including ‘the size of the service, current service demand, maturity of any existing PIFU, staff engagement, IT system set up and the time and capacity the team leading this work has’. Alongside this guidance, NHSE has produced a more detailed PIFU implementation plan and pre-implementation checklist.

However, as the NHS seeks to expand PIFU further, there are several questions about how well it works, and for whom. These include:

• The evidence that PIFU reduces unnecessary appointments and frees up clinical capacity for patients who need it the most.

• How patients experience PIFU, and what risks might it involve.

• How PIFU might affect health inequalities and different patient groups.

• What differences there are in how patients are selected for PIFU.

While there are potential benefits of PIFU to patients, clinicians and the NHS, there are relatively few studies exploring them. 26 Where evidence does exist, results are mixed both in terms of impact on patient experience and satisfaction, as well as the ability to reduce volumes of appointments. 26 Modelling carried out in December 2020 by the Outpatient Recovery and Transformation Policy and Strategy team at NHSE (based on two trusts for a number of specialties) suggested that PIFU could reduce follow-up patient attendances by between 0.8 and 1.8 follow-ups per patient depending on the specialty. 53 Further evidence of the impact of PIFU on, for example, outpatient activity, waiting lists, the potential diversion of activity to other services, patient and staff experience, and variations in access and use, would help in refining its design and implementation.

Aims and objectives

The RQs for this evaluation were:

1. 3.1 How is PIFU being implemented? What components and processes does it involve? What is the involvement of technology in the pathway?

2. 3.2 What are the relevant outcomes for evaluating the impact of PIFU services? To what extent can we measure the different impacts of these services? How are the data being used to monitor the progress against these outcomes and how can the data be used?

3. 3.3 What are the unintended consequences (e.g. on other services) of PIFU and how are potential risks being managed? What strategies have been adopted to address potential inequalities along the PIFU pathway? What data are being collected to understand potential disparities? Is there variation in how different patient populations access and engage with PIFU?

4. 3.4 What are patient experiences of engaging with PIFU services? What is the level of patient engagement?

5. 3.5 What are staff experiences of delivering PIFU? What is the impact on staff satisfaction, workload, and capacity across different roles?

After describing the methods (see Methods for the evaluation of Patient-Initiated Follow-Up), we present findings from a rapid evidence review of evaluations of personalising outpatient follow-up (see Rapid Patient-Initiated Follow-Up scoping review). This is then followed by sections describing the findings relating to implementation (see How is Patient-Initiated Follow-Up being implemented?), impact on outpatient and other hospital activity (see Measuring the impact of Patient-Initiated Follow-Up), health inequalities (see What is the impact of Patient-Initiated Follow-Up on health inequalities and how is this being measured?), experiences and views of staff (see Staff experiences), and of patients (see What is the impact of Patient-Initiated Follow-Up on patient engagement and experience?). In Chapter 4 we present an evaluation guide to assist with ongoing and future evaluation of PIFU services and, finally, key findings and implications are described in the Discussion (see Chapter 5). Recommendations for policy-makers and further research are summarised in Chapter 6.

Design

The evaluation of PIFU was designed as a convergent mixed-methods study in two phases preceded by a rapid evidence review. The two-phase approach allowed the team to understand many of the important issues about data, implementation and experiences that would inform more in-depth investigation at a later stage. It also allowed the team to provide early feedback to NHSE to inform evolving policy.

Phase 1 was a 4-month rapid evaluation using national data sets alongside qualitative interviews with local and national stakeholders. Phase 1 took place between June and September 2022. The qualitative aspects focused primarily on how PIFU was being set up within specialties where it was most widespread or established. This included high-volume specialties like Physiotherapy or Orthopaedic surgery where pathways tend to be short term or time-limited in nature. Quantitatively, we analysed national data to assess the impact of implementing PIFU on a range of outcomes relating to patients and use of hospital services.

Phase 2 took place between January and July 2023 and built on the insights from phase 1. The qualitative part of this phase consisted of in-depth interviews with both patients and staff and a workshop to discuss the challenges and enablers of PIFU uptake. Compared to phase 1, the purpose was to investigate issues in greater depth and across a wider range of specialties and conditions. This included a focus on how PIFU was being implemented within sites and applied to patients with longer-term conditions that tended to involve more complexity and variation in practice. It also included gathering patient experience data and quantitative analysis of both local and national data.

The design was agreed with the advisory group including the PPIE representative.

Evidence review

For the first stage of the evaluation, we conducted a rapid scoping review which investigated the available evidence to understand what was already known about the impact of personalised follow-up initiatives for outpatient care on service use, patient experience and patient outcomes. This also enabled the team to gain a better understanding of key implementation issues (see Rapid Patient-Initiated Follow-Up scoping review).

Analysis of the implementation and impact of Patient-Initiated Follow-Up at national level

Analysis of the implementation and impact of Patient-Initiated Follow-Up at a local level

To understand how PIFU is being implemented at a local level as well as gain views and experiences of staff and patients, we selected five study sites. Three sites were included in phase 1 [one integrated care system and two acute NHS trusts] and two sites in phase 2 (two acute NHS trusts). A COREQ checklist for the staff and patient interviews has been completed and is available as Report Supplementary Material 2.

Workshop on challenges and enablers for Patient-Initiated Follow-Up

In phase 2, in addition to the interviews with case study sites, we held a workshop with NHS staff to explore perspectives and experiences of PIFU. The workshop took place on Tuesday, 6 June 2023 and lasted two-and-a-half hours. Attendees were staff from NHS trusts who were invited to attend the workshop through an open invitation on the future NHS platform. People were asked to express their interest in attending to share experiences of PIFU implementation and put forward two representatives from their trust to cover both operational and clinical roles. In total, 22 external participants attended the workshop representing 13 trusts from both clinical and operational roles. All attendees were provided with a participant information sheet in advance.

The workshop was structured around three separate discussion topics which covered how participants viewed the aims and objectives of PIFU, challenges they were experiencing with implementation and factors which would encourage an enabling environment for PIFU implementations. These discussions were facilitated by members of the research team. All discussions were recorded and content captured through a combination of note-taking and virtual whiteboards before being fed into a RAP sheet (see Report Supplementary Material 5).

Findings from the workshop were summarised in a short document provided to NHSE.

Combining results

We triangulated the qualitative findings by data source and combined the results from the patient and staff interviews, and workshop to identify common themes in relation to the RQs. 68 Quotations are used throughout the report. Staff were identified by whether they had a trust or specialty level role, and whether their role was operational or clinical. Patients were identified according to the specialty.

Evaluation guide

We developed an evaluation guide for adoption by local services to facilitate their own ongoing evaluation of PIFU. The development of the guide was informed by the findings from all phase 1 and phase 2 workstreams. The guide includes both quantitative and qualitative elements. It covers planning and design, data collection and analysis, and reflecting on and reporting of results. Further details on the methods adopted for this guide are described in Chapter 4.

Overview

Introduction

Following the scoping review of outpatient innovations, we carried out a rapid review to examine what we know from the available evidence about the impact of personalising follow-up. This review has been published as a Nuffield Trust report. 26 Below we provide a summary.

Aims

The review aimed to examine the available evidence to understand how personalising follow-up impacts service use (including outpatient activity, wider health service use, and costs), patient experience and outcomes (including satisfaction, quality of life, and clinical outcomes). We also sought to identify gaps in the available evidence. We aimed to examine how PIFU has been implemented across a range of contexts and conditions and to understand key implementation considerations that might impact its effectiveness.

Methods

The rapid PIFU scoping review emerged from the wider scoping review (described in Rapid scoping review of outpatient innovations) conducted to understand the evidence that exists relating to different outpatient service innovations.

We adopted an evidence-mapping approach27,29,69 to provide timely information to inform the later stages of the evaluation and we followed recommendations from similar evidence mapping reviews. Where feasible, we have reported the methods and findings according to PRISMA guidelines. 30 See Appendix 5 for a PRISMA checklist.

The review search was conducted (initially in August 2021 and renewed in July 2022) across four databases: MEDLINE, EMBASE, SSCI and HMIC for studies published between January 2015 and June 2022. The review was limited to published articles (grey literature and commentaries were not included).

To be eligible, studies had to examine the effects of patient-initiated approaches to follow-up on service use (e.g. waiting times, referral rates, attendances, missed appointments, costs) as a primary or secondary outcome. We also report on patient outcomes, such as experience and clinical outcomes (where reported). The broader search strategy of the initial scoping review (see Rapid scoping review of outpatient innovations) was supplemented with PIFU-related terms adapted from Whear et al. 70 The quality of included studies was assessed using a modified version of the Mixed Methods Appraisal Tool (MMAT). 71

Further details on eligibility, search strategies and quality assessment are described in Appendix 5.

Findings

We identified 17 studies that examined the effects of PIFU on service use, 7 also reported on costs, and 14 on patient experience and/or clinical outcomes (see Appendix 5, Figure 26). See Report Supplementary Material 7 for details of all included studies and quality assessment.

The review found promising evidence that PIFU might result in fewer overall outpatient appointments, however results across studies were mixed (Table 6). The impact of PIFU on wider health service use and costs is unclear due to a lack of evidence. PIFU seems to have little or no impact on patient quality of life or clinical outcomes but might have a small beneficial impact on patient satisfaction.

Discussion

This review not only provides an update of the available evidence, but also extends the evidence by including a broader range of study designs and synthesising evidence across a wider range of specialties and conditions. The review is novel in its focus on implementation and understanding the models of components of PIFU.

Despite some promising findings, available evidence is limited and often lacks generalisability. In many cases results across studies were mixed. Only a small number of studies have been conducted across a few specialties, few studies were rated as good quality, there is uncertainty about the clinical significance of statistical findings, and design issues were common. Only a small proportion of studies were conducted within the NHS and many PIFU models involved other innovations and service changes. Most studies involved a short follow-up duration and there was substantial variation in which outcomes were measured.

The review identified several key gaps in the available evidence where further work is needed: in particular, the impact PIFU is likely to have on patients, outcomes across patient groups and potential inequalities in access or engagement, the range of PIFU models that have been delivered and how components might be related to effectiveness, and staff experiences of delivering PIFU.

Overview

Introduction

In this section, we present evidence that addresses RQ 3.1 for the PIFU evaluation:

• How is PIFU being implemented? What components and processes does it involve? What is the involvement of technology in the pathway?

We begin by providing a national overview of how PIFU is being implemented in the NHS before discussing the key components of a PIFU pathway with reference to examples from the case study sites. We then summarise how staff in the interviews perceived the aims and objectives of PIFU and the factors that have been considered to enable and hinder PIFU implementation. Finally, we summarise what staff perceived to be the key opportunities and risks associated with PIFU.

The findings on staff and patient experience are explored in more detail in Staff experiences and What is the impact of Patient-Initiated Follow-Up on patient engagement and experience? respectively.

Methods

Findings in this section draw on:

• semi-structured interviews with clinical and operational staff across five case study sites

• analysis of national and local site documentation

• a workshop with clinical and operational staff representing 13 NHS trusts

• analysis of national data.

Triangulating data from the different sources, it was clear that there was significant overlap between the themes identified in the interviews and workshop discussions, so we present these as one data source. However, where quotes are provided, we specify the source.

Further details on methods and data sources can be found in Methods for the evaluation of Patient-Initiated Follow-Up.

What components and processes are involved in a Patient-Initiated Follow-Up pathway?

Figure 10 (see Introduction) illustrates a typical PIFU pathway. Each stage involves different design and implementation choices and the approach taken will depend on a range of contextual factors, including the condition, patient need and local site processes. 53

Table 7 illustrates some of the key design considerations and choices available at each stage of the PIFU pathway and distinguishes between higher and lower intensity input from services in implementation. It is drawn from the evidence review and phase 1 interviews.

National overview: how is Patient-Initiated Follow-Up being implemented across the National Health Service?

Since reporting of PIFU activity began in September 2021, the recorded numbers of times patients were either moved or discharged to PIFU pathways across the English NHS had approximately tripled by March 2023 (Figure 11) to about 185,000 per month. In the same month, the total number of reported outpatient attendances was about 7.4 million, which means that reported movements to PIFU pathways is about 2.5% of this. Since not all submissions of PIFU activity data to P-EROC are recorded as ‘complete’, this number is likely to be an underestimate.

FIGURE 11.

Total numbers of times patients were reported to be moved or discharged to PIFU between September 2021 and March 2023. Source: P-EROC.

Patient-Initiated Follow-Up is most commonly used in the Trauma and Orthopaedics and Physiotherapy specialties (Figure 12), which together have 35% of all reported patient transfers to PIFU pathways between September 2021 and March 2023. However, these two specialties also treat relatively high volumes of outpatients. Measuring numbers transferred to PIFU as a proportion of all outpatient activity over the most recent 6 months in the data (from October 2022 to March 2023), of these specialties, PIFU is most commonly used in Orthotics with a proportion of approximately 11.5% (see Figure 12). Proportions in Trauma and Orthopaedics and Physiotherapy are still comparatively high at 6–7%, whereas 12 of these specialties rates below 5%.

FIGURE 12.

Total numbers of times patients were reported to be moved or discharged to PIFU between September 2021 and March 2023 (only trusts that record PIFU activity in P-EROC for these specialties are included). ENT, Ear, Nose and Throat.

There is a large amount of variation between trusts within the same specialties. Across the 10 specialties where most patients have been moved or discharged to PIFU in the 6 months between October 2022 and March 2023, proportions of numbers moved or discharged to PIFU per outpatient attendance for each trust are shown in Figure 13. Although the majority of these proportions are below 10%, there is some notable variation, with two proportions over 50%.

FIGURE 13.

Variation in proportions of numbers of patients moved or discharged to PIFU per outpatient attendance for October 2022 to March 2023. The top 10 specialties for volumes moved or discharged to PIFU over the same period. ENT, Ear, Nose and Throat. Note: The mean value is marked by an ‘X’ for each specialty, while the median value is marked by a horizontal line across the box. Quartiles were calculated excluding the median. Outliers were defined using the IQR (inter-quartile range) x 1.5 threshold. Only trusts that record at least 100 outpatient attendances a month and PIFU activity in P-EROC for these specialties are included.

Local perspective: how is Patient-Initiated Follow-Up being implemented within National Health Service trusts?

The following section explores how PIFU was being implemented locally across NHS trusts in England during the study period (June 2022–July 2023). It draws on interviews with staff from five case study sites and the workshop with staff from a wider range of sites. As illustrated by the quantitative analysis, hospital trusts in England are at different stages of PIFU implementation. Also, PIFU pathways vary widely in their design. This was observed both from the workshop and the case study sites.

The differences we observed in how PIFU was being implemented included:

• Whether implementation was staggered specialty by specialty or whether it was done across the trust in one go.

• The extent to which PIFU (or similar approaches) had been in use previously within the site.

• The influence (and level of emphasis placed on) the NHSE ambitions (including original 5% target) on adoption of PIFU.

• Who led the implementation (e.g. whether it was done by a central outpatient team, or whether it was by individual clinicians) and the level of support they provided.

• The extent to which PIFU implementation was given allocated or dedicated staff time (e.g. dedicated PIFU leads or outpatient transformation leads).

• The electronic PAS in use and number of changes required to record PIFU activity.

• The involvement of technology to support PIFU implementation and delivery.

• The development and use of standardised materials such as SOPs.

• The types of short-term and long-term PIFU pathways in use (e.g. PIFU with or without clinical review, discharge to PIFU or PIFU with fixed recall).

• How sites approach PIFU in relation to discharge.

• The priorities of sites (e.g. codifying existing activity or increasing the use of PIFU in long-term follow-up pathways).

We also observed differences in how PIFU is implemented across clinical areas. Table 8 illustrates a sample of four PIFU pathways we heard about from phase 2 of the data collection, where we looked in more detail at two case study sites to understand how they are implementing PIFU across clinical areas. These were selected to illustrate the range of approaches that we heard about in the research. It is important to note that given the wide variation in how pathways are designed, these may not be representative of how PIFU is being used more widely within the NHS.

We now describe the key stages of these pathways in more detail.

What factors influence Patient-Initiated Follow-Up implementation?

Interviewees and workshop participants were asked to describe the enablers and barriers they had experienced with regards to PIFU implementation. In this section we summarise these findings in relation to PIFU implementation using the structure of the NASSS framework. 67 This framework notes that innovation adoption is affected by several factors including those relating to the intervention itself, the condition in which it is being used, those using the innovation (staff, patients and carers), the organisation in which it is being implemented and the influence of the wider system. These categories reflect those present in other analytical frameworks, such as the Consolidated Framework for Implementation Research. 72

Below we consider the findings considering each of these elements.

Value proposition

Within the NASSS framework, ‘value proposition’ refers to the value of the technology as viewed from the perspective of the healthcare system and patients. 67 We have used it to explore the aims and objectives of PIFU from the perspective of staff in relation to the system, service, and patients. How the value or purpose of the innovation is viewed is a factor in how it is perceived and implemented.

Multiple staff, both in clinical and non-clinical roles, spoke of disparities in perception of the aims of PIFU at different levels of the organisation. Table 9 outlines how staff in the interviews perceived the benefits of PIFU for patients, staff and the service.

Nature of the condition

The nature of the particular condition, specialty or pathway appears to be a key factor affecting PIFU implementation. PIFU seems to be common or more straightforward in conditions with lower complexity, easily detectable fluctuations, or changes in symptoms. Where people had multiple complex conditions requiring different levels of input from different services, PIFU was considered less straightforward.

They come in, we sort them out, operate on them and then hopefully we’ve done what we need to do and we can say goodbye to them, whereas I understand within medical specialties you’ve got people [with] long-term health conditions and I completely understand that that is a whole different ball game. But from our perspective it was quite an easy thing to implement.

Operational, specialty level – Site 2.2

In most cases, condition-specific criteria shaped how PIFU was implemented, and each specialty established their own set of standards and principles for choosing patients for PIFU. In some areas, trust-wide protocols have also been developed that consider factors such as a patient’s ability to self-manage, carer support and other risk factors. Although most specialties within the case study sites had set criteria, it did not appear that they were routinely communicated more widely. For example, interviewees in primary care were unclear as to how patients were selected for PIFU and there was a perception that it was based on the judgement of an individual clinician.

Condition-specific guidance, criteria, and examples areas were also thought to be beneficial and where national guidance exists, this was seen to facilitate support for PIFU. This has already been developed in some specialties (such as Gynae-oncology). 73

National guidelines are really important because I think clinicians need that backup of ‘This is the right thing to do. This is the evidence. This is who we think are the right patients from a technical point of view. These are not the right patients.’ So I think national guidelines are really, really key.

Clinical, specialty level – Site 2.2

There are a range of conditions or specialty areas that are perceived to be of higher risk, where PIFU tends to be less well established and where more guidance would be beneficial (e.g. conditions where it may be less easy for patients to identify symptoms for deterioration or deterioration is asymptomatic). Work to identify conditions within specialties where PIFU could be used most effectively was considered valuable.

We’ve done a lot of work both within our trust looking at what you could do within the PIFU envelope and looking at each specialty in isolation ... let’s look at everything and then think about the ones we could.

Clinical, specialty level – Site 2.1

Characteristics of the specialty, condition or clinical pathway may also be relevant. Within Gynae-oncology, interviewees noted that the likelihood of identifying recurrence of some gynaecological cancers from a follow-up routine physical examination alone is low (and that recurrence usually occurs with symptoms), so the value for both staff and patients in attending such appointments is more limited. Another interviewee pointed to the extent to which some conditions are also managed in primary care, which may affect how patients (and staff) feel about PIFU because they are more aware of alternative places to get support.

Specialties like Ophthalmology, which don’t have any primary care, there is that concern that you’re just going to abandon people. And I think specialties with lots of good primary care don’t have to worry about that.

Clinical, specialty level – Site 2.1

PIFU was perceived from those in primary care to work well with conditions with ‘protocol-driven pathways’ with clearly defined stages and treatments (such as certain cancers). However, pathways which involved investigations and treatment for symptoms, where cancer had been ruled out, introduced complexity and uncertainty not only for the patient but also in determining the role of the specialist and GP. This may influence who patients approach (or think they should be approaching) for different aspects of their condition. One GP described this as follows:

The challenge is when you’ve got patients with conditions like inflammatory bowel disease, where most GPs would be able to manage straightforward flare-ups and we know what to do, but there’s a bit of a grey area between the level of expertise that the GP would have, and perhaps the very latest evidence-based specialist intervention for using immunological drugs, etc. So, I think sometimes that grey area creates a bit of confusion and sometimes patients do cycle around the system a bit.

GP

Adopters

Internal factors: the organisation

External factors: the wider system

What do staff consider the opportunities and risks associated with Patient-Initiated Follow-Up?

We asked staff what they considered to be the greatest opportunities and risks associated with PIFU going forward. The various issues raised are laid out in detail in Appendix 6, Table 16. Opportunities included expanding the use of PIFU to other suitable clinical areas and making greater use of technology. They also saw PIFU as creating opportunities for reimagined ways of working within organisations and as well disposed to better collaboration and information sharing between organisations. However, concerns were raised around the implications for patient wellbeing and safety if they did not or could not get in touch with the service when needed. Some staff were also concerned that PIFU could disadvantage patients for whom it did not work as well or were not offered the opportunity of PIFU when it could work for them. In addition, there were worries that PIFU could increase demands on staff capacity, both within outpatient teams and elsewhere in the system.

Summary of findings

In the NHS, PIFU is most commonly being used in short-term pathways (e.g. physiotherapy or following surgery) although there are several examples where it is being used for people with long-term conditions. Several trusts were already using patient-initiated approaches to follow-up before December 2021, although it was unclear how this was being captured in national or local data. Models of PIFU vary widely across trusts and clinical area, with a significant degree of variation in the approach to patient selection, monitoring, and discharge.

Where PIFU had been implemented successfully, enablers included clinical engagement, supporting guidance and champions, dedicated staff capacity and flexible recording systems. Barriers to implementation included patients not being aware they were on PIFU, staff resistance, competing priorities and limited capacity to dedicate to PIFU, a lack of engagement with primary care and challenges amending EPR systems to record PIFU activity. More generally, factors relating to how the benefits of PIFU were perceived by staff and the nature of the condition were also relevant.

Overview

Introduction

In this section we address RQ 3.2 for the PIFU evaluation:

• What are the relevant outcomes for evaluating the impact of PIFU services? To what extent can we measure the different impacts of these services? How are the data being used to monitor the progress against these outcomes and how can the data be used?

Methods

We describe the general approach and the data we have analysed in Methods for the evaluation of Patient-Initiated Follow-Up. In this section we provide more detail of the methods we used.

Results

Table 11 gives a sense of the scale of outpatient activity and PIFU activity for all specialties, and for the 30 we have focused on. PIFU counts in the period represent < 2% of all attendances.

When P-EROC data were first collected in September 2021, half of NHS trusts reported complete data and 85% complete or partial data. These proportions had increased to 58% and 98%, respectively, by March 2023 (Figure 14).

FIGURE 14.

Proportions of trusts reporting complete or partial data within P-EROC, by month.

Multispecialty analyses

Analysis at National Health Service trust and specialty level (Poisson regression)

A total of 116 trusts had complete or partial data reported for each month of the period over which PIFU rate was measured (April–September 2022).

We found no statistically significant relationship between the PIFU rate and each measured outcome (Figure 15). These findings were consistent across a range of different assumptions tested in the sensitivity analysis (see Appendix 7, Tables 18–23).

FIGURE 15.

Relationship between the PIFU rate and outpatient activity across the 30 specialties with the highest volume of numbers moved/discharged to PIFU. Change in outcomes associated with a 5% increase in PIFU rate. Error bars show 95% confidence intervals.

Analysis of time to next event (survival analysis)

We included 56.7 million attendances in the multispecialty models (details of the exclusions we have made are described in Appendix 7, Figure 27). Figure 16 shows adjusted hazard ratios (HRs) for five outcomes (four outpatient-related and one ED-related). The HRs are expressed for each five-percentage point increase in the PIFU rate.

FIGURE 16.

Adjusted HRs for five outcomes. HR expressed for each five-percentage point increase in PIFU rate, with 95% CIs. Note: HR = 1 (bold line) signifies no association between PIFU rate and time to next outcome. HR < 1 signifies that higher PIFU rates are associated with a longer time to next outcome (i.e. less frequent events).

An example interpretation of the HRs follows. We found that for each five-percentage point increase in PIFU rate the adjusted HR for the structured review attendance outcome (‘time to next structured review’) was 0.980 [95% confidence interval (CI) 0.978 to 0.980]. This was statistically significantly below 1, suggesting an association between increasing PIFU rates and decreasing frequency of structured review attendances (or longer time to next structured review). However, the effect size was modest: for each five-percentage point increase in PIFU rate, the probability of a structured review on any given day (for individuals yet to have one) was 2% lower (1 – 0.980 = 0.020 = 2.0%). It is important to note that this effect is not cumulative over time. However, given that this is impact measured over the whole outpatient population, this may correspond to a larger effect on those patients on PIFU pathways.

All four outpatient outcomes showed statistically significant results on the same scale – with HRs between 0.984 (next attendance) to 0.979 (next not attended/DNA).

The adjusted HR for next ED visit was 1.001 (95% CI 1.000 to 1.001), statistically significantly greater than 1, but in practical terms only marginally so.

Single-specialty analyses

Of the top 30 specialties by volume of PIFU activity, one was only reported at a single trust and therefore not included in any of the single-specialty analyses.

Analysis at National Health Service trust and specialty level (Poisson regression)

Fifteen of the 29 specialties had data reported by 30 or more NHS trusts and therefore were included in our single-specialty Poisson regression models carried out at trust and specialty level. (Numbers of trusts that were included for each specialty are shown in Table 12.) As illustrated in Figure 17 (left-hand side), 10 out of 15 specialties reported a percentage reduction in structured reviews per patient for every 5% increase in PIFU rate. Of these, as individual statistical tests, the relationship is significant for Breast Surgery (18.7% lower attendances for every 5% increase in PIFU rate, 95% CI 2.4% to 32.3%; p = 0.026) and Ear, Nose and Throat (ENT) (18.9% lower attendances for every 5% increase in PIFU rate, 95% CI 5.7% to 30.2%; p = 0.006). However, neither of these results become statistically significant after adjusting for multiple testing (using the Bonferroni correction – revised p-value threshold = 0.003).

FIGURE 17.

Structured review attendances – model results for individual specialties. Two model variants: (a) Poisson regression model (showing percentage change in attendances per patient) and (b) survival model (showing adjusted HRs, for subsequent events), both per five-percentage point increase in PIFU rate. Ninety-five per cent CIs included. Specialties are ordered by decreasing volume of PIFU. For survival model * denote statistically significant results, having adjusted for multiple testing.

TABLE 12 - The number of trusts corresponding to each specialty from which data have been analysed in the Poisson regression model where the relationship between structured review attendances and PIFU rate is investigated by specialty

Specialty	Number of trusts
Urology	45
Breast Surgery	38
Trauma and Orthopaedics	69
ENT	58
Ophthalmology	39
Pain Management	36
Gastroenterology	49
Cardiology	44
Dermatology	52
Respiratory Medicine	44
Neurology	40
Rheumatology	55
Paediatric	42
Gynaecology	63
Physiotherapy	46

With regard to other outcomes, Breast Surgery and ENT did not report a significant reduction in total outpatient attendances, follow-ups or unattended appointments. However, for Urology there was a significant reduction in overall attendances per patient (−4.2%, 95% CI −6.8% to −1.5%; p = 0.002), follow-ups per patient (−3.4%, 95% CI −5.3% to −1.6%; p < 0.001) and DNAs (−15.4%, 95% CI −24.2% to −5.6%; p = 0.003). We found no significant relationship in Urology with structured reviews per patient or proportions of attendances that were structured reviews. For Respiratory Medicine there was also a significant reduction in overall attendances per patient (−15.4%, 95% CI −23.8% to −6.1%; p = 0.002). Further details are described in Appendix 7, Section 2.

With the sensitivity analysis, under different assumptions regarding the denominator to use for measuring PIFU rates and the lag between PIFU rates and outcomes, results were similar. Using assumptions that treat missing PIFU data as no PIFU, more specialties became eligible for inclusion within the analysis. Under these assumptions, we found significant reductions in structured reviews associated with increased PIFU use in Paediatric Trauma and Orthopaedic, significant reductions in DNAs within Dermatology and Midwifery, and significant increases in DNAs within Rehabilitation Medicine, Podiatry and Clinical Oncology (see Appendix 7, Tables 18–23).

Analysis of time to next event (survival analysis)

We were able to analyse single-specialty models for 29 specialties using the same data as for the multi-specialty model (56.7 million attendances in total). Figure 17 also shows adjusted HRs for the time to next structured review attendance by specialty (right-hand side).

After adjusting for multiple testing (revised p-value threshold of 0.0017), of the 29 specialties, 15 had statistically significant results with adjusted HR < 1 signifying a longer time to next structured review (i.e. less frequent attendances).

However, the effect sizes were generally modest. For the two largest PIFU specialties Trauma and Orthopaedic and Physiotherapy, the adjusted HRs were 0.992 (95% CI 0.989 to 0.995) and 0.977 (95% CI 0.975 to 0.979) respectively. In the former, each five-percentage point increase in the PIFU rate was associated with a reduction in the probability of a structured review on any given day of 0.8%.

The effect size was largest for Midwifery attendances, where the adjusted HR was 0.854 (95% CI 0.851 to 0.857; each five-percentage point increase in PIFU was associated with reduction in the probability of a structured review attendance by nearly 15%). Gynaecology, Urology and Pain Management were specialties with adjusted HRs of about 0.95 (a 5% reduction in the probability of a structured review for every five-percentage point increase in PIFU rate).

Some specialties (7 out of 29, having adjusted for multiple testing) appeared to show more frequent structured review attendances associated with increasing PIFU rates. This was the case in, for example, Ophthalmology (adjusted HR 1.039, 95% CIs 1.020 to 1.059) and Cardiology (adjusted HR 1.024, 95% CIs 1.011 to 1.038).

Regarding other outcomes, findings for time to next attendance and time to next follow-up were similar to those described for structured reviews, although with slightly fewer specialty results reaching levels of statistical significance. Time to next DNA results were less likely to be statistically significant. Only 4 specialties of 29 (Trauma and Orthopaedics, Physiotherapy, Midwifery and Pain Management) had adjusted HRs < 1 (with p-value < 0.0017), and a single specialty – Podiatry – had HR > 1 (so more frequent DNAs associated with increasing PIFU rates). Time to next ED visits results were also significant for four specialties (Trauma and Orthopaedics, Gynaecology, Dietetics and Clinical Oncology) with increasing PIFU rates associated with less frequent ED visits. Further details are shown in Appendix 7, Figures 28 and 29.

Discussion

Overview

Introduction

In this section we present findings in addressing RQ 3.3:

• What strategies have been adopted to address potential inequalities along the PIFU pathway? What data are being collected to understand potential disparities? Is there variation in how different patient populations access and engage with PIFU?

Methods

Data are drawn from interviews with clinical and operational staff, four interviews with patients and a workshop with staff. Further details on the methods can be found in Methods for the evaluation of Patient-Initiated Follow-Up.

From one acute hospital trust, different to the ones where we conducted interviews, we acquired aggregated data on numbers of patients put on to PIFU pathways and, of those, the number who made a return outpatient visit within 1 year of being put on to the PIFU pathway. These were broken down separately by age band and ethnicity. We then analysed these data to observe if there were any differences in return rates by age and ethnicity.

How are services measuring the impact of Patient-Initiated Follow-Up on different groups of patients?

Most staff interviewees in the study reported limited understanding on the impact of PIFU on different groups of patients and limited insight into how (if at all) this was being measured within their service. However, they did consider that there was a risk that PIFU would have an unequal impact on different groups (see How is Patient-Initiated Follow-Up being implemented? on implementation for further discussion of the risks of PIFU). One ICB lead noted that PIFU had been considered within broad equality impact assessments on outpatient care while others noted that it had not come up at all in these contexts. Although the ICB had produced guidance for trusts on this, there was concern about how consistently this was being applied in practice. There was significant concern that the national attention on targets (in particular, reducing follow-up appointments by 25%) was overshadowing attempts to address the risks around health inequalities.

There’s often a disparity between what is pushed in order to recover from national levels towards what’s right and best for the patient. And they sometimes clash, and sometimes this push is causing harm to these people.

GP/Operational – ICB

At a trust level, although most interviewees recognised that addressing health inequalities as part of the recovery from COVID-19 more generally is an NHSE priority,79 they also recognised the need for a greater understanding of the particular impact of PIFU for people from different backgrounds. The current lack of alignment of demographic data with PIFU data seems to be a barrier to some trusts doing this effectively (see Measuring the impact of Patient-Initiated Follow-Up). Some service leads – especially those working in more rural trusts – felt that their local area was not particularly diverse so their understanding of how PIFU might impact people with different protected characteristics was even more limited.

I don’t think we’ve identified anything related to PIFU, but we certainly have identified issues related to health inequity within outpatient settings. But it’s baby steps at the moment. We need to do so much more.

Operational, trust-wide – Site 2.2

We heard limited examples of specific actions taken to address or mitigate against potential inequalities in the sites we looked at, although, as noted, understanding of this issue is limited. Some felt that a person who was at risk of not engaging with PIFU would not be put on to the pathway in the first place and therefore the issue was less likely to arise. In terms of supporting people who may otherwise struggle to engage with PIFU (such as older people or people with a learning disability), staff pointed to the valuable role of formal and informal carers.

If you’ve got a patient that maybe lives in a care home then a carer would come with them and, [ ... ] is it right to be dragging somebody who’s 88 back to the hospital? No, it’s not appropriate [...] ... Very often, some of these people don’t have a next of kin, so as long as we make it very clear to the care home that [...] if a woman started to bleed again, [...] this is what you need to do [...]. So yeah, we are very careful in the instructions that we give to carers or nursing staff that are looking after patients and care homes.

Administrative, specialty level – Site 2.2

What is the perceived impact of Patient-Initiated Follow-Up on health inequalities?

Although there is limited information on the impact of PIFU on different patient groups, staff described how they thought health inequalities could arise. Although staff interviewed considered that it is important to not make assumptions about someone’s engagement just because of certain characteristics, there was also concern that staff may not be putting people on to PIFU who may benefit. The opposite was also raised, mostly in the context of the concern around using PIFU as part of a waiting list validation exercise, where decisions may be made without proper interaction with patients. This was identified as being particularly likely in areas where the overt focus of PIFU was reducing activity, rather than on supporting patients.

Clinicians shouldn’t make assumptions that because [they] have an older patient they won’t use PIFU or that a younger patient necessarily will use PIFU.

Operational, trust-wide – Site 1.1

When asked about health inequalities, staff delivering services often referred to people of different ages or people who may be digitally excluded. However, they also reflected that because there were alternative options available such as contacting the service by phone (and staff suggested that were patients really at risk they would not put on to PIFU anyway) they felt that this was less of an issue. Regarding digital exclusion, the extent to which this was currently a problem was considered limited given that there are usually non-digital ways to contact the services. As described previously (see How is Patient-Initiated Follow-Up being implemented?), the opportunity to use technology to support PIFU implementation (such as patient portals or apps) was noted by several staff who did acknowledge that this could have the potential to exclude people who may not feel confident using these tools.

In terms of health inequalities, [the] elderly population isn’t comfortable using electronic systems. But since telephone is one of the ways they can get in contact, they are more comfortable with it.

Clinical, specialty level – Site 2.1

Rather than focusing on demographic characteristics, interviewees referred to an individual’s motivation and ability or willingness to advocate for themselves (this was based on anecdotal experience rather than evidence and also is not necessarily specific to PIFU but more generally around engagement with services). Staff felt this was in part linked to socioeconomic status but also the local area and how accepting people are around how outpatient appointments have been previously managed. There was also recognition that people from some backgrounds (e.g. those from lower socioeconomic backgrounds) do not routinely engage with services as much, and this may be a particular concern for PIFU. This concern was raised particularly in the interviews with primary care staff.

If I’m really honest, does it close the gap [on health inequalities], probably not – it probably increases the gap actually because it’s the activated, it’s the motivated, it’s the well-educated who are more likely to navigate the system quickly and effectively.

GP

Interviewees from secondary care suggested that PIFU could be better for people who may, for whatever reason, be unable to engage with a more fixed appointment schedule. For example, people with ‘chaotic lifestyles’, which may affect their ability to engage with healthcare services more generally. PIFU therefore was not only seen as an opportunity for supporting their engagement in their own health care, but also a way to reduce missed appointments. However, there was concern that such people may be least likely to be put on to PIFU, or that they would be put on PIFU without proper discussion or engagement.

I have a gentleman who’s homeless. We put him on PIFU because every appointment we gave to him, he DNA’d. He’s not very good at sticking to set times for appointments, so by giving him the ability to phone when he needs and setting an appointment that will work for him, actually he’s attended.

Clinical, specialty level – Site 2.1

Impact on Patient-Initiated Follow-Up engagement by patient characteristics

For one study site, the proportions of patients on PIFU pathways across all specialties who make a return appointment within a year by ethnic group are shown in Figure 18 and by age group in Figure 19. On average, across all ethnicities, 12.9% of those placed on a PIFU pathway made an appointment within the following year. Variation among reported ethnicities ranged between 10.4% for the Bangladeshi population and 15.4% for people reported as mixed ethnicity. It is not clear from these data whether these results reflect underlying ethnic barriers to engaging with the service after being put on to a PIFU pathway. Differences could result due to different ages and case mixes among different ethnic groups.

FIGURE 18.

The proportion of patients on a PIFU pathway who make a return appointment within a year, by ethnicity. Data provided by one study site.

FIGURE 19.

The proportion of patients on a PIFU pathway who make a return appointment within a year, by age group. Data provided by one study site.

For children and adolescents between the ages of 0 and 17, 16.7% of those placed on a PIFU pathway scheduled a return appointment within a year. This compares with an average of 11% for those aged 18 and above. The distribution of return rates among adults lies within a 2% margin for all age groups. The lack of granular data for individuals aged 60 years and older makes it hard to comment on PIFU engagement for the oldest patients.

Summary of findings

There is limited understanding of the impact of PIFU on different patient groups although staff who participated in the research recognised this as a concern. Digital exclusion, demographic characteristics, socioeconomic status, and patient characteristics were all thought to be relevant to how patients engage with or are impacted by PIFU.

Data from one site indicate some differences in reattendance of patients on PIFU pathways within a year by ethnic group and across adult age groups. It is unclear whether observed variation by ethnic group reflects underlying ethnic barriers or other confounding factors. There are notably higher return rates for children than for adults.

However, these data come from only one trust, and we were not able to analyse multiple characteristics so caution should be made when assessing the generalisability of these findings.

Overview

Introduction

In this section, we present evidence that addresses RQ 3.5:

• What are staff experiences of delivering PIFU? What is the impact on staff satisfaction, workload, and capacity across different roles?

We describe:

• Any changes to staff roles, such as the assumption of new responsibilities and moving away from historical responsibilities.

• The variable ways in which different staff groups we spoke to perceived PIFU to have impacted their workloads.

• Broader satisfaction with PIFU as applied to the areas participants worked in and compared to other models of outpatient care.

Staffing factors which were considered to affect PIFU implementation specifically are discussed in How is Patient-Initiated Follow-Up being implemented?.

Methods

Findings in this section draw primarily on semistructured interviews with clinical and operational staff across five case study sites. They are also informed by analysis of national and local site documentation and discussions at the workshop with NHS staff. Further details on methods can be found in Methods for the evaluation of Patient-Initiated Follow-Up.

When asked about ways in which PIFU had impacted their experience and roles, as well as that of their colleagues, participants provided a diverse set of reflections. Key influencing factors include the extent to which PIFU was implemented within their service, how far it was perceived to be a departure from existing practice, and the volume of patients on PIFU.

It’s not had much of an impact really – it’s what we do for all our patients all the time. Patients get in touch with concerns related to their gynae cancers.

Clinical, specialty level – Site 2.2

Impact of Patient-Initiated Follow-Up on roles

For some staff participants, implementing PIFU entailed distinct new approaches to their clinical practice. Aspects of PIFU, which are discussed in more detail in How is Patient-Initiated Follow-Up being implemented?, place demands on staff which may have not previously existed. One example is patient selection processes, which rely on the judgement of individual clinicians. The need to perform end-of-pathway clinical reviews could also substantially affect day-to-day roles, as could the necessity for more proactive forms of patient monitoring or safety-netting relative to previous practice. There was some concern that these additional activities had not been adequately considered within individual work plans. Administrative demands of PIFU could also impact the roles of clinical staff in terms of their interactions with administrators and EPR.

[Patient selection] might be outlined in the SOP but it would be done very subjectively [on a patient-by-patient basis].

Clinical, specialty level – Site 1.3

Those who are placed on a PIFU list, they might be monitored and certainly when it gets to the end of that PIFU period there should be a form of clinical review.

Administrative, specialty level – Site 1.2

Where PIFU-specific roles, such as nurse specialists (as discussed in How is Patient-Initiated Follow-Up being implemented?), had been instituted, it helped simplify the workload of the individual with the PIFU-specific role, while taking some pressure off colleagues. One nurse specialist interviewed had roles that straddled PIFU and other responsibilities, but the low volume of PIFU patients and the minimal changes in practice brought on by PIFU meant that they had not noticed significant changes in their day-to-day role.

Impact of Patient-Initiated Follow-Up on workload

Changes to workloads were more noticeable for staff in areas which had previously relied on fixed follow-up schedules for outpatients, or in areas where implementing PIFU increased administrative demands compared to previous, more rudimentary, ‘open appointment’ models. Another key variable was the volume of PIFU patients in a specialty or sub-specialty. Where this was very small, it was often difficult for staff working in those specialties to register any substantial impact upon their day-to-day work. Implementing PIFU had varying implications for different roles, which is why impact on each broad category of role is discussed separately.

As discussed in How is Patient-Initiated Follow-Up being implemented?, in many specialties, administrative staff were responsible for acting as intermediaries between PIFU patients and clinicians. This included receiving contacts and booking slots, sometimes with a degree of triage, as well as recording PIFU on EPRs. This exposed them to significant changes in workload. Often administrative teams had to absorb PIFU alongside their existing set of responsibilities, and tasks were especially onerous where trust EPRs were not well adapted to recording PIFU. Additionally, where communication from clinicians regarding placing patients on PIFU was unclear and incomplete, PIFU could be very demanding on administrative staff time.

. . . for non-clinical staff, I think it’s been a nightmare. Because they have to deal with all the outgoing patients, and which pathway and partial booking list [to place patients on]. And also, clinicians are not writing particularly clear letters.

Clinical/Operational, specialty level/region – Site 2.2

Hospital staff also mentioned a variety of experiences of PIFU. Those who managed very few PIFU patients saw little to no change in their workload as a result. We heard of a limited number of examples where PIFU eased some pressure on clinics because they could reduce the amount of time spent on follow-up appointments and dedicated more time to new patients. The impact of PIFU here varied according to the characteristics of services, as discussed in How is Patient-Initiated Follow-Up being implemented?. For example, those in services requiring patients to be seen within a short time frame, such as Oncology, were more likely to see PIFU free up staff time that would not be immediately reassigned to other patients. This is because these services were considered to have greater flexibility to adapt their clinics. However, we heard little from most staff to suggest that their overall workload had been reduced by PIFU, as any capacity which may have been freed up by reducing the number of follow-up appointments would immediately be filled by new patients or those on waiting lists.

Several participants raised the concern that PIFU would mean that more appointments were spent with patients with more complex needs. More high-intensity appointments could in turn increase the overall intensity of a clinician’s workday, and the participants who raised the issue were concerned about the implications this may have for staff stress and burnout. This was because PIFU would reduce the number of routine ‘check-ups’. Additionally, in some cases, PIFU was considered to increase workload because of the amount of time required to do clinical reviews and the capacity required in PIFU clinics to manage requests.

Obviously, seeing a new patient is a lot more time consuming and difficult than seeing a follow-up patient. And so you used to have clinics where every third or fourth patient was oh, hello, nice to see you. How are you? I’m really good. I feel much better. Great. Thank you. Nice to see you. Bye. That doesn’t exist anymore in my clinic.

Clinical, specialty level – Site 2.2

The more patients are placed on to PIFU, the higher proportion of patients will have greater acuity (because those being seen have a particular reason for initiating that appointment, vs. a fixed follow-up), meaning appointments will need to be longer on average, even if there are fewer appointments overall – NHSE/commissioners need to understand this. The patients on PIFU are not going to be the ‘well’ – we need more time. I know from colleagues PIFU patients may take 20 minutes rather than the standard 15, they’re just a higher demand and you know they’re flaring.

Clinical, specialty level – Site 2.1

Although a small sample, none of the GPs we spoke to noticed any reduction in their workload due to PIFU and raised concerns that PIFU may result in a transfer of workload to general practice if patients were struggling to contact outpatient departments. One reported first-hand experience of where patients who had failed to get through to a PIFU service resorted to visiting their practice, which raised concerns about the possible impact (or lack of positive impact) of PIFU on GP workload, particularly if a PIFU service is not accessible or responsive.

Some hospital staff reported that their workload had increased because of PIFU, particularly as it had become more embedded within the service. There were instances raised where high volumes of PIFU could lead to increased demand, thus potentially increasing rather than decreasing follow-up activity. These were more prevalent in long-term pathways, where patients may have a diverse range of reasons to contact the service over a long period of time and require clinical review at the end of their pathway. This contrasts with short-term pathways, such as post-operative, ‘just in case’ pathways, where most patients were acknowledged to recover without issue. The impression that PIFU might increase workload was cited as a key driver of resistance from staff.

PIFU very much built into the role now – [we’ve] increased [the] team further to manage that with addition to two additional consultants doing more varied surgeries which need more space in the PIFU clinic.

Clinical, specialty level – Site 2.1

[Clinical review] almost defeats the object as it puts an additional appointment in for every individual who is put on PIFU. The criticism from some people is well, you’re still forcing me to do further clinical admin. Clinical, specialty level – Site 1.2

This raised a concern that this could intensify if more patients are placed on PIFU in the future. While low volumes of PIFU patients could be managed without major restructuring of how care is delivered in their specialty, high volumes of PIFU would necessitate a more structured approach, incorporating elements, such as safety-netting and revised methods of delivering appointments.

If we start moving thousands of patients on to PIFU that might create a bigger issue. But in terms of admin workload, especially if you’re having to generate questionnaires and monitor patients closely – it could have an impact and we don’t actually know what that looks like yet. But at the moment, there haven’t been any significant changes.

Operational, trust-wide – Site 2.1

Impact of Patient-Initiated Follow-Up on staff satisfaction

We asked staff to provide their reflections on PIFU as an intervention (staff perceptions of the aims and objectives of PIFU are described in detail in How is Patient-Initiated Follow-Up being implemented?).

Several participants in the research were open advocates of PIFU, believing it to be an overwhelmingly beneficial intervention. One key reason for this was that PIFU gave them more confidence to address patient needs as well as possible, by giving those who needed to be seen most a simple route back into the service, while causing less stress upon capacity than the use of fixed follow-up pathways. This was also mentioned as a source of reassurance because it was seen to increase the overall visibility of patients with the most acute need.

Now we have the option of going, ‘if there’s any problems then you can contact us and we can get you an appointment however it’s needed be it face-to-face or telephone.’ So I suppose that the biggest impact for us is knowing that there’s a different way people can contact us if needed.

Clinical, specialty level – Site 2.1

By contrast, staff who did not have confidence in PIFU functioning as intended expressed anxieties that it could be making some patients less able or likely to be seen when needed, to the detriment of their safety. Levels of confidence varied by team, although numerous interviewees reported a range of confidence levels towards PIFU within their own teams.

Additionally, some staff, with various roles, viewed PIFU positively as a welcome step forward, away from antiquated and paternalistic models of care. Many patients on short-term pathways, such as those who had undergone common procedures, did not need any follow-up appointments (or only a small number of people did). This suggested to some that fixed follow-up pathways were less efficient in terms of the time and effort of both staff and patients. PIFU, meanwhile, could account for the needs of those who did experience issues without expending resources on those who did not. Previously mentioned themes, such as patient empowerment, were also raised as contributors to positive attitudes towards PIFU among staff, with the innovation appearing to align with contemporary shifts towards more inclusive models of care. Conversely, attachment to habitual practices was often cited by staff reflecting on colleagues who were more resistant to PIFU.

I’ve been in the NHS a long time and you would find that patients would be brought back every six months, every 12 months, you know it’s just ridiculous amounts of follow-ups that were happening. Because it seemed to be historical, and whoever saw the patient last then did exactly the same thing as what the previous clinician did, and also patients very often just liked the security of having that follow-up appointment in hospital. Well, now we all know that that’s not necessary.

Administrative, specialty level – Site 2.2

Staff satisfaction with the concept of PIFU often varied depending upon what they believed were the primary aims of adopting the intervention in their specialty. As mentioned previously in How is Patient-Initiated Follow-Up being implemented?, staff perceived distinct purposes at a service level and at a staff or patient level, with a framing of PIFU centred upon improving patient outcomes far better received than one based around targets and service recovery. Some participants reported target fatigue and cynicism towards initiatives passed down from the ‘centre’, with concern expressed that PIFU rollout aimed at maximising volume of uptake would result in placing patients on PIFU for whom it would not lead to an improvement in outcomes.

Currently I think they feel it’s something that they [staff] are kind of being forced to do, especially with the ICB asking how many patients are on PIFU pathways all the time. So I guess it’s really about, actually, can we turn the conversation around and not say how many people, but who would this benefit, who would be better off?

Clinical/Operational, specialty-wide/trust level – Site 2.1

Summary of findings

Staff experience of PIFU is not uniform, being heavily influenced both by the specific characteristics of the model of PIFU being employed in their service, as well as by what practice looked like before PIFU. The volume of PIFU patients in a service affects the extent to which any differences from previous practice are felt. Where it does entail significant changes in practice, PIFU can have a notable impact on the roles and responsibilities of staff and can sometimes also lead to the creation of entirely new roles to support implementation. Different staff roles, even within the same specialty, can experience different impacts on their workload, such as high administrative burdens falling on certain staff. There were some concerns about PIFU increasing the average intensity of appointments or even the overall volume of follow-ups. Staff attitudes towards PIFU beyond its impact on their roles specifically were affected by several factors. These included their level of confidence that their service could implement it as intended, their views on existing (pre-PIFU) practices, and their perceptions of the motivations behind its rollout.

Overview

Introduction

In this section we present findings in relating to RQ 3.4:

• What are patient experiences of engaging with PIFU services? What is the level of patient engagement?

We first present findings from the interviews with patients. We then explore how staff in the case study sites are measuring and capturing patient feedback. We then discuss staff perceptions of patient experience and engagement with PIFU. We also explore the enablers and barriers to patient engagement.

Methods

Data are drawn from interviews with clinical and operational staff, patients, and a workshop with staff. Given the small number of patient interviews, we also drew insight on the perception and experience of patient experience from the staff interviews. Further details on the methods can be found in Methods for the evaluation of Patient-Initiated Follow-Up.

How do patients perceive and experience Patient-Initiated Follow-Up?

We conducted four interviews with patients on PIFU pathways. Three patients we spoke to were on PIFU pathways for Breast Care and one for Gynaecology. All patients had completed their treatment and were on fixed-term pathways of between 6 months and 5 years in length. Although this is a very limited sample, the interviews provided useful insight into how patients experience and engage with PIFU and the factors that affect this.

How do services measure patient experience and engagement?

Most staff we interviewed had not conducted any formal research to capture patient feedback and were not aware of any work that had been done across their trust. Some had done short surveys within specialties or engaged with broader patient groups about PIFU (and more generally around alternative approaches to outpatient care).

We did a patient forum at the start. Patients liked [the] concept of PIFU – coming in when they needed the service – but didn’t like the name. Some patients struggle to understand what PIFU is.

Operational, trust-wide – Site 2.1

Several staff interviewees mentioned that capturing more formal patient feedback was something they were planning or wanted to do. Existing approaches (such as the Friends and Family Test) were considered too generic. As such, most staff reported on anecdotal experience based on whether patients had agreed to being on a PIFU pathway and how they had subsequently engaged with the service.

Perceptions of patient experience were sometimes based on a lack of contact or complaints rather than positive comments (adopting a ‘no news is good news’ approach). One person in a trust-wide role reported that problems related to PIFU had not been raised as a concern in ICB-wide meetings and this was interpreted as there not being any significant issues. This was also reflected in interviews with individual service leads.

I can’t remember a complaint that has ever come in that said you’ve discharged me and I didn’t want to be. So I think they’re very well informed when they have that last consultation. So as far as I’m aware, patients are happy with it, plus they don’t have to keep coming back to hospital. And I think some of them quite like that.

Operational, trust-wide – Site 2.2

Where patient feedback had been collected, a common theme was that patients were not aware they were on PIFU (discussed previously in How is Patient-Initiated Follow-Up being implemented? and see Appendix 6, Table 16). This suggested that patients were not being properly informed about PIFU when it was introduced to them. One reason given for this was that the time pressure clinicians are under means they may not engage in the conversations in the way that they should. For central outpatients teams, this pointed to the importance of communicating how PIFU should work consistently across the trust and in particular distinguishing it from previous approaches.

We’ve done surveys, but [the] feedback was all about trust communications – so [I’m not sure] whether clinicians are having shared decision-making conversations with patients ...

Operational, trust-wide – Site 2.2

It’s about making sure that the same conversations and equal conversations are happening across the trust and with every member of your team, particularly when you have a transient workforce coming through like registrars and locums.

Operational, trust-wide – Site 2.1

So, if PIFU takes a long time to explain if you’re in a hurry because you’ve had a couple of urgent patients and they’ve got something really complex to do, are you going to spend all the time you need to do to explain?

Clinical, specialty level – Site 2.1

How do staff perceive patient understanding and engagement of Patient-Initiated Follow-Up?

We asked staff in the interviews how they thought patients perceived and engaged with PIFU and what the impact on patient experience was. Perceptions of patient experience often related to how staff perceived the aims and objectives of PIFU for patients, such as increasing convenience and access to health care when people need it (see How is Patient-Initiated Follow-Up being implemented?). Several interviewees mentioned that PIFU was more convenient for patients and meant they did not need to travel a long way or struggle to find parking. This was particularly the case in trusts where patients were spread across a wide geographical area and may need to travel to sites where there is limited public transport. Then, in some cases, travelling a long way for a short appointment only to find that everything is fine. In this way, PIFU was considered a more desirable option than fixed follow-up.

Summary of findings

We conducted a small number of interviews with patients on PIFU pathways at the case study sites. Patients liked having the option to contact a specialist when they needed to, rather than needing to be re-referred via their GP. All either planned to or had already contacted the service and were positive about their experience.

Across the case study sites, limited activity had taken place to capture formal feedback from patients, although staff reported that anecdotally patients were positive about PIFU as an approach and the support they had received. Reasons for patients declining PIFU included a preference for regular interaction, a desire to stick to their routine and a concern that they would be unable to get an appointment if they needed one.

There appears to be a gap in understanding about whether patients are contacting the services when they need to. However, when they do, this can be for several reasons, not all of which require a face-to-face appointment with a consultant. Enablers to patient engagement include clear avenues of access, support and communication and ensuring patients do not feel abandoned by the service.

What was already known

• A rapid evaluation of PIFU with a handful of study sites would not be able to study the impact of the variety of modes of implementation across all different clinical specialties – particularly as PIFU evolves over the long term. Therefore, there is a need for local evaluations to improve the depth of evidence available at a specialty level, which in turn can inform improvement in PIFU services.

• To support local evaluators, there is a need for guidance on how to analyse and learn from PIFU services.

What this section adds

• We developed a guide to support teams to evaluate the outcomes and impacts of their PIFU services at a specialty level, thus helping address research gaps and encouraging the sharing of learning.

• A version of the full evaluation guide is available in Report Supplementary Material 8.

Introduction

New policies, such as a nationwide push towards introducing PIFU pathways in outpatient services and decreasing unnecessary outpatient activity, typically move through a cyclical policy process. First, the policy is formulated by central policy-makers (despite in this case PIFU existing in various forms at local levels before becoming a national policy), then implemented by health and care services and the final step for most policies is an evaluation. Policy evaluation involves the results of policies being monitored, which often leads to the rethinking of policy problems and solutions and the start of a new iteration of the cycle. 80

To evaluate PIFU as a policy, RSET undertook the national level evaluation under workstream 4. But as we describe in the Results sections of How is Patient-Initiated Follow-Up being implemented? and Measuring the impact of Patient-Initiated Follow-Up, undertaking this evaluation has been challenging because of the variability of PIFU services and the data that are available at a national level. For example, national data are not able to describe who is on PIFU pathways. Furthermore, during the evaluation, it was difficult to obtain access to relevant local level data because local IT systems, at the time of writing, are not yet set up to enable evaluations. But that is expected to change with planned improvements to data recording processes enabled by actions taken by NHSE and IT suppliers. An important next step in understanding how and why PIFU works requires providing local evaluators with appropriate, easy-to-use tools and encouragement to undertake and share findings of local evaluations that match their available capacity.

While policy-makers are keen to understand the impacts of PIFU-related policies, there is also a need for local services to understand what is (and is not) going well in their service. They should also know how to improve PIFU delivery, demonstrate the value of PIFU pathways and their trade-offs and, if appropriate, how to expand PIFU pathways to other outpatient services. Furthermore, they need to engage with patients to ensure PIFU’s aims of improving personalisation of services are being realised from patients’ perspectives.

Evaluation of PIFU services across a range of specialties is particularly important considering the variability in how PIFU is being implemented and the mixed evidence of PIFU’s effect on health outcomes and costs. 26

Step 1: build an evaluation working group

The initial step focuses on the composition of the evaluation working group. Considering that evaluations demand a significant amount of time, data access and potential regulatory approvals, it is advisable to involve 6–10 senior leaders, managers and patients. The guide provides a list of 12 questions that the evaluation working group should discuss and agree on, some of which are practical, such as ‘Does everyone in the working group have a role and know what they’re doing? Is there anyone missing?’ But also, theoretical questions such as, ‘Are there any unintended consequences (positive or negative) that would be worth measuring in an evaluation?’. By working through the 12 questions with the right people in the (potentially virtual) room and capturing what the group agrees in writing, the guide becomes a resource that can be shared among the group and with wider stakeholders to communicate the scope of the evaluation.

Step 2: decide the aims of Patient-Initiated Follow-Up in your specialty

The second step entails determining which national level objectives for PIFU policy most align with the local values, aims and objectives (Figure 20), such as ‘empowering patients to take ownership over the timing of their care’ or ‘reducing unnecessary appointments’. The emphasis in this section lies in the working group identifying and prioritising those objectives that most closely align with local initiatives and, where possible, working with other stakeholders such as hospital leaders and patients to seek out their opinions.

FIGURE 20.

National programme aim and objectives.

Step 3: describe how and why your Patient-Initiated Follow-Up service works

The third step is designed to assist local evaluators in articulating the functionality and rationale behind their service by employing a logic model template (see Report Supplementary Material 8, p. 13). While the prospect of creating a logic model (i.e. a visual description of the components of the planned change which describes how and why PIFU works) may be daunting for some, we provided them with a pre-existing model describing NHSE policy team’s national PIFU logic model, which they can edit to ensure alignment with local plans.

Step 4: agree on evaluation questions

The fourth step urges evaluators and their evaluation working group to contemplate which evaluation questions hold the greatest significance and importance to their aims and objectives. The guide groups questions by outcome area, such as patient experience or effectiveness. In most outcome areas, the guide provides multiple possible evaluation questions to show local evaluators the different ways in which they can approach their evaluation. While the guide does not provide advice on how many questions a local evaluation should include, it suggests that each new question adds time and resource implications for evaluators.

Step 5: select indicators and outcomes

On having developed a local-level logic model and identified evaluation questions, the fifth step involves reviewing the diverse array of PIFU-related indicators possible and suggests collaborating with business analysts and information teams to assess whether any existing data could inform these indicators or whether new data are needed. While there is currently a lack of much existing administrative data, this is set to change, and the guide makes clear that using existing data can be easier than collecting new data.

Step 6: collect and analyse data

In the sixth step, the guide offers valuable advice regarding the various types of qualitative and quantitative data that evaluators can collect to supplement any pre-existing information. These data collections will serve to enrich evaluators’ understanding and address the evaluation questions effectively.

Step 7: reflect and report on your findings

The final step of the guide provides practical advice and illustrative examples on how to disseminate evaluation findings to different audiences, ranging from patients to trust boards. This comprehensive approach aims to ensure that evaluation outcomes are effectively communicated and contribute to informed decision-making.

Discussion

We developed this guide to support anyone interested in carrying out an outcomes evaluation of a PIFU pathway at a service level. Its development supports the ‘democratisation of evaluation’ movement described in the evaluation literature by encouraging and guiding local researchers to address gaps in knowledge about PIFU services.

The guide is also a direct response to Lamont et al. ’s81 call to action for researchers to support evaluation democratisation by ‘working with service leaders to articulate the goals and describe the components of planned change; to synthesise helpful evidence on related interventions; to identify key stakeholders, appropriate methods, and outcome measures; to test early findings with target audiences; and to consider the best ways to share results’. The guide does this by encouraging local evaluators to work with all relevant stakeholders, adapt national goals and plan change to best suit their local plans, leverage existing data, and offer findings to stakeholders in varied ways.

Overall, the guide encourages local evaluators to plan their evaluation carefully and thoughtfully at the start of a project and engage the right people throughout, which should pay dividends regardless of whether a simple patient satisfaction survey or complex mixed-method evaluation.

Conclusions

We have developed a detailed but accessible evaluation guide for managers of PIFU services that guides them through a specialty-level outcome evaluation in seven steps. The guide builds on existing guidance for evaluators by following their key steps and combines it with policy guidance on PIFU, resulting in a guide that enables novice evaluators to adapt national objectives and data collection tools for their local context. The guide will support the evaluation of PIFU as a policy and provide valuable evidence to inform current research gaps identified in the scoping review and the national evaluation, as well as hopefully help local services improve their PIFU offer based on the local challenges and successes they identify in their evaluations. However, the use of the guide depends on how much time and resource is made available to evaluators and their teams.

Patient-Initiated Follow-Up implementation

• The recorded numbers of times patients were either moved or discharged to PIFU pathways across the English NHS approximately tripled between September 2021 and March 2023, to about 185,000 per month.

• PIFU is most commonly being used in short-term pathways (e.g. physiotherapy or following surgery) although there are several examples where it is being used for people with long-term conditions.

• There is a large amount of variation in how PIFU is implemented between trusts within the same specialties. This includes the extent to which adopting PIFU is a departure from existing practice, given that some specialties were already using a system like PIFU (such as open appointments).

• Models of PIFU vary widely between trusts and clinical area, with a significant degree of variation in the approach to patient selection, safety-netting, and discharge. Also, the nature of the condition was a key factor in how PIFU was implemented.

• Where PIFU had been implemented successfully, enablers included factors related to the nature of the condition, specialty or pathway, organisational and service context, staff and patient factors and the wider system. This included clinical engagement, supporting guidance and champions, dedicated staff capacity and flexible recording systems.

• Barriers to successful implementation included patients not being aware they were on PIFU, staff resistance, competing priorities and limited capacity to dedicate to PIFU, a lack of engagement with primary care, and challenges amending trust IT systems to record PIFU activity.

Impact on service use

• In 15 of 29 clinical specialties, we found evidence that greater use of PIFU is associated with reduced frequency of outpatient attendance for individual patients, most notably in Midwifery, Gynaecology, Urology and Pain Management. For seven specialties we found evidence that greater use of PIFU is associated with increased frequency of attendance, most notably in Ophthalmology. Within some specialties, although there were statistically significant changes, the practical effect was small. However, since most patients were not on PIFU, we may not expect to see a large impact across a whole specialty.

• Similar significant findings were not replicated when we analysed outpatient attendance per patient aggregated at the trust-specialty level, which may be because such an approach, using aggregated outcomes, would be less sensitive.

• These findings correspond to views from the study sites where impact on attendance is seen to be variable depending on how the PIFU pathways are implemented.

• We found no practically significant association between PIFU rates and frequency of ED visits overall (results were statistically significant but of negligible effect size), but for four specialties number of specialties appeared to have increasing PIFU rates associated with less frequent ED visits.

• Staff at local sites have noted fewer individual patient appointments within some specialties, whereas in others they expected no reduction because they gave patients on PIFU pathways the opportunity to arrange additional appointments as necessary, while maintaining fixed follow-up.

• Some local IT systems limit the degree to which services can evaluate their own use of PIFU.

• It is unclear whether any observed reductions in outpatient attendance are offset by increased use of telephone conversations or GP visits.

Staff experience

• PIFU was viewed by most staff as a positive intervention to support patient autonomy and self-management and ensure that time was directed towards those patients with the greatest clinical need.

• The impact of PIFU on individual staff roles and workload was mixed and dependent on the extent to which PIFU was used routinely within the service and considered a departure from previous ways of working.

Patient experience and engagement

• Across the case study sites, limited activity had taken place to capture formal feedback from patients, although staff reported that patients were positive about PIFU as an approach and the support they had received.

• Reasons reported by staff for patients declining PIFU included a preference for regular interaction, a desire to stick to their routine and a concern that they would be unable to get an appointment if they needed one. We were not able to speak to people who had declined PIFU as this information was not captured in the case study sites.

• Although we only spoke to four patients, those we spoke to were positive about their experiences and liked having the option to contact a specialist when they needed to.

• Staff were unsure whether patients were contacting the services when they needed to. However, when they did, this could be for several reasons not all of which required a face-to-face appointment with a consultant.

• Enablers to patient engagement included clear avenues of access and support, communication and ensuring patients do not feel abandoned by the service.

• Barriers to patient engagement included a lack of awareness and understanding about PIFU, the wider context on access to services and factors related to specific conditions.

Inequalities

• There was limited understanding of the impact of PIFU on different patient groups, although staff in the research recognised this as a concern. Digital exclusion, demographic characteristics, socioeconomic status and patient characteristics were all thought to be relevant to how patients engage with or are impacted by PIFU.

• Staff perceptions of the impact of PIFU on health inequalities were mixed. Some staff were concerned that PIFU could exacerbate inequalities because those with greater individual motivation and ability to advocate for themselves would be more able or willing to initiate contact when they needed. However, some services have used PIFU as a way to engage with patients who otherwise may not engage (or were more likely to miss appointments) by enabling them to contact the service at a time that was convenient for them.

• Data from one site indicated some variation in reattendance of patients on PIFU pathways within a year by ethnic group. However, it is unclear whether this reflects underlying ethnic barriers or other confounding factors. The difference in return rates between different adult age groups was small, although we were not able to isolate the oldest patients within the data. For children (ages 0–17), return visits of patients on PIFU pathways was approximately 50% higher than for adults.

Other contributions

Jenny Negus (Patient representative) PPIE contributions.

Preprints

Morris J, Georghiou T, Appleby J. Changes in English NHS outpatient activity during the early COVID-19 period. medRxiv. https://doi.org/10.1101/2021.04.28.21256176

Conference papers and seminars

Reed S. PIFU: emerging findings from a rapid evaluation. In: Harnessing the Value of PIFU in an Evolving Healthcare System. NHS Confederation Events. December 2022.

Hutchings R, Sherlaw-Johnson C, Georghiou T, Spencer J, Reed S, Crellin N, et al. Patient-initiated Follow-up: Findings from a Mixed-methods Evaluation. HSRUK 2023, Birmingham. July 2023. URL: www.youtube.com/watch?v=C-jAanv0JdY

Hutchings R. Patient Initiated Follow-Up – Learning from a Mixed Methods Evaluation. British Sarcoma Group conference, Leeds, February 2024.

Background

Each month, NHS trusts that submit data on PIFU activity to P-EROC will state whether their submission is complete or partially complete. For March 2023, the latest month in the anthesis, 58.5% of submissions were reported as ‘complete’ and 39.4% reported as ‘partial’ (see Figure 14, Measuring the impact of Patient-Initiated Follow-Up). (The remaining 2.1% were classified as ‘null’ or ‘unknown’.)

From interviews in phase 1 of the PIFU evaluation, and from views of stakeholders, we were told of potential inconsistency in how different trusts interpreted their returns as ‘complete’ or ‘partial’. For example, when trusts submit complete data for the specialties they decide to include but no data for other specialties where there are patients on PIFU, the issue is whether they classify this as ‘complete’, because the data is complete for the submitted specialties, or ‘partial’, because not all PIFU specialties were submitted. For this reason, we decided to compare levels of PIFU reporting between complete and partial submissions and determine if there were any substantial differences. This would then inform the decision about whether to include partial submissions in the analysis.

Methods

We used Poisson regression to compare levels of PIFU reporting between complete and partially complete submissions. The Poisson regression model used the number of people moved or discharged to PIFU each month as a dependent variable with type of submission and month as explanatory variables. The natural logarithm of the number of outpatient attendances each month was the offset variable. Both the provider trust and the treatment specialty were included as repeated measures.

Provider Elective Recovery Outpatient Collections has separate fields for numbers of patients moved to PIFU pathways each month and numbers discharged to PIFU. The sum of these numbers was the dependent variable. If both these numbers were not reported in a given month, the dependent variable was treated as missing. If one of these values was not missing, then that was used as the total. We limited the analysis to the top 30 specialties by numbers of patients moved or discharged to PIFU between September 2021 and March 2023.

Results

Monthly PIFU counts per attendance split by reported data completeness are shown in Appendix 7, Table 17. The mean uptake among partial submissions is approximately 18% compared to 8% for complete submissions.

Poisson regression analysis accounting for changes over time within provider trust and specialty, showed no significant difference in uptake between trusts reporting complete or partial submissions (8.5% increase – 95% CI 10.1% decrease to 32.2% increase). For this reason, we included data from both complete and partial submissions in the baseline analysis.

Discussion

The findings suggest that there is no difference in the ratios of patients transferred to PIFU each month per outpatient attendance. This provides evidence to support the view that the differences between complete and partial submissions maybe more due to how different organisations account for what specialties are left out of the data rather than the completeness of the specialties that are included.

This analysis has not been carried out within specialties, and there may be differences observed at that level. Also, it may be possible that trusts with more PIFU activity are more likely to report partial submissions, although we have no evidence that this is the case.

Introduction

The Poisson regression models relied on many assumptions relating to:

1. the lag between putting patients onto PIFU pathways and observed impact on attendance

2. whether we include P-EROC submissions that are reported to be ‘partially’ complete

3. how we handle missing data. In particular, where a trust has reported no data for a given specialty in a given month, when we know, either from other P-EROC months or HES that the trust has outpatient activity within that specialty

4. the denominator used for measuring PIFU rate: whether per outpatient attendance over the period or per unique patient.

In this appendix we present the impact these different assumptions have on the findings and so identify which are the more important areas of uncertainty.

The range of assumptions

The different assumptions are listed below. Those labelled ‘baseline’ were applied to our analysis presented in Measuring the impact of Patient-Initiated Follow-Up of the main report.

Additional outcomes and sensitivity analysis