Current experience and future potential of facilitating access to digital NHS primary care services in England: the Di-Facto mixed-methods study

The move to online primary health care services

The National Health Service (NHS) is under pressure due to the increasing demands of a growing and ageing population, technological progress and changing expectations among the public. Successive governments have viewed technology as part of a solution in addressing the pressures facing primary care, and policy for delivery of general practice has reflected this thinking. 1

Recent years have seen a push towards the adoption of online services in primary care, ranging from booking appointments and ordering repeat prescriptions through to the use of alternatives to face-to-face consultation with a patient (e.g. e-mail, video). By 2022, the reach of such services had extended to 22 million users. 2 Current NHS policy in England emphasises use of digital services in primary health care, and to date 28 million people have downloaded the NHS app, and 40 million people have acquired an NHS login. 2 Drivers behind the move to online primary care provision include the assumption that online services lead to improved choice, convenience, and ease of access for users, improved triage systems and streamlining of service delivery. 3

Context of this research: the COVID pandemic

In March 2020, the UK recognised the global health emergency of a pandemic associated with rapidly spreading COVID virus. In the UK, COVID led to widespread changes in society and within the healthcare system. From the perspective of this research, some relevant policy initiatives4 are summarised in Box 1. Overall, the pandemic was associated with increasing and substantial pressure for many staff in primary care, and for primary care within the healthcare system. The British Medical Association has estimated that numbers of full-time equivalent general practitioners (GPs) fell by nearly 6% between 2015 and 2021. 5 Moreover, the telephone first consultation, which was available pre-pandemic and widely adopted during the COVID pandemic, initially generated a 33% increase in the mean number of GP contacts per patient per 28 days,6 and during the pandemic increased telephone consultations from 39% to 51%. 7

General practice settings moved to using remote methods of consultation to enable safe delivery of care. The shift was dramatic, with a rapid change to 90% remote GP consulting (46% for nurses) by April 2020. 11 The researchers in that study noted a universal consensus that remote consulting was necessary. Telephone consulting was sufficient for many patient problems, video consulting was used more rarely, and appeared less essential as lockdown eased. Short message service (SMS)-messaging increased more than threefold. 11,12

Figures from 2022 for use of technology and online primary care services show an increasing use of NHS online services13 (increased from 44% of respondents reporting that they had ‘recently used’ such services, to 55% in the year to July 2022) – covering areas such as ordering repeat prescriptions online (26–31%), having an online consultation or appointment (18–22%), booking appointments online (19–21%) or accessing personal health records (7–17%). Geographical variation across England is evident in the application of online general practice services (GP services) by integrated care systems (ICS), with usage ranging from 46% of patients using an online GP service in the past 12 months to a maximum of 70% across 42 systems. Increases were evident in the proportion of patients reporting use of their GP practice’s website (from 36% in 2018 to 60% in 2022) with around 66% of respondents reporting the practice’s website being ‘easy to use’ in 2022.

Prior to the pandemic, research studies had established that for patients a face-to-face consultation is seen as the ‘gold standard’ and was the most used type of consultation. 13 The rapid change occurring in general practice means that digital services are now more widely available to all patients, and they may be expected to use them, regardless of digital confidence, proficiency and ease of access to the required technology and connectivity. For some patients, this change in patterns of access has been beneficial, for others it has been more challenging. 14

Moving to online provision – impacts and sequelae

The introduction of digital services and the move to a digital-first health service envisaged at the time this research was commissioned, raises important questions about digital access in the population. The research reported here has taken place in a changing policy environment – from the early days of a ‘digital-first’ vision for access to primary care services, through the ‘total triage’ model encouraged by NHS England during the COVID-19 pandemic to reduce footfall in general practices and to protect staff and patients, and, more recently, to a refined and nuanced vision of GP access, encompassing a range of modes of access, responding to the needs and ambitions of the NHS and of local patients and populations. In order for GP practices and patients to gain the potential benefits that technological innovation can bring to primary care,15 patients must be able to, and wish to, access and use online services. Digital inequalities tend to adversely affect certain groups of people. In this context, individuals from older age groups, non-white ethnicities, those in lower socioeconomic groups, those in poorer health and individuals in rural settings are recognised as vulnerable groups who may struggle to access digital services. 16–19 As online delivery of primary care is a key priority for policy-makers,2,20–23 it is important to understand how barriers to uptake might be overcome and inequality avoided. In attempting to mitigate the potential for digital inequalities, the research literature suggests that meaningfully involving users, tailoring services and interventions to target groups’ contexts, delivering credible messages and having a clear understanding of how services using technology improve health are key elements in developing relevant interventions. 24 One way to combat these potential inequalities is via digital facilitation (DF) – actively supporting patients and carers in using practice-based online services.

Challenges to patient engagement with online services

The reasons for the lower engagement of some sections of the population with digital and online health services are complex and include factors that limit access to technologies as well as factors affecting motivations to use the technologies. Specific barriers to engagement with online services for these groups include a lack of experience with using the internet,37,51 lower health literacy37,52 and a lack of trust towards the information being provided through online interfaces. 37,53 Issues relating to ‘usability’ can also impede older users from accessing information through patient portals. For example, when older users lost required access codes after registering on patient portals, they became discouraged in their use of the service. 37 Research in Scotland has identified technical and practical considerations (poor connection, ‘frozen’ images, poor sound quality, slow broadband) in adopting IT innovations,54,55 including amongst rural populations, who may have limited access to good quality broadband services. In research undertaken in homeless populations,56 qualitative research findings have highlighted the importance of addressing practical and technological barriers as well as supporting communication and choice for mode of consultation. In their reviews of the literature, Irizarry et al. 52 and others23 have concluded that the ability of patients to access online health services is strongly influenced by combinations of personal factors such as health literacy, health status, age, ethnicity, education level and whether individuals have caring responsibilities. 57 Likewise, in a systematic review of qualitative studies examining the factors affecting patient recruitment to digital health interventions, O’Connor et al. 58 concluded that greater investment is required to improve computer literacy to ensure that technologies are accessible and affordable. Cognisant of these issues, we have been involved in research examining the unintended consequences of providing some online services in primary care,59,60 and investigating patient use and experience of online booking in general practice.

Supporting the move to online services

For the purposes of this research, our focus is on services accessed via a primary care practice website (e.g. booking appointments, access to records) and via online platforms provided by general practices. From 1 November 2022, patients were due to have online access to new entries in their personal medical records,61 although this initiative has been delayed in implementation at the time of writing.

Outcome

The ultimate outcome of the project is a summary of the current status of DF as presently implemented within primary care. This includes what is known about the likely effectiveness, cost and equity of access implications of the approaches identified, and an indication of the prevalence of various approaches in four regions of England (East of England and North London; South-West; West Midlands;). We have provided recommendations for future development and implementation of promising approaches to DF, and a framework for future evaluations to assess the effectiveness, cost effectiveness and impact on inequalities of access to the online services, of relevant facilitation approaches within primary care settings.

Theoretical framework

We have used Weiss’s theory-based evaluation as our theoretical framework to understand how, and in what ways, different models of DF bring benefits and challenges to general practice. 70 Weiss distinguishes between ‘programme theory’, which specifies the mechanism of change, and ‘implementation theory’, which describes how the intervention is carried out.

We have done this by drawing on the findings of the evidence synthesis, surveys and case studies to develop the ‘programme theory’ and ‘the implementation theory’.

To develop the ‘programme theory’ we used a realist approach to describe provision of DF. A realist approach asks ‘What works for whom, in what circumstances, in what respects, and how?’71

We explored this in terms of:

• context (e.g. characteristics of the general practice, the target patient population, the policy framework, and the IT infrastructure)

• the theory and assumptions underlying the intervention (how and why DF might lead to benefits)

• the flow of activities that comprise the intervention (the key processes that occur when patients make use of DF)

• intended benefits/outcomes (those deemed important to patients and practitioners).

The ‘implementation theory’ explored moderating factors which influence the extent to which the process and outcomes were achieved, such as factors acting as barriers and facilitators to practices offering DF or to different groups of patients using them.

Patient and public involvement and engagement

This research is underpinned by recognition that patient and public involvement and engagement (PPIE) is an integral part of research practice. Patients’ voices should be reflected and addressed in the design and delivery of health research in order to ensure that research outputs are meaningful and relevant to them. As a research team we have followed the UK standards framework for public involvement in research including: inclusive opportunities, working together, support and learning, governance, communications and impact. 72

From the outset we have worked with patients to shape the scope and objectives of this research and continued to work collaboratively with patients throughout the research. Patients and carers have been active partners and included at all stages of the research process, including in setting the research agenda and analysing data. 73 The approach to involvement, the impact on the research and reflections on how it worked are included throughout the report.

Searches

Screening

Publications for Stages 1, 3 and 4 were restricted to English-language articles published since 2015. We initially included articles from 2010 to January 2020; however, the resulting number of included publications from Stage 1 (n = 154) exceeded the scope of the project. We therefore then restricted to publications published since 2015, with the understanding that DF and widespread use of online services in primary care are relatively recent phenomena and therefore the most relevant literature was still likely to be captured by our revised inclusion criteria. For Stage 2, where publications were identified through snowball-type searches of reference lists, we expanded the eligibility to 2010–20 so as not to omit references still evidently deemed important post 2015.

A key inclusion criterion for all publications was that they addressed facilitation of online services. We operationalised this criterion as detailed in Table 2. Further eligibility criteria were tailored to the stage of the screening process (e.g. primary care literature, non-healthcare sector literature). The detailed inclusion and exclusion criteria for each stage of the screening process are presented in Table 1.

Prior to full screening, we undertook a pilot exercise examining 2% of the 11,853 publications from Stage 1 (n = 237), during which publications were jointly screened by two reviewers (EG and SP). The reviewers compared and discussed their results, and amended the inclusion and exclusion criteria, raising any points of concern with the members of an internal advisory group convened to advise on the scoping review. This group comprised members of the research team who were not directly involved in the literature search and screening process, but who have expertise in literature reviews (see Acknowledgements).

Data extraction and preliminary analysis

Data from eligible studies were extracted independently by two reviewers (EG and SP) using a data-charting form developed for this study. The form was piloted to ensure data extraction was consistent across reviewers. We extracted data relevant to DF (digital technology type, facilitation purpose, method, mode of delivery, target population, setting) and study details (study type, outcomes, size, setting), aiming to capture health outcomes; staff and patient/carer experience; impact on service use, cost and equity of access to healthcare services and information; and the nature and extent of other reported outcomes. Studies were not formally assessed for quality as this was a scoping review and furthermore, the breadth of study and article types included makes the use of formal quality tools impractical. However, reviewers noted the quality of the evidence source, clarity of aims, quality and comprehensiveness of the work, and any conflicts of interest from the authors (see Appendix 1, Table 28 for details) to assist with judging the quality of the overall evidence base for DF. A list of all data fields captured in the form is available in Appendix 1 (see Table 28).

During data extraction the team met frequently to discuss emerging findings, resolve uncertainties regarding the boundaries of DF activities and clarify eligibility criteria.

Before conducting further searches (see Figure 2), we undertook preliminary analysis of the extracted Stage 1 data, including in a study team workshop (attended by EG, BL, SP and JS) to identify themes emerging and gaps in the literature that might indicate a need to expand or alter the scope of our search during later stages.

We then discussed the identified themes and gaps at a second workshop with seven members of the PAG. The PAG members suggested that a barrier to DF could be the use of locums. This was an important gap in our preliminary analysis and one that was under-represented in the Stage 1 literature. The use of locums, and high staff turnover more generally, is an important potential barrier to DF because these staff might not have the proper training or buy-in for DF programmes. The PAG members at the workshop also confirmed the legitimacy of some themes we identified in the literature, including that awareness of online services by GP practices and training for staff members may be important enablers of DF, and that data privacy or security concerns by GPs or patients might hamper DF efforts. The PAG participants highlighted that although it can be challenging for members of some vulnerable patient groups to access or use online services, for others, such as those with limited mobility, online services might improve access to health care. These discussions aided in the identification of themes in the literature and later data synthesis.

Data synthesis

Data analysis followed the principles of narrative descriptive synthesis. 79 We began by identifying key themes captured during charting, which were then refined and expanded upon during preliminary synthesis. We did this through a process of study team discussions, analysis and writing. Preliminary synthesis was followed by further refinement of themes through a workshop including members of the wider study team, including a PPIE representative. The narrative synthesis further aimed to characterise and differentiate between different types of facilitation and to synthesise evidence relating to effectiveness or cost effectiveness, inequalities of access to online services, or potential advantages and challenges of different approaches.

Typology of digital facilitation

There are a wide variety of DF efforts discussed in the literature. Table 3 illustrates the typology we developed based on our findings. Although DF efforts are usually directed at patients, some are aimed at primary care staff to enable them to better support patients in using digital services. The DF efforts discussed in the literature review include those that are part of routine service delivery, and those that were introduced as part of an experimental study where facilitation was implemented for research purposes.

Is digital facilitation associated with increased uptake and use of digital services?

The evidence relating to whether different DF approaches increase uptake and use of digital services is summarised in Table 4 and described below.

What makes digital facilitation successful?

The success of DF is influenced by the following factors.

Deviations from the original proposal

There were no deviations from the original proposal.

Strengths and limitations of the current literature review

We conducted a scoping review. This was appropriate given the exploratory nature of the study and our aim to capture the breadth of DF in primary care. We employed an iterative approach to identify literature relevant to DF in primary health care, including literature about health care more widely and, indeed, about non-healthcare sectors where that may have useful implications for DF in primary health care. Our focus was on the DF discussed, rather than on the quality of the study. Our approach was strengthened by PPIE inputs.

The limitations of our approach are evident from the inclusion and exclusion criteria presented earlier. In addition, by limiting the review mostly to articles published since 2015 we may have omitted earlier documents of interest. However, given the acceleration in recent years of the availability of online services for patients, we expect to have captured the large majority of relevant literature. The last few months of our literature review period overlapped with the first few months of the global COVID pandemic, but due to publication lags, we found very little literature that reflected the impact of the pandemic on the use of online services.

Gaps in the evidence base

We identified a number of areas that were not well addressed. We found few formal evaluations of DF effectiveness. While some studies examined the effects of particular DF efforts, most did not evaluate the benefits or costs of existing facilitation efforts; rather, their main focus was on ways of overcoming barriers to the uptake and/or sustained use of online services.

There were nevertheless a few formal evaluations of DF in the literature we reviewed, including some RCTs. Due to short time horizons, these studies concentrated on initial access to a service, such as registering with an online portal. There is little evidence available on how DF affects ongoing use of services, such as estimating how many more patients would routinely book their GP appointments online.

We found little comparative evidence on which types of facilitation are most effective, in what context, or for whom. Evidence of the effectiveness of DF in routine use in primary care settings is also thin, since most of the articles in our review reported on DF efforts introduced as part of research studies.

A key aim of the scoping review was to identify evidence about which, if any, models of DF are effective at reducing inequalities in the use of online services. But we found little evidence regarding the impact of DF with vulnerable groups. Just one article directly addressed the topic and a handful of other studies discussed differences in outcomes between population subgroups. Given the potential for online services to increase inequalities in access to health resources, this represents an important weakness the evidence base.

We found no studies that evaluated the cost effectiveness of a DF method, or even that quantified the costs. Some articles described time constraints in GP practices as a barrier to undertaking DF, which is presumably related to costs as well as workforce demands. However, a clearer understanding of the costs associated with different facilitation approaches, along with their effectiveness, would be important for assisting primary care practices in decision-making around implementing DF approaches.

Finally, evidence on the impact of COVID on DF is still emerging. Understanding the pandemic’s impact on the provision of digital services in primary care and how practices facilitate access to those services is a current challenge.

Suggestions for future research

Based on what we did not find, there is scope for more, and more formal, evaluations of approaches to DF. The emphasis in these evaluations could usefully be on the comparative effectiveness of different approaches, and with particular attention to the ability to sustain patients’ use of online services beyond initial sign-up. Given the push by the NHS for primary care practices to move services online,156,157 the lack of evidence about how best to facilitate patient access to these services is a significant gap to fill through future research.

The absence of economic evaluations of DF approaches is striking. There is a clear need for assessment of the costs of facilitation and the impact on the costs borne by health services, practices and patients as a result. Assessments of the cost effectiveness of DF and of different approaches in comparison with one another are much needed, to guide the use of scarce primary care resources.

Finally, although the literature we reviewed contains some examples of consideration of the differential effects of examples of DF on various subgroups of the population, the evidence overall about impact on (in-)equality of access to health care, particularly for vulnerable groups, is weak. It falls well short of providing clear guidance on how to avoid worsening inequalities, let alone reduce them, by means of DF. Starting to rectify that lack of guiding evidence is an important priority for future research.

Practice survey instrument

A questionnaire was designed to be completed in approximately 10 minutes by either practice managers or GPs and was produced in paper (postal) and online (using SmartSurvey Online Software and Questionnaire Tool) formats.

The practice survey questionnaire (see additional files www.fundingawards.nihr.ac.uk/award/NIHR128268; accessed 3 April 2024) was developed by a subgroup of the research team, with input and feedback from the PAG, the wider research team and additional GPs. The questions were developed using an iterative process, informed by the findings of the Di-Facto literature review (see Results). The questionnaire included questions addressing the range of online services offered to patients (currently and prior to the COVID pandemic), the activities used to promote online services or to support patients in using them, the staff involved in the provision of any support to patients and whether specific groups were targeted with that support. The questionnaire also sought the respondent’s views on responsibility for support, the influence of various factors on service provision and the rationale behind providing online services. There was also one question which gave the opportunity for a free-text response, inviting comment on anything else that the respondent saw as relevant.

Patient survey questionnaire

Patient survey questionnaire development commenced with a scoping meeting with the PAG, followed by a series of iterative workshops with the patient survey team. The questionnaire (see additional files www.fundingawards.nihr.ac.uk/award/NIHR128268; accessed 3 April 2024) drew on the literature review findings and adopted (revised) wording from existing questionnaires on eHealth literacy,159 the GPPS and the Get Digital basic skills assessment. Where possible, the patient survey content mirrored the practice survey questions (tailoring wording for patients with PAG input) to allow for direct comparison. It included sections exploring patients’ digital confidence, their awareness and uptake of online services and experience of the support provided by the practice to use online services. The questionnaire was piloted by volunteers. Feedback was that it was straightforward to complete; however, some words and layouts were subsequently changed for clarity.

For the Mental Health Boost sample, the questionnaire was identical to the main part of the patient survey. We consulted with a wider PAG, who had experience of mental health conditions, to identify acceptable language for the patient invitation letter and information sheet, to explain to patients how they would be identified, and to explore how to safeguard privacy when opening the invitations. The researchers involved with the mental health survey were experienced in and acutely aware of mental health issues and dealt very cautiously and sympathetically to the few patients where any concerns were raised.

General Practice Patient Survey

The GPPS is a national postal questionnaire that is sent to 2.3 million patients annually on behalf of NHS England. Further details of the survey and its development can be found elsewhere. 13,160,161 Data from the 2019, 2020, 2021 and 2022 GPPS162 were analysed both on their own and in relation to the practice survey. 160,163–166 We restricted the data to those who reported attempting to make a general practice appointment in the last 12 months to ensure that responders had the opportunity to use online services. We made use of two questions about the awareness and use of online services:

• As far as you know, which of the following online services does your GP practice offer? Booking appointments online, ordering repeat prescriptions online, accessing my medical records online, none of these, don’t know.

• Which of the following general practice online services have you used in the past 12 months? Booking appointments online, ordering repeat prescriptions online, accessing my medical records online, had an online consultation or appointment (e.g. completed an online form or had a video call).

The question on awareness of online services was removed from the questionnaire in 2021 and so responses from 2019 to 2020 were used for this question, while data from 2021 to 2022 were used for the analysis concerning use of online services. In addition, we used responses to questions addressing ease of getting through on the phone, helpfulness of practice receptionists, patients having a preferred GP and ease of using the GP practice website, as well as patient reported demographics, health status and the Index of Multiple Deprivation (IMD) based on the patient postcode of residence.

Descriptive statistics

Practices that took part in the practice survey were compared with other practices in England considering deprivation, rurality, list size and the percentages of registered patients over the age of 65 and (separately) with a non-white ethnicity. Deprivation and ethnicity data were obtained from the Public Health England Fingertips website167 based on 2021 data; list size and patients over 65 were obtained from NHS Digital December 2020 data, and rurality was based on practice postcodes as used in the 2019 GPPS. Descriptive statistics were produced for responses for the practice and two patient surveys. Following factor analysis indicating a single construct underlying the responses to Q2–Q6 of the patient survey (see Appendix 2, Tables 30–32 and Figure 17), a composite measure of digital confidence was formed from these questions. For each question, a response of ‘Not very confident/I can’t do this’ scored 0, ‘Quite confident’ scored 1 and ‘Very confident’ scored 2. Scores for the five questions were summed giving a score between 0 and 10. This was further split into three categories of 0–3 (not confident), 4–7 (quite confident) and 8–10 (very confident). For the subsample of responses to the patient surveys where double data entry was performed, the number and percentage of data items where there was disagreement between the primary and double data entry were recorded overall and by question.

Comparisons of the percentage of patients reporting being aware of and using different modes of facilitation in the main patient survey were made between practices who reported using the corresponding mode of facilitation in the practice survey. Similar comparisons were made for patients being aware of, and of using, any mode of facilitation between practices who reported undertaking ad hoc promotion or support and those who did not, and between those who reported employing a practice champion with specific responsibility for supporting or facilitating online services and those who did not. Finally, comparisons were made of awareness and use of any modes of facilitation between patients registered at practices that reported targeting older adults (65 years or older), people with physical health conditions, mental health conditions, limited or no internet access, non-English speakers/English as a second language, ethnic minorities and carers, restricting comparisons to patient respondents reporting that they belonged to that group.

Frequencies and percentages of awareness and use of online services by patient groups were calculated using national GPPS data using weights supplied with the data which accounted for sampling design and non-response bias.

Regression analyses

Analysis of free-text responses in the patient survey

We examined all the free-text responses for the main patient survey (i.e. not mental health boost sample) to the following questions:

• Q16: What can the practice do to help you access the online services?

• Q17: Is there anything else you would like to add about online services and the support of the practice to help you use them? If so, please tell us about it here.

We conducted a text-based analysis to reflect the topics emerging from the data. One researcher (NK) initially read and re-read all of the responses to familiarise herself with the data. The data were then coded to capture key concepts from the free-text responses. A second researcher (EP) independently read and re-read and double coded the responses. Both researchers met to discuss the initial coding to compare and contrast their findings and to resolve discrepancies. Codes were then organised into higher level themes based on the content and relationship between the codes. We used NVivo 12 (QSR International, Warrington, UK) qualitative analysis software to manage the data.

Practice survey

Of the 499 practices invited (2 were latterly found to have merged at the time of invitation), 156 (31%) sent back one or more questionnaires (in total 168 questionnaires were returned). This included 19 practices who had participated in the pilot version of the survey. Compared with other practices in England, participating practices were more likely to serve less deprived populations (26% in the least deprived quintile vs. 20% in other practices), to be in rural areas (30% vs. 15% in other practices) and have over 12,000 registered patients (35% vs. 23%). Smaller differences were seen in the age profile and ethnicity of the registered populations (see Appendix 2, Table 33).

Summaries of the responses to the practice survey are shown below (full details Appendix 3, Tables 34–43). Figure 4 and Table 34 show the online services that practices reported offering relative to the current COVID pandemic. All services were currently offered by over 80% of practices with the exception of online access to test results (77%) and online appointment booking (51%). There was a substantial reduction in online appointment booking provision associated with the pandemic (44% of practices offering the service pre-pandemic, but not at the time of the survey, due to temporary suspension). All other services had seen an increase in provision with between 25% (online access to test results) and 86% (video consultations) of practices providing a service that was not available pre-pandemic.

FIGURE 4.

Online services offered by practices responding to the practice survey pre coronavirus and at the time of the survey.

There was a clear division in the survey between the endorsement of modes of DF that might be described as passive (displays, leaflets, text messages, e-mails, social media and material on practice websites) and those that might be described as active (ad hoc support, employing a practice champion, holding workshops/events and the provision of tablets/computers). Passive modes of facilitation were reportedly used for either promotion of online services and/or supporting patients to use them in a majority of practices (see Figure 5, also Appendix 3, Table 35). In contrast, most active modes of facilitation (employing a practice champion, holding workshops/events and the provision of tablets/computers) were reported by only a minority of practices. An exception to this observation was the provision of ad hoc support which was reported in 91% of practices. Most practices reported using DF across a wide range of online services with 97% reporting using it to promote or support online repeat prescription ordering (see Appendix 3, Table 36).

FIGURE 5.

Digital facilitation activities used to promote online services and support patients to use them.

The vast majority of practices reported DF being carried out by administrative (86%) and reception (86%) staff, with doctors (62%), nurses (53%) and other healthcare professionals (42%) also frequently reported as being involved. IT staff (28%) volunteers (20%), external contractors (4%) and those with specific responsibilities around DF were reported less often (see Appendix 3, Table 37). Around 22% of practices reported targeting their facilitation activities at every patient group considered, with most practices reporting targeting multiple groups. However, the only specific group targeted by most practices were older adults (56% of practices) (Table 5).

Most practices reported either reducing or removing in person access to services (including the provision of paper prescriptions) following the national lockdown starting 23 March 2020 (see Appendix 3, Tables 38 and 39). However, although most practices had not changed phone access, 10% of practices reduced access to booking appointments by phone, and 28% reported removing (13%) or reducing (15%) access to ordering repeat prescriptions on the phone.

FIGURE 6.

Level of agreement with statements about DF by practices responding to the practice survey. (continued)

Most practices agreed in the survey (selecting ‘either agree’ or ‘strongly disagree’ when asked if they agreed with a statement) (see Figure 6, also Appendix 3, Tables 40–42) that:

• COVID had been a crucial driver in the uptake of online services by patients

• COVID had led to an increase in the support they gave to patients to use online services

• it is the practices’ and NHS’s responsibility to inform patients about online services and provide support

• support provided to patients was in response to patient demand rather than from the CCG, practice staff or monetary incentives

• supporting patients to use online services benefited the practice and patients

• they lacked the adequate capacity to provide this support

• they had increased the uptake of online services by supporting patients to use them and that they hoped to further increase uptake through support

• some patients were unlikely to use online services regardless of the support provided

• some patient groups required more support than others

• increased patient uptake of online services led to operational efficiencies for the practice

• online access to primary care services was complementary to traditional forms of access.

Analysis of free-text comments made by (staff) responders throughout the practice survey identified two main themes which illustrated issues reported:

• COVID had had an impact on digital services: many services made a shift to being mainly online. However, COVID also curtailed a few attempts to provide support (e.g. provision of workshops prior to COVID).

• There was a relatively negative attitude to implementing digital services in terms of limited practice capacity, finances and motivation:

GP practices do not have the capacity or resources to facilitate online access.

Practice 1455, GP

• There was a recognition that guidance, support and funding from CCGs or at the national level was required in order to provide more help to patients:

[F]unding would be useful to provide tablets/pc for waiting rooms where patients could access online/self- help/information/advice.

Practice 1451, Nurse

It makes sense for support to be delivered at a national level and outside of GP practice, as is the case with the NHS App.

Practice 1385, GP

Patient survey

Sixty-two practices participated in the patient survey, sending invitations to 12,822 patients, of whom 3051 (23.8%) responded (see Appendix 5, Table 49). Analysis of double data entry detected a difference of 0.4% in data points entered which was deemed acceptable. Response rates were lower in practices serving deprived populations; however, due to the approach to sampling we adopted, responses were reasonably representative in terms of deprivation (Figure 7).

FIGURE 7.

Main patient survey: invitations and response by deprivation quintile.

Of responders to the main patient survey, slightly more were female than male, 45% were 65 or older, the vast majority were white (93%), 9% reported that English was not their first language, 44% were working either full-time or part-time, and 43% were retired (Table 6).

Nine per cent of responders reported not having access to the internet at home, and nearly all of these completed a paper version of the survey (see Appendix 5, Table 50). The percentage of responders not confident or unable to complete digital tasks ranged from 17% for using search engines to 31% for installing apps (see Appendix 5, Table 50). The lack of confidence was strongly associated with response mode, with at most 4% of online responders lacking confidence in these tasks.

Figure 8 shows a clear bimodal distribution in the digital confidence score constructed from responses to questions 2–6. Almost 40% of responders scored 10, implying they feel very confident in undertaking all of the tasks considered, while around 12% of responders scored zero, implying that they are not very confident/cannot complete any of the tasks considered.

FIGURE 8.

Distribution of digital confidence score for responders to the patient survey.

Table 7 provides a summary of responses to the core patient survey questions (i.e. not those concerning digital confidence or patient characteristics). Over a third of responders (37%) had not tried to use their general practice’s website, though most of those that had done so found it easy to use. Awareness of online services was generally low (≤ 37% of respondents were aware) for all online services apart from booking appointments and ordering repeat prescriptions online (55% and 64% respectively) and use of online services was universally low with at most 33% of patients reporting having ordered repeat prescriptions online. Awareness of DF efforts was low, with 17% or less being aware of any modes apart from texts or e-mails for which 39% of responders were aware of their use. The use of those facilitation efforts was even lower, with < 10% of patients making use of any mode apart from e-mail or text messages (28%).

Only 30% of patients reported that they had been told about online services by someone at their general practice and only 13% reported being helped to use them. However, of those that did have help, they generally rated it as helpful (90% said ‘quite’ or ‘very helpful’ for repeat prescriptions, and 65% said ‘quite’ or ‘very helpful’ for video consultations). Various reasons were selected for not using online services, but most common was preferring to speak to a person (47%) with 20% either not knowing how to register (14%) or finding registration too difficult (6%).

The adjusted (Model 1) logistic regression analyses are shown in Table 8 with unadjusted results shown in Appendix 4 (see Table 44). Age was positively associated with a lower likelihood of all four outcomes (p < 0.05 for all), particularly for being helped to use online services (adjusted OR for ≥ 85 vs. 55- to 65-year-olds 0.08, 95% CI 0.02 to 0.36; p = 0.006). Patients in receipt of repeat prescriptions were more likely to have experienced all four facilitation outcomes (p < 0.05), for example, the adjusted odds ratio (OR) for being told about online services was 1.70 when compared with those not in receipt of repeat prescriptions (95% CI 1.36 to 2.13; p < 0.001). Patients of non-white ethnicity were more likely than those patients describing themselves as white to be aware of DF, have made use of DF and been helped to use online services, but no difference was seen between white and non-white patients for being told about online services (adjusted OR = 0.91, 95% CI 0.61 to 1.36; p = 0.651). Similarly, people for whom English was not their first language were more likely to use DF and be told about online services, and with very weak evidence (p < 0.1) of similar differences for the other two outcomes. Gender, being deaf or having a hearing impairment, and working status were not associated with any of the four outcomes (p > 0.1). Including the digital confidence measure in the models (see Appendix 4, Table 45) showed that lower digital confidence was associated with lower awareness and use of facilitation, and a lower chance of being told about, or being helped to use, online services. Comparing Model 1 with 2 we found that digital confidence explained some of the difference attributable to older age for awareness and use of facilitation and being told about online services. However, the reductions in odds ratios were fairly small (e.g. OR for 85 + awareness of facilitation reduced from 0.32 to 0.59). In contrast, digital confidence hid some difference attributable to older age in being given help indicated by ORs becoming larger when adjustment for digital confidence was made. Digital confidence also masked some differences in the four outcomes between those who spoke English as a first language and those who did not. Sensitivity analyses excluding those who reported having helped to complete the questionnaire did not lead to materially different findings (see Appendix 4, Tables 46 and 47).

1. Awareness of any facilitation efforts includes respondents who ticked any of the first seven options of Q10.

2. Use of any facilitation efforts includes respondents who ticked any of the first seven options of Q11.

Considering digital confidence as an outcome (see Appendix 4, Table 48), we found a strong age gradient, with over 85-year-olds much less likely to be very confident than 55- to 64-year-olds (adjusted OR = 0.04, 95% CI 0.01 to 0.11; p < 0.001). We also found that parents were much more likely to be digitally confident than non-parents. People with long-term health problems, whose first language is not English, who are retired, or not in work/education were less likely to be digitally confident.

Combined analyses of the practice survey and main patient survey

We found only limited evidence of differences between the reported awareness, and use of, different modes of DF (as reported in the patient survey) by patients registered at practices who reported using the corresponding mode for either promotion of, or supporting patients to use, online services (as reported in the practice survey – Table 10). There was strong evidence (p = 0.004) that patients registered at practices who reported using in-practice displays to support DF were more likely to be aware of such displays (18% vs. 11% of patients). Similarly, awareness and use of social media for DF were higher in those practices that reported using it (6% vs. 1% for awareness and 4% vs. 1% for use) and awareness and use of DF workshops/events were higher in those practices that reported using them (1.5% vs. 0.5% for awareness and 1.5% vs. 0.4% for use). No evidence of differences in patient awareness and use was seen between practices reporting using leaflets, text messages/e-mails, putting content on practice websites and providing tablets or computers, and those that did not. There was no evidence of differences in the patient awareness and use of any mode of DF between those practices that reported using ad hoc support for DF and those that did not (see Appendix 6, Table 51). Employing a practice champion was associated with higher patient use of facilitation but no greater awareness of that form of facilitation. Furthermore, there was no evidence of differences in the patient awareness and use of any mode of DF within any particular patient groups where practices reported targeting that group (see Appendix 6, Table 52).

Mental health boost sample

Of the 26 practices (who had already participated in the main patient survey) invited to participate in the mental health boost sample element of the patient survey, 18 agreed to take part (3 serving deprived populations). In total, 3456 patients were sent questionnaires (maximum of 200 per practice) of which 756 (21.85%) responded (see Appendix 7, Table 53). Analysis of double data entry detected a difference of 0.4% in data points entered.

Table 11 shows the demographics of responders to the patient survey mental health boost sample along with the responders to the main survey registered at the same general practices. Sixty-one per cent of those responding to the mental health boost sample survey reported that they had a long-term mental health condition. A higher percentage of responders to the mental health boost sample survey were female, aged 25–64, reported a long-term physical health condition and were much more likely to be in receipt of repeat prescriptions (94% vs. 66% in the main survey).

A summary of responses given by respondents to the patient survey mental health boost sample are shown in Table 12. Responders to the mental health boost sample survey reported higher levels of digital confidence, and they were considerably more likely to have used practice websites than responders to the main survey. Reported awareness and use of online services were generally higher for those responding to the mental health boost sample survey compared with those responding to the main survey (with the exception of awareness of booking appointments online) as was awareness and use of DF efforts (with the notable exception of social media). Responders to the mental health boost sample survey were also more likely to have been told about online services and helped to use them. Finally, a higher percentage of responders to the mental health boost sample survey endorsed all reasons for not using online services, with the exception of not having internet access.

The higher awareness and use of facilitation efforts by responders to the mental health boost sample survey, along with increased likelihood of being told about online services, were confirmed by univariable logistic regression (see Appendix 7, Table 53), but the difference in terms of being helped to use online services was not statistically significant. In adjusted models the only difference between responders to the mental health boost sample survey and the main survey that remained statistically significant was awareness of facilitation (see Appendix 7, Tables 54 and 55) indicating that observed differences in use of facilitation and being told about digital services may be related to differences in the demographics of the samples.

Findings from analysis of national General Practitioner Patient Survey data

Of the 1,510,149 responders to the 2019 and 2020 national GPPSs and the 1,569,343 responders to the 2021 and 2022 national GPPSs, 1,212,801 (77.3%) and 1,304,574 (86.4%) reported trying to make an appointment at their GP practice in the 12 months prior to responding to the survey, respectively.

Among this subset of responders, awareness of online appointment booking and ordering repeat prescriptions online was moderate (2019–20 percentages weighted for non-response = 49% and 45%, respectively), but awareness of the ability to access medical records online was low (2019–20 weighted percentage = 18%). Use of all three services (based on 2021–2 data) was lower at 23%, 32% and 13%, respectively. Around a quarter (25% in 2021–2) of responders reported use of online consultations.

The results of the adjusted logistic regression models are shown in Figures 9–13 and in Table 56, Appendix 8. There was a strong decrease in the awareness and use of all services (making appointments online, ordering repeat prescriptions online, accessing medical records online, and online consultations) with older age; individuals aged over 85 years were much less likely to be aware of, or to report using, online services than younger people (see Figure 9). For ordering repeat prescriptions online, there was a smaller reduction in awareness and use in younger responders than in 55- to 64-year-olds, but for the other services awareness plateaued, and use increased, in all but the youngest responders. Similarly, there were very strong deprivation gradients in the awareness and use of all online services (see Figure 10). Females were more likely to be aware of all three services considered (appointment booking, repeat prescriptions and access to medical records) and more likely to use online consultations and order repeat prescriptions online, but less likely to actually book appointments online or access medical records online (see Figure 11). All non-white ethnic groups were less aware of all three online services that were considered than white responders and were generally less likely to actually use the services, with the exception of booking appointments online where higher use among ethnic minorities was seen (see Figure 12). Those with long-term conditions were more likely to be aware of and use all online services, particularly ordering repeat prescriptions (see Figure 13). Finally, we found that the practice in which a patient was registered was, by far, the strongest predictor of awareness and use of online services (Table 13). For example, the OR comparing the practice with the highest awareness of online booking with the lowest among the middle 95% of practices (i.e. ignoring the most extreme) was 16.2 (95% CI 15.2 to 17.0).

FIGURE 9.

Variation of awareness (red) and use (blue) of online services by age obtained from adjusted logistic regression models (odds ratios and 95% CIs).

FIGURE 10.

Variation of awareness (red) and use (blue) of online services by deprivation obtained from adjusted logistic regression models (odds ratios and 95% CIs).

FIGURE 11.

Variation of awareness (red) and use (blue) of online services by gender obtained from adjusted logistic regression models (odds ratios and 95% CIs).

FIGURE 12.

Variation of awareness (red) and use (blue) online services by ethnicity obtained from adjusted logistic regression models (odds ratios and 95% CIs).

FIGURE 13.

Variation of awareness (red) and use (blue) of online services by long-term condition status obtained from adjusted logistic regression models (odds ratios and 95% CIs).

Augmenting the regression models with patients’ experiences of other aspects of the practice (see Appendix 8, Table 57) found that:

• Patients who say it is easy to get through on the phone tend to have lower awareness and use of online services than those who say it is not at all easy.

• Patients who report receptionists being helpful tend to have higher awareness but lower use of online services, than those who report receptionists being not at all helpful.

• Awareness and use of all online services is most often reported for patients who have a preferred GP for some appointments. With the exception of online consultation use, this is followed by those who report a preferred GP for all appointments, with those who do not have a preferred GP least likely to report awareness or use of online services.

• Patients who say the practice website is easy to use are much more likely to report awareness and use of online services.

The effect of ease of use of practice websites was particularly strong within the models of awareness and use of online provision; the ORs comparing those who rated practice websites as ‘very easy’ to use compared with those who said it was ‘not easy at all’ being 5.29 (5.12, 5.48), 5.73 (5.54, 5.93) and 4.07 (3.92, 4.24) for awareness of online appointment booking, ordering repeat prescriptions and accessing medical records online, respectively. Because these ORs may reflect individuals’ general difficulty with using websites we also repeated models replacing person-level responses with a practice score for ease of website use. In that case the odds ratios were more modest, but still substantial at 1.40 (95% CI 1.32 to 1.47), 1.51 (95% CI 1.45 to 1.58) and 1.45 (95% CI 1.38 to 1.52), respectively.

In the final set of models examining GPPS data in isolation (see Appendix 8, Table 58) we looked at whether reported ease of use of practice websites modified the effect of sociodemographic factors. We found evidence of interactions between age, deprivation, gender, having a long-term condition, rurality and one or more of awareness of online appointment booking, use of online appointment booking, or awareness of repeat prescriptions. In all cases good websites were associated with higher awareness and use and the direction of demographic associations did not change for when practices had good or poor websites, though the magnitude of the associations did change, but only to a modest extent. Age, ethnicity, long-term condition and rurality differences were slightly larger when websites were rated as good compared with when they were rated poorly. In contrast, gender and deprivation differences were slightly smaller when websites were rated as good compared with when they were rated poorly.

The results of the analysis of GPPS data from practices who participated in the WP2 practice survey are shown in Appendix 8, Table 59. After adjustment for patient characteristics, we find that 7 out of 63 associations examined are statistically significant at the p < 0.05 level and as such care must be taken with the interpretation of these findings. However, we do see a consistent effect whereby patients registered at practices who report the use of leaflets for DF are less likely to report being aware of online services (appointment booking, medical record access and repeat prescriptions) than those registered at practices that did not report use of leaflets.

Practice survey

Most practices reported offering a range of online services. With the notable exception of online appointment booking (which was withdrawn nationally in response to the COVID pandemic), the number of practices offering them had substantially increased during the COVID pandemic (particularly video consultations). While most practices reported using passive modes of DF to promote or support patients’ use of online services (social media), more active modes of DF (e.g. employing a practice champion), were rarely reported. In the vast majority of practices, administrative and reception staff conducted DF efforts, with clinical staff contributing in more than half of practices. Most practices reported targeting their facilitation efforts at older adults. While most practices were generally positive about DF, many of them felt that they lacked the capacity to deliver it.

Patient survey

Among responders to the main survey there was bimodal distribution in digital confidence, with many patients indicating very high confidence, but a substantial number indicating very low confidence – highlighting a group of patients who lack the capability to use digital tools unaided. Furthermore, around a third of patients had not used their general practice’s website. While more than half of the patients were aware of online appointment booking and potential for ordering repeat prescriptions online, awareness of other online services (e.g. accessing test results) was much lower, and use of online services was low apart from online appointment booking. Patients endorsed a range of reasons for not engaging with online services, in particular reporting their preference to speak to someone and also reporting having difficulties registering to use online services. Such difficulties indicate the potential lack of effective DF. Older patients, and patients with lower digital confidence, were less likely to be aware of or to use DF efforts offered by practices, or to have been informed about or helped to use online services. However, it was encouraging to find that those of non-white ethnicity and those for whom English was not their first language were more likely to report experiences consistent with engagement with or being engaged in DF. Being in receipt of repeat prescriptions was also a factor associated with such engagement. Free-text responses indicated a range of barriers when accessing online services. These included personal barriers (e.g. access to resources), concerns about data security, difficulties with technology design, practice-level barriers and difficulties with credentials. Broadly similar findings were found in responders to the mental health boost sample survey, although these patients generally had higher awareness and use of online services, higher awareness and use of DF efforts, while also reporting more barriers to using online services. While there were some associations between the DF efforts reported in practice and patient survey responses, they were not always present, indicating a potential lack of effectiveness.

Responders to the mental health boost sample survey were more likely to be digitally confident, were more likely to have used general practice websites and were more likely to be aware of and use online services than responders to the main survey (though this may in part reflect the fact that responders to the mental health boost sample survey were younger, on average, than responders to the main survey). Responders from the mental health boost sample were also generally more likely to report that they were aware of, and made use of, DF undertaken by their practice. However, these associations (except for awareness of facilitation) did not persist once adjustments were made for the demographic composition of responders indicating that differences between responders to the main survey and the mental health boost sample survey were largely attributable to factors other than their mental health status.

General Practice Patient Survey

Of patients who recently tried to make an appointment, just under half were aware of online appointment booking and ordering repeat prescriptions. Considerably fewer were aware of the potential to access medical records online, which was consistent with the findings of our patient survey. Between 13% and 32% of these patients reported use of these online services, with around a quarter reporting the use of online consultations. Older patients, those from deprived areas and those from non-white ethnic groups were generally less likely to be aware of and use all online services. In contrast, those with long-term conditions were more likely to be aware of or to use all online services. While differences between patient groups were substantial, the practice with which a patient was registered was, by far, the strongest predictor of patient awareness and use of online services. Some of this between-practice variation may be attributed to the ease of use of the practice website, indicating the potential for DF.

Deviations from the original proposal

Our original proposal stated that we would aim for 60% of general practice responding to the practice survey. In reality, only 31% did so. This deficit can largely be attributed to the COVID pandemic which placed demand on practice staff, reducing the capacity to take part. Furthermore, we had planned for CRN staff to make phone calls to non-responding practices to encourage participation. However, at the time this research was undertaken, CRN staff were prioritising COVID-related studies and some CRN staff were redeployed to other duties resulting in relatively few practices actually being contacted by CRNs. In light of the smaller than anticipated number of practices participating in the practice survey, with the agreement of NIHR, we reduced the target number of practices participating in the patient survey from 120 to 60. In tandem the target number of patient surveys mailed from each practice was increased from an average of 100 to 200. To ensure representativeness, the number of patients mailed a survey was higher in practices serving deprived populations than those serving less deprived populations. Despite these changes, we were unable to recruit 60 practices to undertake the patient survey from within those practices who had responded to the practice survey. Consequently, we invited practices from outside of our original practice survey sample in order to achieve our target. When doing so we focused on practices in deprived areas to ensure representativeness overall. To facilitate comparisons between patient and practice responses, these additional practices completed a practice survey questionnaire, but were not included in the main practice survey data analysis (as they were not part of the original random sample).

A difference occurred with the analysis of data from the GPPS. One of the questions we had intended to analyse was removed from the questionnaire in 2021. So instead of using the most recent data, analysis of responses in relation to awareness of online services related to data from 2019 to 2020.

Strengths and limitations

There are a number of strengths to this quantitative work. First, we achieved a broadly similar number of responses from patients across all deprivation quintiles by over-sampling patients in deprived areas. Second, we adopted a multimodal approach, collecting data at the practice and patient levels and using national GPPS data, and combining data from the different sources. Third, we refined the survey procedures by piloting the survey and obtained extensive PPIE input. Finally, this work was undertaken as part of a large-scale research project commenced at the start of the COVID pandemic which presented multiple challenges. However, our agile approach allowed us to deliver the WP despite the challenges faced.

There were also some limitations. We had a low response rate from the patient and practice surveys. The response rate could have been an artefact of our efforts to target patients in more deprived areas where the response rate is typically lower. However, our patient survey response rate (24%) is comparable to other primary care-based surveys169–172 and should be considered credible considering the challenges presented by the COVID pandemic.

Furthermore, we did not have information on the socioeconomic status (e.g. deprivation) of patients or the rurality classification of where they lived. This was largely due to not having access to patient addresses which would allow linkage to data sets containing this information.

COVID pandemic

The study was designed prior to the COVID pandemic. COVID led to significant changes in the provision of primary care with an increase in remote consultations and use of digital services. It was necessary to adjust our approach in light of both the changes in primary care due to COVID and national COVID guidance. Fieldwork was conducted between September 2021 and July 2022. During this time, on 8 December 2021, England moved to ‘Plan B’ for Omicron which affected the fieldwork and recruitment, and the ‘living with COVID’ guidance was published on 24 February 2022. 173 The case studies focused on DF reported at the time of the research and within the context of the COVID pandemic, the research explored how general practice made changes to service delivery in light of COVID.

Rationale for the methodological approach

The study required an approach which would enable the research team to explore in-depth how DF occurred within general practice. As noted by Reeves et al. 174 ‘the central aim of ethnography is to provide rich, holistic insights into people’s views and actions, as well as the nature (that is, sights, sounds) of the location they inhabit, through the collection of detailed observations and interviews’. A focused ethnographic approach was selected. Unlike traditional ethnographic approaches, focused ethnography enables shorter and more focused periods of ethnography and focus on a predetermined topic. The approach had also been successfully used by researchers on this project (HA) in previous work. 175 Interviews were used to explore the views of stakeholders on DF and complement the general practice-based research.

Qualitative exploration team

Case study recruitment

Sampling of general practices

Our aim was to include a varied sample of eight practices that would provide findings that are relevant across a range of primary care settings. Sampling of general practices focused on seeking variation according to two factors.

Experience of and/or delivery of digital facilitation using different approaches

As outlined in the protocol we aimed for variation according to the types and level of DF in use within practices. Individual practices were identified via the practice survey responses (see Practice survey) and where needed by obtaining information from practices directly about the types of DF in use.

Practice characteristics considered to inform sampling (illustrated in Figure 14)

FIGURE 14.

Illustration of sampling considerations for the selection of case study sites.

We selected practices for variation according to range of characteristics:

1. Practice setting (rural/semi-rural/urban) appears to be systematically associated with variation in access to general practice, for example, older people in rural areas have poorer access to GP services176 and face digital exclusion due to poor availability of broadband internet. 177

2. A range of IMD scores at practice level, as socioeconomic status appears to mediate health disparities via reduced health literacy. 167

3. Number of patients aged ≥ 65 years in the general practice, as although levels of use are increasing, older adults (≥ 75) are less likely to use the internet, and the 16–34 age group are the highest users of the internet.

4. List size of the practice [from a range across small (< 6000) to large (> 12,000) registered patients]. 167

Stakeholder interview study

Figure 15 indicates the design of the stakeholder interview approach.

FIGURE 15.

Design of stakeholder interview approach.

The experience of patients living with mental health conditions

The project was extended to include an additional focus on patients with an experience of a mental health condition. The aim was to explore within case study practices how patients living with mental health conditions experience DF.

Practice staff

The majority of staff participants were female. We interviewed 36 staff from a range of roles and age groups (Table 15).

Interview lengths ranged from 8 minutes to 1 hour. Interviews were conducted in person, via video conferencing or telephone.

Patients

Patients had a variety of self-reported experiences of using GP online services ranging from not at all to being frequent users. They had varied experiences with DF. We interviewed 33 patients across 8 practices, 7 were carers (Table 16).

Interview lengths ranged from 14 minutes to 50 minutes. We achieved a maximally variable sample in relation to patient population demographics.

Digital facilitation focused solely on individual digital platforms

Patient and carer facing digital facilitation as the responsibility of administrative staff

Digital facilitation via passive ‘paperwork’

Digital facilitation via workarounds

Family/social network led digital facilitation

‘Tablet computer in the drawer’ digital facilitation

Scene-setting themes

We identified three ‘scene-setting themes’, as being integral to our understanding of the potential benefits and challenges of DF. These were ‘COVID’, ‘the value and purpose of digital services in general practice’ and ‘conflation of access with digital routes to an appointment or contact’.

Digital facilitation themes

We identified three themes relating directly to DF. These were comprised of subthemes: ‘the operationalisation of digital facilitation – making digital facilitation happen’, ‘usability of digital services in practice and the consequent need for digital facilitation’ and ‘assumptions and stereotypes’.

The operationalisation of digital facilitation – making digital facilitation happen

Digital facilitation as a concept

For the purposes of this research study, we developed and applied a definition for DF: The range of processes, procedures and personnel which seeks to support NHS patients (or their carers) in their uptake and use of online services.

When explained to staff, patient and stakeholder participants, DF is perceived to be conceptually valuable and regarded in the positive. However, it is not used or understood as a term in practice:

I, I don’t understand what that means, I don’t think other people would, not everyone … I wouldn’t understand what that means? … I don’t want to pigeonhole anyone, but if you said it to my mum, she would have no idea what you’re talking about … Yeah, I don’t, I just don’t, yeah, like, the, like, patient care and digital facilitation don’t sound like they go together, do they

Stakeholder U

It is regarded as difficult to have a working definition or usable term for something (DF) that is movable and relative to a person’s circumstance. We observed that it was difficult for participants to ‘grasp’ it as a concept, being mixed up with the digital services themselves.

Sometimes exemplars are used by stakeholders and general practice staff to illustrate the concept of DF, and these link to specific digital platforms (e.g. using iPads to support registration with the NHS App) or organisations (Barclays Bank and their Eagles programme to support people in accessing the internet). Stakeholders felt this was helpful in communicating examples of DF to the public:

And Barclays have done some great work on this with their Digital Eagles … work I don’t know if you’ve come across. But it’s definitely worth looking at. So during the pandemic Barclays Bank, they’ve got something called Digital Eagles. And we did get them doing some work with patients. And they would, like, get them, sort of, sorted out with iPads and … this is obviously when loads of people had nothing to do during the pandemic.

Stakeholder E

While the different participant groups (stakeholder, staff and patients) varied in their understanding and experience of DF, they were united in understanding it as a way to achieve the perceived end point; that being the successful use of digital services to access health care. It is not seen as anything more complex than this. Stakeholders felt that DF would improve access, and subsequently improve health care:

I suppose, so, you know, if you’ve gotta, gotta wait on a telephone for, for 25 minutes on a morning then actually we need to redirect them to the website and they need to be using online services, you know, that will then will free up the time of the wait for those who don’t want to use that system and need to speak to somebody.

Stakeholder B

The interpretation of DF as a means to achieve successful use of digital services to access health care assumes that digital services are desirable and positive. In this example the staff member refers to patients being asked to have belief in ‘eConsults’:

Yeah I’ve told them that they can use it yeah I have told them that they can use it and, and, and the benefits of using eConsults, they don’t always believe me.

Practice E, Lead Practice Nurse, Female

Engaging with the concept of DF requires some cognitive dissonance between what it is perceived that digital services can deliver (the positive elements) and the challenges and reality of how they work in day-to-day general practice.

Responsibility for delivery of digital facilitation

There was lack of clarity on the responsibility for delivering DF. There is a ‘bystander effect’ with patients, different staff groups and stakeholders assuming responsibility for DF to lie elsewhere.

Some participating stakeholders and also some patients expressed that the responsibility for DF should lie with the general practice, including reception staff, administrators and clinical staff:

for me I think it should be the GP practice’s responsibility to show their patients how to use their service ‘cause it is in their best interests. The more people they get using the automated services theoretically, the less, again, through on the phone every day.

Stakeholder C

General practice staff believe that patients bear some responsibility for being able to access their own health care and that support can be sought from friends and families:

a lot of people will get help from family members.

Practice C, GP Partner, Male

Within the general practice team, clinical staff were observed to pass responsibility for delivering DF onto reception staff and administrators:

… Online medication request system as well that patients can use, I’m not entirely familiar with but often I have perhaps direct patients to our admin staff or our receptionists if they needed to get set up with that. […] I guess use of online services that would be more for admin or reception staff.

Practice H, GP (salaried), Female

Patients stated that the responsibility for providing DF lay with the wider health system and regional or government level organisations as well as being a societal issue (described in ‘value and purpose of digital services’ theme). This may relate to how DF was viewed by patients, as being a means to use digital services so that they can achieve access:

It’s difficult, isn’t it? I, I don’t see the role of the practice to be educating us in, in how to adopt technology. I think it’s just too much. I think the remit of the NHS is so wide.

Practice D, Patient, Male, aged 45–54

There was an additional view from some patients that they themselves had some responsibility for trying to use digital services if they wanted to achieve access:

I think it’s like going to school again, isn’t it? You’ve got to practice and do it and learn it and, and remember. And, yes, yes, absolutely, you’ve gotta be a bit disciplined and do it.

Practice A, Patient, Female, aged 75–84

See Figure 16 for a visual representation of how different groups view responsibility for delivering DF.

FIGURE 16.

Perception of responsibility for delivering DF in different groups.

What is consistent across staff, patients and stakeholders is that, in practical terms, the responsibility for any facilitation in relation to digital access currently falls to reception staff.

Logistics of digital facilitation

There are day-to-day logistics underpinning the delivery of DF. Regardless of how DF is delivered and at what level there was awareness amongst participants that well-developed approaches to DF require organisation, planning, time, financial input and training resources:

Then it has to come from the NHS and government as well at the same time. Well, what provision have you put into place? Or what funding have you put into place that we can give time to patients say, this is how to do it.

Practice C, GP Partner, Male

At present, DF is largely occurring in an ad hoc fashion, without external support and with a lack of training or systematic approaches. As a result, organising DF is beyond the capability of most general practices.

Summary of key findings

We identified different types of facilitation occurring in practices, and these were often reactive and in response to individual patient need. Context was important, with the value and purpose of digital services not always well understood by either patients or staff, and with access to an appointment often conflated with digital routes to an appointment or service. COVID had influenced how services were delivered with an increase in digital platforms used by practices.

Digital facilitation was seen as valuable but was poorly conceptualised within general practice. It was unclear where the responsibility for DF lay and there was a tendency to assume it was the responsibility of someone else. Well-developed DF requires support, resources and training and these were not available.

The usability of digital services influenced whether DF was needed. Where multiple digital platforms were used, this required support for patients to understand which to use and how to use them. General practices often made changes without informing patients and patients did not know what would be available to them.

We observed assumptions being made by both patients and practice staff about who could use digital services and who could not, including stereotypes about age that were applied to both staff and patients. When it came to who needed DF, individual circumstances mattered, with needs being movable in line with personal circumstances. Vulnerable populations were particularly in need but were also least likely to be considered when services are designed.

Deviations from the original proposal

We expanded from three to four geographical areas.

This allowed us to recruit a practice in an area of high deprivation and with a varied population profile relative to our other practices. This in turn ensured our sample was maximally variable.

We sampled practices according to their experience of and/or delivery of DF using different approaches. We had anticipated at the design stage of the study that we would be able to identify discrete approaches to DF, based on what we knew to be happening in general practice, and from the literature review. By the time we were recruiting general practices, the COVID pandemic had changed how GP services were delivered. To counter this, we ensured that we were fully appraised of DF activity at the point of recruitment rather than relying on the practice survey data or prior reports. We were alert to the possibility of changes due to the COVID pandemic while we conducted fieldwork.

Strengths and limitations

We were able to recruit a varied sample of general practices during the COVID pandemic and conduct ethnographic fieldwork as intended over the period from September 2021 to July 2022. During this time there were various levels of lockdown and restrictions which the team successfully negotiated to ensure the fieldwork was completed. We were conducting the fieldwork at a key time for general practice, with the COVID pandemic leading to an increased use of digital services. This allowed us to capture how practices responded to an increased need for DF in real time, and our sample included populations with high numbers of patients in ethnic minority groups, and in areas of high deprivation. We recruited a broad range of 19 key stakeholders at a time when those in professional roles were under extreme pressure.

The COVID pandemic has impacted the ways in which general practices were operating and in their relationships with their patients. There was little patient footfall in the practices’ premises, and this changed the nature of the non-participant observation, which was focused on staff. Because fewer patients were attending the practice in person, the main recruitment tool to access patients was a letter sent via the practices. In some practices, initial responses were very limited which resulted in second mail outs being necessary. However, we did recruit and interview a wide range of patients with varied characteristics.

Recognition of the need for and subsequent funding of the additional interviews was a strength in ensuring inclusion of this under researched group. However, in line with the broader challenges associated with conducting research in this area, recruitment of practices and patients for the study of patients with mental health conditions was challenging. We aimed to recruit the case study practices to this additional interview study but two declined for capacity reasons and a further declined due to sensitivities about the patient group. In the practices that were recruited, workforce capacity was an issue which delayed mail outs.

We were able to recruit patients in groups that are vulnerable to digital inequalities, which included individuals from older age groups, those in minority ethnic groups, those in poorer health, and individuals in rural settings. Several participants intersected these characteristics. We were not successful in recruiting patients who were in marginalised groups. This reflected the additional and substantial challenges in recruiting general practices and patients during the pandemic.

The response rate to the invitations to take part in an interview was lower amongst individuals living with a mental health condition than for the main study; and the majority of participants in this group were female and aged in their 40s. This may reflect the timing, with some invites sent over the summer period. However, it may also be due, in part, to the stigma associated with having a mental health condition, or a lack of recognition amongst some patients of their diagnosis. Even some participants that took part in interviews did not consider themselves to be ‘diagnosed’ with a condition but instead felt they had ‘a touch of anxiety’, or ‘a bit of low mood’. This suggests difficulties with the labelling of a mental health diagnosis for some patients. The approach may not have been appropriate for this cohort of patients and another strategy may have been more successful. The findings should be viewed in this light.

In conclusion, the qualitative exploration allowed an in-depth analysis of how DF is realised in general practices in England. It highlights areas that could be targeted in developing DF as a concept and in applying it to the delivery of care to patients.

Scene-setting themes (Table 18)

1. Value and purpose of digital services. The usefulness of DF depends on the value of the digital services whose use might be facilitated.

2. Digital is conflated with other routes to accessing care. Patients desire access to care, and digital is only one type of route to accessing NHS primary care services, along with face to face and telephone.

3. COVID. The onset of the pandemic, and of the response of primary care to the pandemic, was associated with a shift away from patients attending GP premises and towards remote transactions. Combined, these had a major impact on the need for, and provision of, DF.

Digital facilitation themes (Table 19)

1. Defining and identifying DF can be difficult. There is lack of clarity among some practice staff and patients about what comprises DF.

2. Types of DF. It can be provided actively (e.g. practice staff recommending to patients that they use an online service and telling them how to do so) or passively (e.g. a poster in the waiting room, a recorded telephone message played to callers to the practice number); and DF can be reactive (helping patients when they ask for assistance) or proactive (without waiting to be asked).

3. Supporting initial sign-up or sustained use over time. When an online service is being introduced or rolled out, facilitation may focus on supporting patients to register for a service. But once they are registered, they may need support (e.g. problem solving) to continue using the service.

4. Who delivers DF. There is no clear consensus over whose responsibility it is and who should be expected to fund the resources needed to provide DF.

5. Enablers of DF, including the usability, or ‘user-friendliness’ of digital platforms.

6. Barriers to DF, including staff attitudes towards digital services and stereotyped assumptions about the capabilities of some patients.

7. Digital access and DF for different subgroups in the population. There is evidence in the literature that some groups are relatively disadvantaged when using online services and they might benefit from targeted DF efforts to support them.

8. Effectiveness. Is DF effective at supporting patients to access and use online services; and which types of facilitation are more effective than others.

Programme theory

Implementation theory

Digital facilitation as an intervention

Our findings have highlighted the amorphous nature of DF and indicate that DF is part of the complex processes involved in delivering access to care for patients in general practice. This has implications for the future evaluation of DF. DF is flexible and adaptable, functioning as a complex intervention. The end point, achieving access, is the important factor for all parties. Therefore, DF may be defined as the support required and provided to enable patients to achieve access digitally. DF is one or more of numerous possible, mutually non-exclusive interventions.

Where an intervention is used, consideration should be made for the understanding of the intervention by all involved parties with our evidence indicating that DF is not a concept that is easily grasped and may be better linked to how patients access and receive health services rather than being a standalone intervention. There should be clear rationale for introducing it.

Responsibility for digital facilitation

We observed that understanding about who delivers DF was key to whether it happened at all. Any future evaluation of effort at DF as an intervention should consider who is responsible for it. This would include:

• Who has overall responsibility for the success or failure of DF?

• What are the responsibilities of general practices and their staff with regard to ensuring safe and appropriate delivery of DF?

• Which staff groups are involved in delivering it? What does this involve and how does it fit with their current role?

• What are patient responsibilities when engaging with DF?

• If DF is delivered by an external party, how will they interface with the practice staff and patients and what responsibilities does this bring?

• How does DF fit into the wider health community beyond general practice?

• Will there be support from regional or national or other external organisations? This might include commercial suppliers of digital platforms, and charities to boost digital literacy in certain population groups.

Patient groups and potential for inequalities

A key element of this study has been to explore how different patient groups engage with and are affected by DF. This has had a particular focus on the potential for inequalities. Future evaluations should consider:

• where DF is most needed and most likely to be successful (there is little point in focusing DF in primary care on people without internet access, for example)

• differences in incidence of benefits and costs/savings to patients, for example, burden on patients and likelihood that it will lead to them being able to achieve access to health care in the way intended

• weight of importance for patients, which may be low relative to importance for practices and other interested parties.

Potential outcome measures for future evaluation

Potential outcome measures for use in future studies should aim to capture quantifiable impacts where possible, and also to explore qualitative aspects of DF. They should consider both intended (e.g. increase in levels of use of online services) and unintended consequences (e.g. staff finding themselves expected by some patients to act as a help desk for IT problems). Here we outline potential outcomes, but these are not exhaustive. The outcomes can be collected for subpopulations within the overall population so as to permit evaluation of inequalities, or their mitigation, between different groups within the population.

Different types of future evaluation including areas of focus

Given the difficulties in defining DF as an intervention and the many involved parties, measuring effectiveness would need to take a holistic approach. This fits with the latest MRC guidance on complex interventions which suggests that evaluation goes beyond whether an intervention works.

Our exploratory research has identified areas for focus that were previously unconsidered:

• the role of general practice administrative staff in supporting digital services and the appropriate ways to support this

• understanding how different approaches to DF work for population subgroups

• supporting sustained patient use of digital services, moving the focus beyond registration

• understanding the factors that need to be addressed to achieve a certain baseline of digital services use, and how service use can then be developed to support specific patient groups with particular needs

• understanding the extent of patient demand for digital services to access and use general practice and thus the level of DF needed to achieve this.

This is not an exhaustive list but highlights the complexity of this area and some appropriate areas of focus to ensure benefit from future research.

Deviations from the original proposal

There were no deviations in methods for WP4 to report from the original proposal.

Future research on digital facilitation

We found only limited associations between the DF efforts described by practices and either the patient awareness and use of online services in those practices or the perception of DF efforts by patients. This is not surprising given the observations from the ethnographic case studies that facilitation efforts were often reactive rather than proactive, and that efforts to perform effective DF were hampered by lack of resources. Future research might therefore further explore the association between awareness by (potential) users of digital primary care and actual use of those same services, especially within the context of the introduction of specific efforts to support the awareness and use of such services.

In Evaluation framework we have laid out a framework for a future evaluation of DF in primary care, and while we set out some of the key facets that such an intervention might encompass, we do not specify exactly what form such an intervention might take. Our intention with the Di-Facto project was to explore DF that was already underway, with the potential of identifying good practice which could be either developed further as an intervention for testing or shared more widely for immediate implementation. However, we did not find an example of what might be considered ‘text book’ DF, and certainly nothing that would form a complete, practical intervention package. Any intervention development could build on our research, but given the findings around the need for digital technologies to be a ‘good fit’, it would be essential to co-develop any intervention both with patients, and with general practice staff who would be delivering it to ensure a responsive and adaptive approach.

Related to the development of DF interventions is the development of patient facing digital tools. Currently, these tools are generally developed by commercial providers, but research into what facets of digital platforms make them easier to use and understand by patients is warranted. The context in which these digital tools are presented to patients, usually a general practice website, also warrants consideration. A study in Scotland showed that 77% of practice websites featured information that exceeded the recommended 9–14 reading age for online content, while 81% scored below the recommended Flesch Reading Ease score for ‘plain English’. 204 Research into optimising the presentation of practice websites for patient engagement may lead to increased usage of digital tools and reduce the need for DF.

While it is clear that DF is likely to need to be tailored to different patient groups, there is currently a lack of evidence about what this should look like. Future research is needed to address this gap along with identifying which patient groups are most in need of DF. DF and the use of digital services have the potential to both introduce new and extend existing health inequalities, while addressing others (e.g. in groups who experience stigma). 205 Any future research on DF also needs to ensure that associated digital exclusion of certain groups, including, but not limited to, older patients, is investigated. 206 In addition to the observation that older patients were much less likely to be digitally confident, we have also found that people with long-term health problems, whose first language is not English, who are retired, or not in work/education had lower digital confidence. Care needs to be taken to remember that these findings represent an average picture, and any targeted support should not automatically stereotype patients into being digitally able or not. We found limited evidence that people living with mental health problems were particularly in need of DF, though they had varying views on the use of digital services. The newly developed framework for digital health equity may help to minimise any negative impacts on certain patient groups. 207

Finally, we would recommend that any research into DF monitors sustained use of online services, rather than the current focus on initial sign-up. Such sustained use will be required if the perceived benefits of digital service use are to be realised.

Research conduct

Taken at face value and in isolation, the findings from the practice survey might have implied that proactive, effective, DF was being carried out at scale across the country. However, comparisons of the practice survey responses with patient responses, and with the ethnographic observations, suggested a rather different picture. This has major implications for the isolated use of organisational-level surveys to examine how services are being delivered. Only by using a mixed-methods approach and synthesising the evidence from different research methods as we have done here, can a reliable and full picture be gained.

In studies of health service delivery as it is happening in everyday practice, it is important to examine a wide range of service providers to obtain findings that will be generalisable to the wider population. In conducting this study, we made efforts to include practices that were not ‘research active’ in order to make our results as generalisable as possible. However, the NIHR CRN, which was set up to support the delivery of NIHR funded research, has a strong focus on ‘research active’ practices. We found that CRNs were not set up to approach non-research active practices, and along with capacity issues, this posed a challenge to the delivery of the research. We would recommend a widening of the CRN remit to encourage contact with all general practices, especially for the conduct of health services research.

Participant representation

Several steps have been taken in this research to ensure that the participants were representative of the wider population. Given the scope of this research, the population of interest is all users of GP services in England. The areas of the country where our research was conducted were selected to cover a diverse population, including a mix of city, urban and rural areas, a range of deprivation levels, and areas with high numbers of ethnic minorities.

Within WP2 we used a random sample of practices across our research area to ensure our findings were generalisable. Despite efforts to engage with all practices, the eventual practice survey respondent sample was somewhat less likely to include practices serving more deprived populations but was reasonably representative in terms of the age profile, and proportion from ethnic minorities, of the registered population.

The responders to the WP2B patient survey were skewed towards those of older age (over 65). However, this is reflective of the differential use of primary care services by older patients, and may denote improved representativeness of the sample relative to the people that exert most demand for primary care services. 208 Only 8% of responders described themselves as having a non-white ethnicity, which is below the national average (18%). This may in part reflect the age profile of responders but is still less than ideal. The respondent209 sample included participants with a wide range of working statuses as well good representation of carers and parents. There was also good representation of people with long-term physical and mental health conditions, with the latter group particularly focused on in the mental health boost sample. There were 358 patients who were deaf or had a hearing impediment, but only a small number describing themselves as blind, or partially sighted, a potentially important group for this study. Crucially, the patient survey was accessible in both paper and online formats to enable patients unable to engage with digital platforms to participate. We worked with our PAG to ensure the survey materials were accessible to as many potential responders as possible, although unfortunately resources did not allow for translations or large print versions. The steps taken to target more deprived populations in the patient survey resulted in an even spread of responders across the range of practice deprivation scores.

The analysis of GPPS data included data from across the country to maximise generalisability. Furthermore, due to the very large sample size of the GPPS survey, it was possible to examine awareness and use of online services in relatively small groups, something that would not have been possible with primary data collection. The GPPS is offered for patients to complete online, or over the phone, in 14 languages other than English as well as British Sign Language. However, given the paper version of the GPPS questionnaire is only available in English, and the strong relationship between response mode (paper/online) and digital confidence in our patient survey, it is likely that responders to the GPPS may be biased towards more digitally confident patients particularly among ethnic minorities.

Within WP3 the participating practices were purposefully sampled to include those serving populations with a range of deprivation levels, and varying proportions of both ethnic minority and older (aged over 65) patients. Practice staff participating in interviews covered a range of ages, and more than half were female, which probably reflects the gender balance in the staff roles included. Similarly, patient interview participants covered a range of ages, and again more than half were female. Around a third of patient participants described themselves as having a non-white ethnicity, all but one of whom described themselves as Asian.

The research team and wider involvement

There was no formal monitoring of the characteristics of the research team, as this was deemed inappropriate on the basis of potential disclosure. However, the research team included researchers and co-applicants across a range of ages, with a mix of genders and ethnicities. By the nature of being employed, none of the research team was over the age of 65 (a key population in this research). There was an even split of genders across the co-investigator team. The remaining research team was predominantly female. The team included researchers with a very wide range of experience and expertise, from undergraduate student interns to newly qualified post-doctoral researchers, more senior researchers, and the principal investigator with 35 years of experience.

Various development opportunities were provided to more junior members of the research team. Junior team members have presented work from the project at conferences and will continue to do so over the months following the end of the project as we continue to disseminate the findings. Furthermore, junior members of the team are taking leadership positions in many of the planned publications arising from this work. Informal training has been provided to those team members undertaking statistical analysis, and formal training was provided for conducting ethnographic fieldwork.

As with the research team, we did not formally monitor diversity within the PAG. However, its membership was predominantly white and had members aged over 50. While this is broadly representative of service users in one of our research areas (Southwest England), more could potentially have been done to make the PAG composition more representative of the broader areas covered. We took care to expand this group to inform the work focused on patients living with mental health, reflecting the importance of including relevant patients in the advice received.

Targeted searches of websites

All searches limited to English, 2015–present.

Search of HMIC database

We searched the HMIC database using the following terms:

Search of academic literature on DF in non-healthcare sectors

We searched the literature on non-healthcare sectors via Google Scholar using the following search terms:

Factor analysis of digital confidence scale

Question two through six of the main patient survey asked patients about their confidence levels in performing various digital task commonly performed in everyday life (see additional files www.fundingawards.nihr.ac.uk/award/NIHR128268; accessed 3 April 2024). Each question had three response options (‘Very confident’, ‘Quite confident’, and ‘Not confident’). Polychoric correlation coefficients were calculated for these five categorical variables. To determine the number of factors explaining the variability in responses to these questions, an unrestricted exploratory factor analysis (EFA) was performed. The eigenvalues from the polychoric correlation matrix were calculated and plotted against each factor. Both Kaiser rule and the Scree plot were used to determine the number of factors. Kaiser rules suggest retaining factors whose eigenvalues are > 1. The scree plot of eigenvalues was used to visually determine the number of factors to retain in the model based on the point of levelling off (the ‘elbow’).

Internal consistency reliability coefficients (Cronbach’s alpha) were calculated from the variance–covariance matrix of digital confidence. The range of Cronbach’s alpha coefficients in each factor when questions were sequentially omitted and reintroduced was also calculated. Polychoric correlation coefficients matrix was computed using the ‘psych’ package, and Cronbach’s alphas were computed using ‘MASS’ and ‘ltm’ packages in R.

Results

There were 1.7% missing values, all of which were among patients who submitted their response by post. The polychoric correlation coefficients presented in Table 30 indicate very strong correlations between each pair of variables. Only the first factor had an eigenvalue > 1, suggesting there is a single factor underlying the responses. The scree plot in Figure 17 for an unrestricted EFA confirms a single underlying factor. The value of Cronbach’s alpha was 0.936 indicating very high internal consistency and possible potential for item-redundancy. Cronbach’s alpha ranged from 0.916 to 0.933 when one question was left out at a time, and it ranged from 0.879 to 0.902 when two questions were left out at a time (Table 31). A restricted EFA model with a single factor resulted in factor loadings > 0.8 for the five questions (Table 32).

FIGURE 17.

Scree plot for the EFA applied to the responses to the five questions addressing digital confidence.