The clinical effectiveness and cost-effectiveness of telephone triage for managing same-day consultation requests in general practice: a cluster randomised controlled trial comparing general practitioner-led and nurse-led management systems with usual care (the ESTEEM trial)

Triaging clinician

Three alternative staffing models emerged regarding which clinician (GP or nurse) delivered the triage: (1) one clinician (GP or nurse) at a time conducted a triage session (on a rotating basis); (2) a number of clinicians (GPs or nurses) conducted triage sessions on the day (on a rotating basis); or (3) in GPT practices only, all of the GPs conducted triage each day, with GPs triaging their own patients to triage-bookable appointments.

Triage availability

Practices were asked to triage all consecutive eligible patients during opening hours to ensure the least disruption and shortest time of trial data collection possible. In the event that limited staff resources prevented triage of all eligible patients each day, practices were permitted to operate triage within specific agreed sessions (‘research window’), amounting to no less than 50% of the week, encompassing all five working days and both mornings and afternoons. These arrangements were agreed with the research team in advance. Practices were advised of the importance of all eligible patients being triaged during an entire session, as opposed to receptionists booking in a set number of triage patients within a session and then ‘closing’ the triage once the allocated slots were taken. Practices operating triage within ‘research windows’ had their data collection period extended until the patient recruitment target was reached.

Triaging babies

Practices were advised that babies (i.e. ≤ 2 years of age) were eligible for trial entry according to the inclusion and exclusion criteria, with the parent or guardian identified as the individual who participated in the triage call. As some practices routinely see babies as soon as possible on the same day of the appointment request, it was expected that some practices would opt to book these patients in for a face-to-face appointment rather than add them to a triage list. Although all of the intervention practices were encouraged to offer triage to babies during intervention training, some practices elected to offer face-to-face appointments in line with their existing practice policy. When this happened, the babies were still included in the trial analysis on an intention-to-treat (ITT) basis.

Requests for same-day, face-to-face nurse practitioner consultations

Some practices operated a nurse practitioner-led clinic for acute or minor illnesses, for which the nurse typically managed same-day consultation requests. This request was deemed equivalent to a same-day GP appointment request, as the patient would see a GP if the nurse practitioner was unavailable. Such patients were deemed eligible for trial entry and managed as per the treatment arm allocation.

Role of commercial providers in triage training

To prepare GPT and NT practices for incorporating telephone triage within their appointment systems, we worked with Productive Primary Care Ltd, a Leicestershire-based, NHS-focused commercial organisation that works nationally across the UK with NHS organisations. Plain Healthcare’s ‘Odyssey PatientAssess’ (derived for general practice use from ‘Odyssey TeleAssess’ or ‘TAS’) CDSS was used to support the nurses in NT practices. Plain Healthcare, then part of the Avia Health Informatics PLC group, supplies the NHS with CDSS systems (e.g. out-of-hours primary care services). Plain Healthcare liaised directly with NT practices to discuss IT and training requirements for the installation of the software. Following a ‘site initiation call’ between Plain Healthcare and each practice, installation and testing took place over a 6-week period. The CDSS was fully embedded within the Egton Medical Information System (EMIS, Yeadon, Leeds) ‘PCS’ and ‘LV’, and installed as a separate ‘stand-alone’ window tab to the clinical notes within all other types of practice IT system [e.g. Microtest Evolution (Microtest, Bodmin, Cornwall), Vision (In Practice Systems Ltd, Battersea, London), SystmOne (The Phoenix Partnership, Horsforth)].

Nurses received CDSS training and training in telephone consultation skills. Following this, there was a pretrial period of 1 month during which nurses practised using CDSS in simulated patient scenarios during their daily work. Towards the end of this period, the nurses’ use of the system was assessed by Plain Healthcare trainers, with all nurses needing to demonstrate proficiency in its use before being recruited into the trial. It is also important to note that the CDSS is designed to support the nurses’ clinical decision-making; there was no requirement in the trial that the recommended advice the CDSS generated had to be followed.

Overview of practice training

Practices were trained in research procedures and the delivery of triage interventions, following a model (Figure 1) devised during the pilot study. Training was delivered in six stages, and by a combination of Productive Primary Care (GPT and NT only), Plain Healthcare (NT only) and trial researchers (all arms). ‘Triage training’ workshops were organised for GPT and NT practices (training components 3–5, described below) and delivered as a package to staff from a number of practices at a central location in each of the recruitment areas. If practices were unable to attend a central training event, the various components were delivered on site at the practice.

FIGURE 1.

Practice training structure.

1. Inception meeting An initial whole-practice meeting with the trial researcher was held to outline the trial, staff group involvement and timeline of training. The meeting was convened for the practice manager, at least three receptionists, at least two GPs – to include the two GPs identified as clinical leads for the trial – and up to four nurses (NT only).

2. Appointment request audit To assess patient demand for same-day and prebookable appointments, a 1-week audit was conducted by the practice reception team. Capacity information was collated relating to the number of GP and nurse sessions provided per week and the number and typical duration of appointments offered within each session (including the proportion reserved for same-day and prebookable use). The trial researcher provided the reception team with Audit Log Sheets (see Appendix 3) and asked them to record all incoming calls each day for a week (Monday to Friday), indicating whether each call was a same-day or prebookable appointment request. The researcher maintained contact with the practice during the first day or two of the audit to ensure that the procedure was followed appropriately.

   1. Organisational training and guidance in organising the triage system Productive Primary Care trained the practice manager, at least two receptionists, at least two GPs (GPT) and up to four nurses and one GP (NT). The training incorporated advice for the reception teams on how to communicate the introduction of the triage system to patients. Practical guidance on setting up the appointment system to incorporate triage consultation slots and face-to-face appointment slots for the use of the triaging clinician was also given. Using the data collected during the appointment request audit, Productive Primary Care used a spreadsheet (the Appointment Redesign Tool) to estimate the number of appointments that the practice needed to provide and the proportion that should be ‘triage bookable’ based on their previous experience of working with a range of practices. The details of the calculations for these estimates are the intellectual property of Productive Primary Care.

   2. GP triage skills In GPT practices, Productive Primary Care delivered training to GPs on how to communicate the operation of the new triage system to patients, and general guidance for consulting over the telephone. At least two GPs at each practice were required to receive this training and were expected to disseminate the information to their other GP colleagues.

   3. Professional issues and telephone consultation skills All nurses at each NT practice who would undertake triage were required to receive this training. A learning and development adviser from Plain Healthcare presented information and advice on the professional issues that may arise from nurses consulting over the telephone. Guidance on telephone consulting skills, including role-play scenarios, was provided. In addition, nurses received one-to-one remote training from an advisor while they practised using the CDSS on computerised simulated patient scenarios in the weeks leading up to the trial, culminating in a proficiency assessment.

   4. Research procedures training Trial researchers provided briefing sessions to practice staff in data collection procedures. Staff groups included the reception team, practice manager, key IT and administrative staff and the GPs and nurses to be involved in triage or dealing with patients requesting same-day consultations.

Inclusion criteria

All consecutive patients making telephone requests for a same-day, face-to-face consultation with a GP during the data collection period were potentially eligible. When an individual patient had multiple same-day consultation requests during the recruitment period, only the first contact was included in the study to avoid confusion in following up the contact (although subsequent calls would be triaged in the same way, according to trial arm). All patients aged < 12 years and ≥ 16 years requesting eligible appointments were included with respect to the primary and secondary outcome measures. Parents or guardians of children aged < 12 years provided consent on behalf of the child.

Exclusion criteria

Temporary residents or young people aged 12.0–15.9 years were excluded, as the study involved receipt of a postal questionnaire along with written consent to review case notes. We believed that receipt of the questionnaire at the young person’s address may have inadvertently led to a breach of confidentiality should third parties have access to the young person’s mail. Although excluded from trial participation, both temporary residents and young people received the intervention as per trial allocation. Adults aged ≥ 16 years were included, unless the practice wished to screen out patients for whom they felt it would be inappropriate to send a questionnaire (e.g. patients with recent bereavement, vulnerable adults).

Patients seeking urgent or emergency care on account of the following conditions were excluded:30 too ill to participate (severe chest or abdominal pain or severe difficulty breathing; vomiting blood; altered consciousness; seizures; pregnancy-related problems; or severe psychiatric symptoms); unable to speak English or difficulties with communicating on the telephone (hearing or speech). Reception staff members were trained to refer to a standardised ‘Receptionist flow chart’ to ascertain patient eligibility (see Appendix 4).

Patients ringing on a second occasion were not included in the trial again (i.e. their point of entry into the trial was on the first occasion they rang the practice seeking a same-day consultation). However, in practice, receptionists would have recorded the patient as being eligible and sent their information to the research team. Our database would have flagged them as a duplicate and excluded them from being entered into the trial database again.

Randomisation procedure

Individual patient-level randomisation was deemed impractical as it was unable to reflect the practice-level triage system implementation and was vulnerable to contamination. A cluster RCT design was adopted, with practices that agreed to take part subsequently randomised to one of the three trial intervention arms (GPT, NT, UC). 36 To manage the recruitment process, practices were randomised in three distinct ‘waves’. Wave 1 took place during spring and summer 2011, wave 2 during winter 2011/12 and wave 3 during spring and summer 2012.

The sequence of practice randomisation was computer generated and minimised for geographical location (Devon, Bristol, Norwich, Warwick), practice deprivation [based on the data provided by the Association of Public Health Observatories (APHO)37 at time of randomisation, practices were classed as ‘deprived’ (i.e. score is above the average for England) and ‘non-deprived’ (score is average or below average) and practice list size of ‘small’ (< 3500 patients), ‘medium’ (3500–8000 patients) and ‘large’ (> 8000 patients)]. A stochastic element was included within the minimisation algorithm to maintain concealment. Practice allocation was undertaken using a password-protected web portal (developed and validated by the UK Clinical Research Collaboration (UKCRC)-accredited Peninsula Clinical Trials Unit) by a statistician who was independent from the trial statistical team. Allocation was communicated by e-mail to the trial manager who then informed the local researchers, who, in turn, informed practices of their allocation.

Replacing practices that withdrew post randomisation

To maintain balance between groups, practices that withdrew post randomisation were replaced with a practice from a waiting list in the same geographical area, and of a similar size and deprivation level, where possible. Owing to the limited number of waiting list practices, it was not possible to select a matched practice randomly. Instead, replacement practices were purposively allocated to a trial arm based on locality, practice size and deprivation index. However, other than knowing that the practice was matched for the above characteristics, the researcher making the practice selection was unaware of any other characteristics of that practice. Replacement practices were not aware of their potential allocation when the researcher checked that they were still willing to enter the trial. A questionnaire exploring reasons for withdrawal was sent to practice managers (see Appendix 5).

Patient consent

The patient recruitment and consent procedure is summarised in Figure 3.

FIGURE 3.

Patient recruitment and consent procedure.

Capturing the patient sample

The receptionist flagged a triage or UC appointment as being allocated for a study participant by entering a free-text comment (‘ESTEEM’) and the date of the appointment request as a note on the appointment slot. Within a week, a dedicated member of the practice staff applied a study-specific Read Code to each ‘ESTEEM’-flagged patient, thus defining the eligible patient sample. This Read Code was dated as per the receptionist’s free-text comment, allowing the practice to run electronic searches to generate an ESTEEM patient list that was passed to the research team as the denominator for the study database.

In the triage practices, if for any reason an eligible patient was booked a face-to-face or any other type of appointment instead of triage (e.g. a patient being unable to receive a telephone call while at work), such patients were still captured as part of the eligible sample and followed up according to the study protocol. Similarly, if a patient was not booked into an appointment (of any type), despite having made an eligible request (e.g. a patient failed to complete the telephone call with the receptionist), the patient’s record was still flagged by reception staff as eligible and included within the study denominator.

It was not possible to routinely record eligibility for all patients telephoning the practice. However, on two occasions during run-in and once during data collection (see Integrity checks) receptionists logged all incoming calls and recorded patient eligibility, flagging those patients who were requesting same-day appointments and who were too unwell or could not communicate clearly (see Appendix 9). We used these data to estimate the proportion of patients making same-day requests who were ineligible for the trial. However, receptionists did not complete the log sheets as fully as we would have liked. For example, in some cases a call had been recorded, but there was no record of whether or not it was a same-day request. Additionally, for other patients recorded as requesting a same-day appointment but who were ineligible, the reason why was sometimes omitted.

Confidentiality

All personal information obtained about patients or staff for the purposes of recruitment or data collection (e.g. names, addresses, contact details, personal information) was kept confidential and held in accordance with the Data Protection Act. Each patient included in the trial was assigned a research number, and all data were encrypted and stored without the subject’s name or address. Electronic data were held on a secure database accessible by only members of the study team, and paper-based information was held in a locked filing cabinet in the research team office. Names and participant details were not passed on to any third parties and no named individuals have been included in the reports.

Regarding case note review, the clinical record was viewed only when patients had given verbal or written consent. Patient records were accessed by researchers holding relevant approvals and letters of access granted by PCT research governance units and NHS staff. Case note reviews took place in the practice and no patient records were removed from the premises. Any data extracted were identified by the research number; all other patient identifiers were removed. When data were temporarily stored on laptops, memory sticks or attached to e-mails prior to transfer to the research offices, these media were encrypted as per NHS security requirements. Researchers adhered to confidentiality agreements stipulated by the practices concerned.

Safety of participants

There were not thought to be any significant risks to patients or staff arising from the trial methods. Triage itself may carry risks but the approaches to triage in this trial were already in routine use in the NHS in other settings. Notwithstanding this, we identified two areas of potential minor risk, and developed systems to ensure that these were minimised.

Patient safety

The number of deaths and unplanned ‘emergency’ hospital admissions (and associated number of bed-days) within 7 days, and A&E attendances within 28 days of the index consultation request were collected at case note review. An unplanned emergency hospital admission was defined when there was no evidence in the notes of advance planning, even on the day the admission occurred. The number of ‘planned’ hospital admissions was also collected. An ‘adverse events’ reporting procedure (see Appendix 10) was developed to monitor patient safety. The clinical lead(s) at each practice were contacted weekly by the trial researcher and asked to return a log sheet completed with any events arising during the intervention period, such as trial patient deaths, emergency hospital admissions or A&E attendances, as well as any problems with the triage system itself or patient complaints.

It became evident that the number of deaths recorded in the weekly log sheets of adverse events from practices did not always correspond to that identified at case note review. On discussion with the DMC, practices were asked to perform a search of their electronic records at the end of the trial to produce a report of all deaths occurring during the data collection period. This became the definitive source of data on patient deaths.

Patient management on the index day

A description of the management of patients on the index day of the same-day consultation request was based on patients’ first two primary care contacts collected for our POM.

NHS resource use

The NHS resources that were combined into the POM were also reported separately as secondary outcomes. Non-attendance rates for allocated appointments in the 28 days following the index request were also collected by case note review. It was not possible to capture patients’ contacts with NHS Direct from practice records. Patients’ use of NHS Direct during the 28-day follow-up period was assessed through self-report by postal questionnaire.

We collected actual consultation length for the index triage and UC (face to face or telephone) consultations on the Clinician Form (see Data collection and management). We also estimated the length of subsequent face-to-face consultations, for both ‘same-day’ and ‘28-day follow-up’ periods. The former was based on the recorded start and end times of a small sample of consultations captured over two randomly selected days in each practice during the intervention period. The latter was based on consultation length used in published unit costs reported within the NHS Personal Social Services Research Unit (PSSRU38).

Patient-reported outcomes

All patient-reported outcomes were collected by the postal questionnaire.

Case complexity

To define patient case mix, the clinician conducting the index consultation captured the complexity of the case (after Howie et al. )47 using an eight-point scoring schedule incorporated on the Clinician Form (see Data collection and management). Each consultation was scored as having either substantial (2 points), attributable (1 point) or no content (0 points) with respect to each of four domains: physical, social, psychological or ‘other’ (e.g. administrative) components to the consultation.

A priori analyses

1. A per-protocol analysis including only those patients in the triage arms who received a telephone triage contact by the appropriate clinician type on the index day (all patients in UC were considered to be per protocol).

2. Exclusion of data from two practices in GPT that did not revert to UC as requested following their period of implementing triage and recruiting patients to the trial.

3. Investigation of the effect of missing primary outcome data (due to lack of availability of a case note review) using multiple imputation methods,59 based on the assumption that missing case note review data were missing at random.

4. Analyses to investigate interactions between treatment arm and practice characteristics (deprivation, location and list size) on the POM, using interaction terms within the regression models. A series of models were fitted, each model investigating interactions between treatment arm and a specific covariate. Age (as a proxy for case complexity), gender and ethnicity (dichotomised into ‘White and other ethnic group’, comprising ‘Mixed or multiple ethnic groups’, ‘Asian’ or ‘Asian British’, ‘Black/African/Caribbean/black British’ and ‘Other ethnic group’) were also investigated for potential interaction with the treatment arm. A significance threshold of 0.01 was used for hypothesis tests for interaction terms. However, the trial was not powered to investigate subgroup interactions and results should be interpreted with caution.

Results of inferential analyses were presented with 95% CIs (with the exception of inferential analyses including interaction terms, where global p-values for the individual arm by covariate interaction are also stated).

Post hoc analyses

1. Combination of all within-practice contacts on the index day into one overall contact, with contacts on the subsequent days counted individually, as for the primary outcome [e.g. a GP telephone contact and a GP face-to-face contact within the practice on the index day would be counted as one contact only, whereas a GP telephone contact and an A&E contact (or any other contact outside the practice) on the index day would count as two contacts].

2. Analysis of POM contacts on the index day only, and analysis of POM contacts occurring during the 27 days subsequent to the index day only.

3. Analysis of GP face-to-face contacts only, during the full 28-day follow-up period, on the index day only, and on days subsequent to the index day only.

4. Analysis of GP face-to-face and GP telephone contacts only, during the full 28-day follow-up period, on the index day only, and on days subsequent to the index day only.

Intervention cost

Resource use associated with the set-up and delivery of the triage interventions (GPT, NT) comprised initial training costs (set-up), CDSS and licence costs, and staff time spent on delivery of the triage contact. These resource use areas were identified during the pilot study.

Economic outcome: costs of primary care contacts (plus accident and emergency) over 28 days

Primary economic (cost) analyses are undertaken using data collected on the primary outcome, contacts taking place in primary care over 28 days, collected within trial at participant level using a case note review.

Primary care and related contacts by type of contact, as included in the POM, are detailed above (see Primary outcome measure). Trial data on service use were combined with unit cost data (Table 4) to estimate a mean 28-day cost for primary care service use, for each of the trial arms. Most unit cost data were taken from those reported by Curtis38 (PSSRU unit costs), with other cost data sourced from credible national data sources (see Table 4). Contacts recorded as GP or general unspecified (n = 30) are treated as GP in surgery consultation, for the purpose of cost analyses. Contacts recorded as ‘Nurse contact unspecified’ (n = 15) were treated as ‘Nurse in surgery’ consultation for the purpose of cost analyses.

Exploratory analyses report data (primary outcome) on the resource use and costs for ‘same-day’ care for participants by intervention arm. This analysis includes the data on index contact and other resource use on the same day as the index contact. These exploratory analyses used unit costs for triage contacts as derived from trial data (as above), and published unit cost data on other contacts by type. A sample of data was collected within the trial to provide information on the duration of GP and nurse face-to-face consultations following a triage contact. These data were considered in the context of exploratory analyses of same-day care, and costs of same-day care.

Data analysis: presentation of analysis

Economic analyses were consistent with the methods described for the main statistical analyses (effectiveness outcomes data). The primary economic analyses were based on the ITT trial data (as described above). Data are presented descriptively, and thereafter cost analyses use a random-effects regression model taking account of the hierarchical nature of the study design (i.e. allocation by practice) and allowing for adjustment for practice-level minimisation variables (geographical location, deprivation level and size of practice) and participant-level covariates for age and gender. Data were initially explored using a GLM fitted with the appropriate choice of family and link function according to the type of data and its properties. Based on findings from these analyses (using GLM methods), main analyses are presented using a hierarchical multilevel model, assuming normally distributed total cost data. The ICC is reported for primary cost analyses.

The primary economic analyses present estimates of the mean cost of care across each of the trial arms, as above. Primary analyses report on participants with data on the POM, i.e. a complete case analysis. Regression-based methods (as above) were used to estimate difference in costs for care between trial arms, based on the 28-day data included in the primary outcome. CIs (95%) are estimated using parametric methods. Typically, where a sample has a large number of observations, as in this case (with > 16,000 participants in ESTEEM), incorporating central limit theorem implies parametric tests are appropriate and may be used for analysis of resource use and cost data. Item-level costs are presented descriptively, consistent with the data presentation in the effectiveness analyses.

In secondary analyses, a per-protocol analysis has been performed (for the primary economic outcome only), including only patients who received the triage intervention, this being consistent with the main statistical analysis plan and effectiveness analyses. As no difference is reported on EQ-5D single index values (quality-adjusted life-year weights), by treatment allocation, no exploratory analyses are considered on cost-effectiveness analyses using this outcome.

Sensitivity analyses were undertaken against the primary analyses to explore the implications of uncertainty in data used and the assumptions made within the primary analyses. Sensitivity analyses included an analysis of primary outcome data (total 28-day cost), with missing data imputed via multiple imputation methods (as used in effectiveness analyses).

Results from exploratory analyses are presented descriptively, and with regression-based methods used to provide comparative analyses where appropriate.

Results are presented in tabular format using mean estimates of resource use and cost, with summary measures on the distribution around the mean. A broader presentation of findings are presented in tabular format consistent with the approach described as cost–consequences analyses, presenting estimates of costs alongside the expected impacts associated with interventions, for example safety outcomes, health status and measures of patient experience. Cost–consequences analyses are regarded as a form of full economic evaluation, even though the costs and outcomes are not brought together in a cost-effectiveness ratio. 61

Sampling, recruitment and consent procedures

Process evaluation data were collected from a purposive sample of 10 of the main trial practices (four practices in each triage arm and two in UC) shown in Table 5. The practices were sampled from three regions (Devon, Bristol and Warwick) and varied in terms of their list sizes and locations (inner city, urban, suburban and rural).

Patient recruitment

Practices provided the research team with ID numbers for the first 20 consecutive patients eligible for trial entry, within 3 days of a same-day consultation request. Eligible patients were aged < 12 years or ≥ 16 years, with proxy participation by parents or guardians of those patients aged < 12 years. Practices provided the research team with an anonymised list of the patients approached, including patients’ internal practice ID number, patients’ initials, age, gender and ethnicity, as well as an indication of their anticipated views on the triage systems if this was known but practices did not always provide this latter information. Clinicians who managed same-day consultations were asked to review the sampling frame and to remove any patient where they believed interviews would be clinically inappropriate and/or likely to cause distress, either to patients themselves and/or to their families (e.g. end-stage terminal illness or death within the immediate family).

We aimed to recruit 15 patients per trial arm, to secure a total of 45 patient interviews. Patients were to be selected purposively to represent as diverse a sample as possible, based on age, gender, ethnicity and anticipated views (where possible) on the triage system. However, sluggish patient recruitment necessitated revising the target sample size downwards, which meant that purposive sampling was compromised.

Practices were provided with letters to send, inviting patients to interview (enclosing patient information leaflets, reply sheets and reply-paid envelopes). Where possible, interviews were secured within 2 weeks of patients’ index same-day consultation requests. Patients who were potentially willing to be interviewed returned the study reply sheets containing their preferred contact details directly to the research team. It was intended that five of the first patients in the main trial who replied from each practice would be contacted by the process evaluation researchers to discuss the study further and to agree mutually convenient interview dates and times with those who wished to proceed. This step was to be repeated until four willing patients from each practice, with diverse demographic characteristics, were found. The slow response necessitated revising the recruitment process, so that all patients returning their contact information and consent to be contacted were approached. The recruitment criteria were also revised to provide patients with the option of a telephone interview. This proved critical in improving the response rate, and the majority of the interviews with patients were conducted by telephone. This seems an appropriate method to use in a study of telephone-based interventions.

Interviewees’ written consent to participate was obtained by the researchers either at interview, if face to face, or by post if interviews were conducted by telephone. Written consent to telephone interviews was obtained retrospectively in some cases, although all patients were asked to provide verbal consent prior to commencing the telephone interview.

Staff participants

We aimed to recruit five staff members from each practice, to secure a total of 50 interviews. All practice staff within sampled practices were invited to participate in the process evaluation; those who responded to the invitation and went on to participate were self-selected. A purposive sampling strategy was designed, selecting potential staff interviewees for occupational diversity to ensure that the views of GPs, nurses, practice managers and reception staff were represented. Staff members of both sexes and of varied ages were approached. Each staff member received a personally addressed letter signed by the investigators in their respective recruitment sites, inviting them to participate in a study interview. The invitation letters were accompanied by staff participant information sheets, reply sheets and reply-paid envelopes. Staff members who were potentially willing to be interviewed returned the study reply sheets containing their preferred contact details directly to the research team. Potential participants were contacted by the researcher to discuss the study further and to agree a mutually convenient interview date and time with those who wished to proceed. This step was repeated until five staff members, representing each occupational group at each practice, were found. The small number of staff members who expressed an interest, but were not selected for interview, received letters of thanks. Interviewees’ written consent to participate was obtained by the researchers at interview. Tables 6 and 7 show the numbers of patient and staff participants recruited.

Reasons for practice dropout

All of the practices that withdrew did so prior to patient recruitment, although one practice in NT did begin the run-in period and had commenced managing patients with a triage system. This practice had two nurses delivering the triage but felt that it could not manage the demand. The practice halted the triage and decided to use the funds made available through the trial to recruit locum nurses to cover the triage. Unfortunately, the practice was unable to find any suitable candidates and felt, therefore, that it could not continue with the trial. The post-withdrawal questionnaire was returned by only one of the two UC practices that withdrew – the practice reporting dissatisfaction with the trial arm to which they had been randomised. We are unable to comment on why the other UC practice withdrew. Nine of the ten practices that withdrew from NT returned a post-withdrawal questionnaire. Although one practice was dissatisfied with the randomisation to NT, the majority of reasons for withdrawal revolved around the practice feeling under-resourced to cope with the demands of the trial: two practices had difficulty recruiting additional staff, four practices had difficulty extending staff hours as needed, and, in one case, nursing staff had left the practice (for reasons other than the trial). Other reasons for practice withdrawal from the trial (each reason cited by only one practice) included retirement of key staff members, key staff on long-term sick leave, insufficient commitment from practice staff, lack of support from the ESTEEM research team, and introduction of a triage system between recruitment and the start of the trial.

Components of the primary outcome

Descriptive data regarding the 20 individual contact types that comprise the primary outcome are shown in Table 16; also shown are descriptive data for all GP within-practice contacts, all nurse within-practice contacts, all within-practice contacts (including GP, nurse and unspecified), all walk-in centre contacts (numbers are similar across all arms, indicating that any differences in recording of walk-in centre contacts across the locations has not resulted in a recording bias across the arms) and all out-of-hours primary care contacts.

Did not attend contacts

Total DNAs had similar means across all treatment arms, and were a slightly higher proportion of the total number of contacts (including attended and non-attended contacts) for the UC arm [2.5% (265/10,447); see Table 16] compared with the triage arms. The proportions of DNAs for telephone contacts made by the triaging clinician type in the two triage arms were similar [1.7% (113/6671) for GP telephone contacts in the GPT arm, and 1.1% (61/5560) for nurse telephone contacts in the NT arm].

Patient self-reported resource use

The proportions of patients who contacted NHS Direct were similar across the three arms, as were the mean numbers of contacts (see Table 16).

Duration of consultations within trial

Based on Clinician Form data, the mean contact duration in UC was longer than for either of the triage arms [9.5 (SD 5.0) minutes compared with 4.0 (SD 2.8) minutes in GPT and 6.6 (SD 3.8) minutes in NT; Table 17, including first management/triage contacts only]. The duration of GP and nurse face-to-face consultations for ESTEEM patients (taken for a sample of patients on 2 days) indicated that the mean duration of GP face-to-face consultations in the GPT arm was slightly longer than in NT [12.4 (SD 7.1) minutes compared with 11.5 (SD 6.4) minutes; see Table 17] and longer than in UC [9.8 (SD 5.1) minutes] when including consultations on the index day or subsequent day only. Nurse face-to-face consultations had a mean duration of 13.9 (SD 8.8) minutes in GPT, 11.0 (SD 6.6) minutes in UC and 11.0 (SD 8.1) minutes in NT, again including only patients being seen on the index day or the following day.

Estimated patient–clinician contact times indicated little difference in overall contact time comparing GPT with UC, although a slightly increased amount of the total clinician time was fulfilled by nurses in the GPT arm. The overall patient–clinician contact time was noticeably longer in the NT arm, with a much greater proportion of the time being covered by nurses, largely due to the nurse telephone triage aspect of patient management in this arm (see Table 17).

Resources required for triage interventions

The GPT and NT interventions involve resource use and cost for (1) training and set-up and (2) the GP or nurse contact time for the triage event. In addition, for the NT intervention there is a requirement for CDSS to support the intervention. Resource-use estimates presented here are the incremental (additional) resources required for delivery of triage interventions, in addition to the UC (control).

Training and set-up

The training schedule required to establish and set up the triage interventions consists of two components, first training on organisational plans and arrangements for triage, regardless of type, and, second, specific skills training for either GP or nurse triage skills (Table 20). This table also highlights the need for CDSS to support the NT intervention.

Tables 21 and 22 report data collected within trial on the staff time, by staff grade, associated with training events for triage interventions. All active intervention practices in the trial (n = 28) had training on an organisational approach to triage; this training was provided by Productive Primary Care. Organisational training sessions were group training events, open to practice staff involved in management of same-day requests by patients. These sessions were relatively short, with a mean (SD) duration of 84 (27) minutes for GPT, and 107 (31) minutes for NT.

General practitioner triage skills training, provided by Productive Primary Care, was delivered in one session with a mean time of 76 (SD 27) minutes. Nurse triage skills training, provided by Plain Healthcare, was delivered using two sessions spread over one full day, with a combined mean duration of 335 (SD 111) minutes. Nurse triage training also included individual training sessions, delivered remotely as one-to-one training with individual staff (nurses only) delivering the intervention; training was delivered over 3 hours (180 minutes) per person (1 × 2-hour session, followed by 1 × 1-hour session). The number of nurses per practice attending these individual training sessions ranged from one to seven, with the mean number being 3.0 (SD 1.4).

Data collected within the trial indicated that the provider of training for GPT practices (Productive Primary Care) delivered both the organisational training and the GP triage skills training, with the time for each component being 84 minutes and 76 minutes, respectively (as above), i.e. a total training time at a mean of 160 (SD 49, range 120–240) minutes.

Further information on data collected to inform the estimate of training costs is presented in Appendix 24.

Software (nurse triage)

Software is used to support the NT intervention. The CDSS provided by Plain Healthcare is renewed annually (see cost below in Table 24), with annual licence renewal, and is inclusive of 24-hour support. A minimum of one software licence is required per NT practice, on a single-user basis; if a practice has multiple users then each user requires a separate licence or software purchase. Implementation (set-up) of the software requires support from Plain Healthcare, with a 2-day implementation process (and associated fee, see Table 23).

Clinician contact time (general practitioner and nurse time) for triage contact

Data on clinician contact time for the triage contact (index contact) are from within-trial data collection, and have been reported earlier (see Table 17) by triage intervention. Summary data are presented in Table 23. To estimate clinician contact time for triage contacts, in cost analyses, participant-level data were used when a GP delivered the GPT intervention (n = 5567) and when a nurse delivered the NT intervention (n = 5535); see Appendix 25 for detail. Mean contact time for GP and nurse triage contacts are 4.00 (SD 2.83) minutes and 6.56 (SD 3.83) minutes, respectively. Appendix 26 reports mean triage contact time by practice.

Estimate of costs for delivery of triage interventions

Resource use data reported above are combined with unit cost data and market prices to estimate the mean patient-level cost per triage contact for GPT and NT. Table 24 presents the unit cost data used to estimate intervention costs. Unit costs for GP and nurse time include cost elements associated with staff costs, practice expenses (associated staff costs, office and general business expenses – including telephone, overheads and capital costs) and an allowance for non-contact time (ratio of direct contact to non-contact time with patients); see the method set out by Curtis. 38 In the ‘base-case’ unit cost estimate the cost per minute for GPs includes cost allowance for qualifications,38 and excludes costs associated with other direct staff costs (reference to direct staff costs applicable only to GP costs). This latter assumption is based on the nature of the telephone triage contact, as a single telephone consultation contact. These unit costs are subject to other assumptions on cost components in sensitivity analyses.

Table 25 presents the estimated cost associated with the training and CDSS required to set up and support the triage interventions. These costs represent upfront training and CDSS costs, in the first instance, but also reflect an expected ongoing cost to support triage intervention in a practice setting over time (i.e. annual software cost, recurrent training need). In order to estimate a mean patient level unit cost for triage interventions, estimated training, set-up and software costs are distributed over an estimate of the number of patients per practice expected to request a same-day appointment (with a GP), and therefore reflect the expected estimated workload against triage interventions (where available).

TABLE 25 - Mean cost of per-practice training and software required for triage interventions, and estimate of mean cost per same-day appointment request

N/A, not applicable.

In the ESTEEM trial, the same provider was used for the organisational training and GP triage skills; this provider had a flat rate fee of £1350 per day. This rate was not reduced if organisational training was provided alone, as in the NT intervention training sessions.

Includes software cost of £5709 plus implementation fee at £1760. Assumes that the average practice can maintain triage with a licence on a single PC/single-user system/basis. For additional users additional licences cost £5709.

Training component		GPT (mean, £)	NT (mean, £)
Trainer’s fees:a
Trainer’s fee for organisational training session		1350	1350
GP triage skills training			N/A
Nurse triage skills		N/A	1129
Nurse triage skills 1 : 1 training (£292.50 per nurse attending)		N/A	878
Practice staff training cost: based on mean count and duration data reported (see Tables 21 and 22, plus detail on 1 : 1 training) and unit costs (see Table 24, detail provided below)		1666	2056
Training subtotal		3016	5413
Softwareb		N/A	7499
Training and set-up subtotal		3016	12,912
Estimated training/software cost per triage contact (assuming 7000 requests per year per practice for a same-day appointment4)		0.43	1.84

Estimates of triage workload (per practice per year) are based on data reported indicating a rate of 3.8 same-day requests per 1000 patients (on GP list) per day;4 this rate is applied using an average assumed practice size of 7000 patients to estimate the mean patient level unit costs, resulting in a projected number of same-day requests per annum at approximately 7000 (estimate 6916 requests). In sensitivity analyses we apply data collected across 1 week in each of the ESTEEM practices, where data indicate a rate of 5.53 (SD = 2.16) same-day appointment requests per 1000 patients on the practice list; this leads to an estimate of 10,000 same-day requests per year with a practice size of 7000.

Triage costs associated with training and software are estimated assuming that costs are distributed over a 12-month period (for base-case cost estimates). In sensitivity analyses the cost is estimated when training and software costs are distributed over a 24-month period (expected 2-year workload) and over a 60-month time frame (although in this latter scenario there are likely to be additional costs associated with the retraining associated with staff turnover).

Using the above assumptions, the estimated mean cost for the training component in the GPT intervention is £0.43 per triage contact, and £1.84 per nurse triage contact for training and software cost components. Sensitivity analyses explore the impact of assumptions on number of same-day requests per year per practice (7000 patients, base case), assuming that the workload for same-day requests per practice is 10,000 requests per year (based on data from ESTEEM), the training component in the GPT intervention costs £0.30 per triage contact, and training and software components cost £1.29 per nurse triage contact.

The higher cost on NT reflects the costs associated with software requirements, although training costs for NT are also twice those estimated in the GPT intervention. These cost estimates involve a number of assumptions; however, this component of cost is a relatively small part of the total mean estimated cost for the triage interventions (see Table 26), and adjustments to these assumptions make a negligible difference to the estimated 12-, 24- or 60-month triage intervention costs [see Sensitivity analyses (primary cost analyses)].

Table 26 presents estimates of the cost for the triage interventions. GPT is estimated to cost £14.03 per triage contact, and NT has an estimated mean cost of £7.62 per contact. The difference in cost estimates reflects the relatively low cost per minute for nurse time compared with GP time. The training cost component for GPT is a relatively small proportion of triage cost (3.2%) compared with NT, for which the training and software costs represent 24% of the estimated cost for the NT intervention.

Table 27 presents summary results for sensitivity analyses. Sensitivity analyses indicate mean triage costs per contact for GPT and NT interventions of £13.82 and £7.11, respectively, for which cost estimates are based on calculation of costs over 24 months and distribution of costs over a 24-month workload (same-day requests). The corresponding figures when costs are distributed over a 60-month (5-year) workload are £13.69 and £6.81, respectively. Sensitivity analyses indicate that the estimate of the mean NT cost is £8.44 per contact when the practice needs to purchase two licences for the software used.

Where an alternative estimate is used on expected number of same-day requests per practice (assuming a practice size of 7000), applying data from ESTEEM indicating 5.53 requests per day per 1000 patients, we estimate unit costs of £13.90 and £7.07, respectively, for GP and nurse triage contacts.

In base-case estimates we apply unit costs of £3.40 and £0.88 per minute for GP and nurse time; however, when making different assumptions (see Appendix 27) and applying the published unit cost (per minute or hour) excluding cost allowance for qualifications, these being £2.80 and £0.72, respectively, for GP and nurse time, we estimate unit costs of triage contact at £11.63 and £6.90 for GP and nurse triage, respectively. When using staff unit costs where there is an allowance for costs associated with other direct care staff costs, aligned to the GP role, we apply unit costs of £3.70 and £0.88 (same as base case for nurse) and estimate unit costs of triage at £15.25 and £7.62 for GP and nurse triage contacts, respectively.

Estimate of resource use and cost: primary analyses

Table 28 presents data on resource use by type of contact and the associated estimate of unit cost by type, for each arm of the trial, and the mean total 28-day cost by intervention arm. Data are presented for the participants with complete data collection in the primary outcome (n = 16,211). The resource-use (contact) data are consistent with the effectiveness data presented earlier, and are presented here for consideration alongside cost estimates. The mean 28-day estimates of cost are similar for all three interventions, with all slightly above a mean of £75.

Table 29 presents cost estimates by summary category, with cost estimates for GP contacts (in practice, via telephone and home visits) combined, and the same for nurse contacts. Data presented show that GP costs account for 88% of the mean 28-day cost in UC, and for 69% and 68%, respectively, in GPT and NT. When the triage cost is combined with the GP cost in the active triage intervention arms, the combined triage and GP costs account for 85% of the mean total 28-day cost. When nurse contacts are added to triage and GP contacts this subtotal of primary care ‘in-practice’ contacts account for 92–93% of the mean total 28-day costs, with remaining contacts in the categories for out-of-hours primary care, walk-in centres and A&E, representing a small proportion of the mean total 28-day cost (7.5–8%). The mean costs associated with triage consultations are £13.01 for GPT and £6.83 for NT.

The results are presented in Table 29 for unadjusted cost differences across the three interventions, for mean total 28-day costs, and also for regression-based statistical analyses, using a multilevel model (see Data analysis), with adjustment for practice-level clusters and other prespecified covariates. Primary cost analyses, using these base-case estimates (regression estimates), show no statistically significant differences in 28-day mean total cost by intervention arm.

Estimate of resource use and cost: secondary analyses

Secondary analyses, using base-case assumptions on a ‘per-protocol’ participant basis (see Glossary), shows the 28-day mean costs for GPT and NT slightly reduced compared with ITT analyses, but with no difference reported across all comparisons of cost data in base-case adjusted analyses (see Table 30).

Sensitivity analyses (primary cost analyses)

Table 30 presents sensitivity analyses, against the primary analyses on total 28-day cost estimates, using different assumptions on analyses. Sensitivity analysis presents results using base-case assumptions but with imputation methods used to replace missing data (imputation methods are consistent with those used in multiple imputation analyses for the primary outcome; see Table 12). Here we see no difference in cost estimates or adjusted differences across the three trial arms. Table 30 reports sensitivity analyses. Sensitivity analysis considers use of different regression methods, using a GLM regression approach with two different combinations of family and link (using Gaussian and gamma distributions; the former being equivalent to assuming normally distributed data). These differing specifications of GLM regression models show no difference in results, and indicate that it is appropriate to assume normally distributed total cost data; therefore, base-case results are from a multilevel model, with random effects, assuming normally distributed data and allowing for cluster data. Consistent with the base-case analyses, GLM models show no difference in 28-day total cost across comparisons by intervention type.

Sensitivity analysis (see Table 30) addresses uncertainty in choice of unit cost data for out-of-hours primary care contacts: in the base case, one standard unit cost is applied for all out-of-hours primary care contacts compared with sensitivity analysis, for which a different cost structure is used for unit cost data on out-of-hours primary care contacts (see Appendix 28). Sensitivity analysis shows no difference in results when alternative data are used for out-of-hours unit costs. Sensitivity analysis presents results for which the unit cost data related to GP time cost, and nurse time cost, are based on the lower estimate of cost presented by Curtis. 38

In this scenario, analyses are based on a unit cost of £2.80 per minute for GP time (vs. £3.70 in base case), with no allowance for costs associated with GP qualifications or for costs associated with direct costs for related staff. In this analysis, unit cost data for nurse time, per minute, is £0.75 (vs. £0.88 in base case), with no allowance for costs associated with nurse qualification. The majority of the mean 28-day total costs are related to GP and nurse contacts, and, in this low-cost sensitivity analysis, the mean 28-day cost is lower for each of the intervention types, at £59.47, £60.28 and £60.89 for UC, GPT and NT, respectively, but we see no difference in costs in comparisons across intervention type.

In the trial results (see Table 17) we see a mean contact time for the UC index contact (contact resulting from same-day request), which is a lower duration of contact than that seen in the standard costing used here for a GP consultation; in the unit cost estimate applied in base-case cost analyses an allowance is made for 11.7 minutes per GP contact (in surgery). This expected consultation duration is based on analyses of data collected in a 2006–7 UK General Practice Workload Survey (Leeds Information Centre, 2007), which involved a representative sample of 329 practices across the UK, with staff completing diary sheets over a 1-week period (as cited by Curtis38). The estimates of contact duration, and subsequent cost, are based on an allocation of GP time across a recorded number of consultations, and not on a duration recorded per contact. The data collected in ESTEEM, in the UC arm, suggests that GP consultations resulting from requests for appointments on the ‘same-day’ have a mean (SD) duration of 9.5 (5.0) minutes (see Table 17), which is shorter than the estimated 11.7-minute duration reported by Curtis38 in relation to the broader category of GP consultations (in practice), of which same-day requests are one component. Data from ESTEEM, although based on over 5600 observations, are from 14 GP practices, and are collected in a trial setting, and are estimates of mean ‘contact time’, therefore some caution is needed on the generalisability of these data, and on any direct comparison with data estimated using a ‘top-down’ approach. Although we succeeded in recruiting large numbers of practices and patients from diverse geographical locations, enhancing the generalisability of our results, our findings may be less applicable to practices serving populations with greater ethnic diversity or those located in inner-city areas with very high levels of deprivation. However, if this difference were to be supported by future research, a difference of 2.2 minutes in consultation duration would reflect a reduction in cost for a GP consultation (in practice) from £43 to £35.15 for a consultation resulting from a ‘same-day’ request.

Therefore, in sensitivity analysis, we use a unit cost of £35.15 (9.5 minutes × £3.70/minute) for the UC index contact (GP consultation). In this scenario we see that the 28-day cost for UC is less than those estimated for both GPT and NT intervention arms. This difference in 28-day cost is statistically significant, with regression analyses predicting a mean cost difference of £5.43 (lower) for UC compared with GPT, and £7.00 (lower) for UC compared with NT.

Exploratory cost analyses: same-day contacts and cost

Table 31 presents cost estimates for contacts that are reported on the same-day as the same-day request, comprising the index contact and subsequent contacts that constitute the POM, on the same day. The total number of recorded contacts over 28 days, aggregating over all participants (n = 16,211 in complete case analyses) is 39,736, and 56% of these (n = 22,471) are recorded as contacts on the same day as the initial request (see Appendix 29). The mean total number of contacts on the same day are 0.99, 1.49 and 1.70, for UC, GPT and NT, respectively. Estimates of mean cost for these same-day contacts are £40.96, £34.91 and £38.25, for UC, GPT and NT, respectively. Comparison of these cost data indicates a statistically significant difference in cost when comparing GPT and UC, and NT compared with UC, but no difference between GPT and NT (although here the reported difference in cost of –£3.17 is close to being statistically significant, with a 95% CI of –£6.31 to £0.03). However, cost estimates are presented on same-day contacts for completeness, and it is difficult to interpret the same-day costs in the context of an intervention (triage) that is expected to have an immediate impact on number of contacts, with the effect of the intervention expected to be seen over the longer term (in this instance 28 days).

As discussed above (primary cost analyses) the estimated duration (time allowance) for a GP consultation, in the practice, is 11.7 minutes, using the data reported alongside the unit cost estimated by the PSSRU (Curtis38); a unit cost of £43 is used in base-case analyses (based on 11.7-minute duration). The data in Table 32 with an adjusted cost for a same-day GP consultation (at £35.15), in the UC intervention arm only, would suggest a UC same-day cost of £33.79 compared with the £40.96 estimated using base-case unit costs, a mean difference (reduction) of £7.17 in the estimated mean same-day costs, with the other triage intervention estimates remaining unchanged. In this scenario (sensitivity analysis; see Table 32) we see a difference in costs, same-day cost estimates, across the three intervention arms, with UC having a lesser estimate of same-day costs, with this being statistically significant in comparisons of UC with NT (UC £4.57 less than NT). However, as discussed above, in sensitivity analyses, it is important to be cautious when comparing data on mean duration of a GP contact from prior workload analyses38 and the participant-level mean ‘contact time’ reported in ESTEEM.

Exploratory analyses: contact duration for general practitioner contact (in practice) following triage intervention

Table 33 presents data from a sample of participants from whom data on duration of GP contact was collected following a triage intervention. These data were collected as part of a sample of data gathered in each of the participating GP practices, over approximately 2 days in each practice. The research question of interest was whether there is a difference in GP contact duration (in surgery), following a triage contact, compared with the estimated duration of 11.7 minutes, as used in the base-case cost data. 38 Sample data, collected over 2 days in each of the GP practices, indicate that consultation duration for GP contacts (in practice), on the same day as either GP- or nurse-led triage contacts, and are not significantly different from the estimated duration of 11.7 minutes. Data on these contacts (following triage), presented in Table 33, are from a relatively small sample size (n = 244; n = 415), therefore data are illustrative and are not able to support any conclusions.

Estimated total contact time for ‘practice staff’ over 28 days

Based on the mean time estimated for the delivery of the triage interventions, and the assumptions over duration of primary care contacts (see Table 33), it is estimated, based on mean number of contacts (reported in Table 12), that total contact time for primary care (practice delivered) contacts, over 28 days, is 21.86 minutes, 22.72 minutes and 27.5 minutes, for UC, GPT and NT, respectively. GP contact time over 28 days is 18.01 minutes, 17.6 minutes and 15.45 minutes, for UC, GPT and NT, respectively. If in UC contacts (only), the same-day request contact (GP in practice) is assumed to have a duration of 9.5 minutes (vs. base-case assumption of 11.7 minutes), the estimate for total contact time (minutes) for practice delivered contacts reduces to 18.86 minutes (vs. 21.86 minutes) and the estimate for the GP contact time, over 28 days, reduces to 16.01 minutes (vs. 18.01 minutes).

Economic analyses: cost–consequence analyses

The cost estimates indicate that there are no differences between interventions by type in the mean 28-day cost estimates (base case). These findings are robust to sensitivity analyses presented, with the exception of the assumption on the unit cost for same-day GP consults. Table 34 presents cost estimates alongside the other potentially relevant outcomes reported in this trial, in the form of a cost–consequence analysis, in which disaggregated cost data are presented alongside outcome data.

Although triage interventions include a higher number of contacts, including the index contact, the 28-day costs, and those costs more directly falling on practice-level budgets, are the same across all three interventions. There is no difference in safety outcomes, DNAs or self-report health status at 28-day follow-up. There is evidence of differences in self-reported patient satisfaction, with NT indicated as not performing as well as UC or GPT in self-reported patient satisfaction outcomes.

Motivation to take part in the trial

Motivations for entering the trial, and whether these were consensual and effectively communicated within practices, affected how the trial intervention was experienced by practice staff. There was a range of motivations reported for entering the trial.

Research-related motives cited by informants included the fact that theirs was a research practice or a practice that valued research, wanting to know the answer to the research question, and the fact that this was seen to be ‘a good trial’.

Challenges in current practice were also catalysts; practices were aware when their appointments system was not working well; they were not coping with increased demand, and increasing workloads and triage was seen as a way of addressing this.

I’m not sure really, just a sort of sense that it would be something that would involve everybody and there’s been a lot of talk about, you know, some people finding it difficult to offer enough appointments and you know, that sort of workload issues and the staff having difficulty with the phones and it just seemed like it was an opportunity to try something out, which sort of shows willing to make some changes if changes proved you know, proved to work I suppose.

GP 05, practice 4 (NT)

Introducing triage had often been mooted in practices but abandoned because of lack of consensus on how to establish and organise it, and the daunting enormity of the task. The ESTEEM trial was seen as a good way of trying out triage without internal disagreements on how to do it, and with a clear and immutable model and support from outside the practice. Part of the training for the ESTEEM trial was a presentation by the organisation ‘Productive Primary Care’, taking place after randomisation, and offering a clear model of efficacy of triage, which some practices found inspirational, although this presentation created some problems, which will be discussed below.

Three of the practices were excited by the fact that the trial was something that the whole practice could be engaged in, and its cohesive effect for practice staff was highly valued. However, staff in four of the practices reported lack of consensus about taking part, with some staff feeling that they had not been consulted or that their reservations had been glossed over. In two of the practices, both in which there was no consensus, the motivation to take part was reported to be financial. Often it was a combination of several of these factors, as well as the persuasive powers of one or two highly motivated individuals which drove the decision to take part. Half of the practices reported that the decision had been made with full consultation and consensus from all of the staff.

The range of pathways to entering the trial was echoed in a variety of communication styles about doing so, from ‘top-down’ authoritarian to democratic and consultative. In practice 1, a NT practice, as an example of the former, one of the partners was reported to have been motivated by the remuneration attached to the trial and himself referred to the ‘status and resources’ associated with being a research practice. Doctors here had been discussing the possibility of triage for some time without reaching agreement, and felt that this could be a way of introducing it without dissent because the protocol could not be argued with or changed. They discussed it among themselves but not with other practice staff, whose opinion was not a concern to them.

Yes and also people understand that we are a research practice and we are obliged to do so much research. And if we don’t do so much research then our status and our resources that come with that will get chopped and if they don’t play the game then that will get chopped so even if they don’t necessarily think this is best thing since sliced bread, that’s going to happen and if they are not on board then . . .

GP 04, practice 1 (NT)

Consequently, other staff, including the nurses who ended up performing the triaging, reported that they had not been included in the decision-making and felt that the decision had been imposed on them by the people least affected by its consequences. The consequent lack of ‘ownership’ of the trial impacted negatively on how triage was experienced and on its acceptability, and appeared to have a divisive effect on the practice as a whole.

No, I think the people who decided they wanted to do it they just thought the money was an attraction and they didn’t think of the consequences. They didn’t ask me who would do the appointments and how it would be, and it was just ‘this is what we’re doing’. And they don’t listen to the people that know. It is a shame really.

Manager, practice 1 (NT)

In contrast, inclusive and democratic decision-making appeared to have a cohesive effect on practices and impacted positively on how the trial was experienced. Participating in the trial also provided an opportunity for good communication and developing good staff relationships. In practice 2, also a NT practice, the practice manager drove the decision to take part but the decision was taken after discussion with clinical and all other staff, and everyone in the practice was involved and motivated. Staff members were pleased to have a protocol to guide the introduction of triage and anticipated that the effect of the trial would be to bring the practice together. The nurses were ‘very pleased’ to be randomised to NT and the learning opportunity this presented.

And the other point I’d make is, I think it brought the practice together in a number of facets, in a number of different ways because we engaged almost the whole practice. Everybody knew it was going on; everybody was involved in it in some shape or form. And with two new nurses, again, I think the rigours of constraints of the ESTEEM trial helped to bring the process into place, that they wouldn’t have learnt in any other way. And I think massaging the information and collecting it, was a lot of fun . . .

Practice manager, practice 2 (NT)

Staff at another NT practice, practice 3, reported mixed reactions to entering the trial. Doctors reported the motivation as responding to a perceived need for a better way of handling same-day appointment requests, and were also attracted by the remuneration. Administrative staff, however, felt that they had been ‘sold’ the idea rather than fully consulted. It is worth noting that neither of the two NT practices that reported being financially motivated to enter the trial provided any additional nurse capacity to absorb the increased workload caused to nurses, as recommended by the research team. Triaging was added to nurses’ already busy working day. Despite not feeling fully included in the decision-making, staff at practice 3 also reported a cohesive effect on the practice, and excitement about trying something new, perhaps reflecting the excellent relationships in this practice, as observed by the researcher. The practice manager described a mood of excitement at taking part and pleasure that patients might benefit:

But you know, also we were pleased that we were doing it in one sense because it’s interesting to do something new and to have a different challenge. And potentially if it was going to have a good effect on the patient and doctor appointments then yeah, we were quite excited I suppose from that point of view.

Practice manager, practice 3 (NT)

In some of the practices there was no consensus among medical staff about entering the trial. Practice 6, a GPT practice, was one of these. Some doctors reported that pressure had been applied by one of the partners to agree to it. Communication in this practice did not appear to be optimal and informants’ accounts of a number of issues, including trial participation, were at variance with each other.

I think that we were pressurised by one partner, yes . . . And when I raised concerns it was read as objections and I was pushed down.

GP 03, practice 6 (GPT)

Motivation to enter the trial: key themes

• Reasons for entering the trial, how the decision was made and by whom, and how the decision was communicated all affected how it was experienced.

• Motivations reported were an interest in research, the high quality of the trial, a perceived need to change the appointment system, an interest in trying triage to manage demand, the fact that a clear model of how to introduce triage was provided, and the financial incentive.

• There was a range of communication styles in relation to the decision to take part, from ‘top-down’ to democratic or consultative.

• There was not always consensus within practices about taking part, even among doctors.

• A number of practices regarded participating in the trial as an exciting challenge.

• Several practices described trial participation as having a cohesive effect on the practice and improving communication.

Attitudes to randomisation

How the trial was experienced by staff was related, at least in part, to their attitudes to the arm into which they were randomised. In two of the NT practices – practices 2 and 3 – staff said that they were pleased to have been randomised to NT and were excited about the learning opportunities NT presented.

Well we were very, you know pleased to be part of research and we liked getting the nurse-led triage because we felt it’d be a good learning curve.

Nurse, practice 2 (NT)

In practice 3, one of the GPs reported being relieved that they had not been randomised to GPT, which was perceived as being more effort for doctors. In practice 1, on the other hand, none of the clinical staff, nurses or doctors was pleased with being allocated to NT. Both of the practice 4 doctors interviewed also had reservations about NT, believing that triage should be done by the most highly trained clinicians, i.e. doctors, especially if they could be triaging their own patients.

I think most people recognise that triage is probably the most difficult thing to do and that it often works best when the most highly trained people do it. And that probably is why personally I feel as though it might be better if I did it for my own patients, which is what I effectively thought I was doing before. And I think that also, you know, that algorithmic medicine is pretty awful and not really as clever as it looks.

GP 05, practice 4 (NT)

In two of the GPT practices (practices 7 and 8) a few staff, but not all, expressed disappointment that they had not been randomised to NT, but most were pleased with GPT, as were those in practice 5.

Signing up for the ESTEEM trial was a gamble really. Because we may have been randomised into one of the other two levels and I tell you what, if we’d been randomised into NT, we’d have been up the creek without a paddle [laughter].

GP 04, practice 5 (GPT)

Reasons given included nurses not being keen and the perception that GPs could do more on the phone. In contrast, practice 6 had a very experienced nurse practitioner who for the last 10 years had triaged, face to face, all patients requesting a same-day doctor’s appointment when the appointments had run out. Randomisation to GPT represented a lack of fit with existing experience and expertise within the practice, wasting the resource of an experienced triaging nurse practitioner. Their duty doctor system also caused additional workload for GPs through lost consultation time.

Attitudes to randomisation: key themes

• Of NT practices, two were pleased with their randomisation arm and two would have preferred GPT.

• There was a perception that GPT was a more appropriate model than NT.

• There was a distrust of CDSS.

• Of GPT practices, three were pleased with their randomisation arm and one was not.

• The GPT practice that was displeased had had a highly effective nurse face-to-face triage system in place prior to the trial and no problems managing same-day appointment requests.

Implementation of the study protocol

Although most of the practices had no problem with understanding and adhering to the trial protocol, the process evaluation did reveal some areas of confusion and misinterpretation and infidelities to the protocol. Practice 2, for example, disclosed that they forgot to record all of the phone calls during the run-in audit and remained unsure of the suitability of some calls for the trial, for example children with breathing difficulties.

The few glitches are basically maybe not understanding possibly how it ran 100% from our point of view and being told different things, like if we booked in for tomorrow it wasn’t ESTEEM and then it was suddenly ESTEEM and why did it change? . . . The other problem I have – and I’m still not quite sure how we get round that – is that if people . . . people are more inclined now to come down to the desk at nine o’clock in the morning. Now the triage appointments don’t start until . . . I can’t remember, at least ten. Now I have a problem saying ‘Can you go home, doctor will ‘phone you and then you might be coming back’ so it just seems a little bit . . .

Receptionist, practice 2 (NT)

Doctors reported forgetting to ask for consent for notes review.

The main problem is remembering to ask their permission, so the last question at the end. And no matter how long you do it for, it does help if it’s the second day in a row but we all work part time, and you often find that you’ve forgotten to ask them the question. Calling them back a second time to say ‘by the way I forgot to ask’ that is quite difficult. I did that a few times but it gets to the point that we’re so busy that all of us stopped doing it.

GP 05, practice 5 (GPT)

Similarly, a patient interviewee from practice 6 reported that she had presented at the desk and been told to go home and wait for a phone call; this same patient denied having requested a same-day appointment in the first place.

Practice 8 also sent home patients who presented at the desk requesting same-day appointments, mainly because receptionists had no appointments to give.

So, okay, can I just make sure I’ve got this straight? So if somebody turns up and wants an appointment that day they can’t have it, they have to go home and make the phone call?

Yeah, they do really. And we don’t like doing that, but it’s not a drop-in centre.

Practice manager, practice 8 (GPT)

Practice 5 had previously attempted and abandoned the ‘Advanced Access’ model promoted by this organisation. There was similar confusion in practice 7.

Because we were doing a triage system and as clinicians we’re all trained in triage and using our skills effectively to triage patients. But then the trial was saying no, do today’s work today and if they need to be seen book them in. So are we just a booking service then?

GP 01, practice 7 (GPT)

Implementation of the study protocol: key themes

• Most practices had no problem understanding and implementing the study protocol.

• One practice was mistakenly including patients who presented in person to the reception desk to request a same-day appointment. They would have been informed during training that such patients were ineligible for ESTEEM, and it was likely that this had subsequently been forgotten and then not communicated among practice staff.

• Practices at which receptionists had no appointments to give were also sending home people wanting a same-day appointment, telling them to phone in.

• One practice forgot to record all of the phone calls during run-in.

• There was some uncertainty in two practices about patient inclusion and exclusion criteria.

• Doctors forgot to ask for consent for notes review.

• One practice abandoned the trial on busy days.

• One practice believed the ESTEEM protocol required practices do ‘today’s work on the day’, i.e. confused it with the Advanced Access principles promoted by Productive Primary Care.

Perceptions of trial-related workload

Staff in some practices reported that the trial added considerably to their workload, others that workload had decreased, with disagreement about this both within practices and within occupational groups, especially doctors. In practice 1, a practice characterised by poor communication and no prior triage experience, perceptions of the trial were almost universally negative. A receptionist described completing the trial log sheets for the integrity checks as ‘just more work’ and ‘time-consuming’, and reported that receptionists hated doing it. One of the nurses complained that the Clinician Forms were badly designed, ‘obviously not designed by a nurse’.

. . . and there’s not much space between that bit and that bit, between the patient computer ID and that blue line at the top there, yeah. Yeah it’s trying to get quarts into pint pots and get everything on to the sheet.

Nurse, practice 1 (NT)

A GP found the forms ‘straightforward and simple’ but confessed to often forgetting to complete them. Doing so added to the consultation time.

Obviously it adds length to the consultation which is a bit of a nuisance. And we haven’t made any provision for that, so the days that I’ve had a lot of patients going through the study, I’ve run late then I would redo them [laughter].

GP 02, practice 1 (NT)

Collating the data was also described by the deputy practice manager as time-consuming; the system of their practice did not allow for bulk Read Coding of patient records to flag that they were part of the trial. Even the visits by the ESTEEM researcher were perceived as something of a time burden. Doctors were in disagreement about whether the trial had increased or decreased their workload, perhaps reflecting real workload disparities and different levels of triaging effectiveness among doctors.

Practice 2 had extensive prior telephone triage experience but had underestimated the requirements of the trial, which had greatly increased their workload. They reported a great deal of tension while it was taking place, particularly in reception. The paperwork was hard to keep up with, although the practice manager found collecting and collating the data ‘a lot of fun’. However, doctors reported feeling that they had more control over their workload. They felt that the training had been good and the run-in excellent, and appreciated the approachability of the ESTEEM researcher.

Practice 3 also had prior triage experience and reported no problems with the protocol, which they found clear, or the paperwork, and reported that the training had made everything easy. Nurses’ workload was reported to have increased enormously, however.

One informant felt strongly that the initial training was much too complicated and had fostered anxiety and confusion:

Yeah, I just thought, ‘Oh I really can’t be bothered with this’, you know, ‘we’ve got enough to do without this’. And I’ll tell you why, it’s because the chap who came to show us it, he went into too much detail for us. We didn’t need to know all that stuff that he went on about and it made us all as receptionists think, ‘Oh my crikey’. Whereas all they really had to say to us was ‘We’re trying out a new system, this is what you’ve got to say to patients’ . . . and it would have been fine. But he made it so complicated. He went through stuff that we didn’t need to know about, about what the doctors have got to do, what . . . so to me that first talk that we had, we shouldn’t have sat through that, because we all felt the same.

Receptionist, practice 4 (NT)

The practice manager in practice 5, which had no prior triage experience, found that the Productive Primary Care session was pivotal in making the trial a success and in getting more sceptical staff enthused about it. However, the GPs in this practice reported extremely high stress levels as triage calls were added to their ever-increasing workload, although two of the doctors conceded that they did feel more in control of their workload despite perceptions of its increased volume.

I think it made life quite difficult, and it is still quite difficult at the moment. It has made it more stressful, it’s made it busier, it’s made the work more intense, erm, so yes, I mean looking at the benefits from it, there must be some and I’m sure we’ll get to that later. But yes, being honest it has been quite difficult.

GP 05, practice 5 (GPT)

Female doctors were under particular pressure, with more patients wanting to consult them. Practice 5 was characterised by poor communication between management and doctors and between management and the reception team.

Doctors’ workload in practice 6, a practice with prior triage experience, decreased under the trial or it increased, depending on doctors’ accounts. One GP reported having to work extra sessions because of the surgery time lost to the duty triage system, whereas two other doctors claimed a decreased workload as patients were dealt with on the phone.

On the mornings when we’re doing the triage I think we are working less intensely than we used to.

GP 02, practice 6 (GPT)

Paperwork was checked and completed every evening by the reception manager, which increased her workload but took the pressure off other staff. Training and support from the ESTEEM researcher was reported to have been good.

In practice 7, which had prior triage experience, response to the trial was mainly positive, although there had been dissent among doctors about it, and some GPs felt that the quality of triage was poor because of the volume of calls. The trial paperwork had not been experienced as a problem. The ESTEEM researcher was regarded as approachable and easy to contact, and the run-in period was considered invaluable. However, a perverse effect of the trial protocol was noted. Patients who would previously have been given an appointment to see a nurse were now put on the triage list if they had asked to be seen that day, thus increasing triaging doctors’ workload.

Because we would allocate, you know, if somebody said they had an earache or a sore throat or a chesty cough we would have just automatically allocated them to the practice nurse, nurse practitioner but because of the trial everything had to go on the doctor, you know, the doctor’s triage. So people were phoning up who would normally have spoken to, say, T [nurse] or L [nurse] and we were saying to them, no, you’ve got to go on to the doctor’s list so they didn’t quite understand that because I think they felt that they were wasting the doctor’s time when they could have been seen that, you know, the nurse because it was something that they’ve seen in the past.

Receptionist, practice 7 (GPT)

Finally, practice 8 had a mixed experience of the trial, with considerable disparity of opinion on how it was experienced and quite a lot of negativity. This practice had previous triage experience but poor communication, with an absence of both formal and informal opportunities for sharing knowledge, information and concerns, and there were major staff continuity problems.

The extra paperwork was described as an additional burden that hindered the ability to manage the workload. This practice did not consider it had received enough support from the ESTEEM team, and had not received the promised feedback from the audit of demand. Setting up had been problematic owing to the practice starting the trial using a software solution developed during the trial and later abandoned (for details, see Strengths and limitations). The software was designed to automatically apply a study-specific Read Code to the electronic records of ESTEEM patients by way of a short keyboard command typed by the receptionist receiving the patient’s telephone call. The software required a macro to be set up on all of the computers used by the doctors, which, at this practice, was problematic due to the number of doctors working in different rooms. Staff members were consequently not confident that everything had been set up properly.

Trial-related workload: key themes

• Trial paperwork was perceived as burdensome in most of the practices.

• Trial paperwork and data collation was time-consuming and hard to keep up with.

• The paperwork added to consultation times, sometimes causing GPs to run late.

• Staff sometimes forgot to complete the paperwork.

• Deciding how patients were to be managed under the trial needed to be discussed.

• Visits from the ESTEEM researcher were sometimes considered time-consuming.

• IT issues had caused additional work.

Nurse-led telephone triage practices

Two of the NT practices triaged only in the mornings, whereas the other two also included afternoon triage sessions. In practice 1, a large practice in a small town with a personal list system but poor staff continuity, two nurses performed triage in the mornings, although usually only one at a time. There were about 30 triage calls per day. No extra nursing capacity was provided, and nurses had to work extra hours unpaid.

Practice 2, a large practice in a small town with no personal list system had three nurses triaging each morning, a number that had needed to be increased from two due to demand. Two extra nurses were appointed and existing nurse hours were also increased.

Practice 3, a small rural practice with no personal list system, did not provide additional nursing capacity and nurses worked extra hours unpaid to cope with the additional workload. Nurses triaged alone for an hour and a half in the morning, and the same in the afternoon. Because one of the doctors finished at 4 pm the nurse would come in early in her own time on that doctor’s duty day to fit triaged patients into his surgery. Timing was an issue in this practice, as in others, because doctors’ clinics coincided with triage times, which sometimes did not leave patients enough time to get in to see the doctor, especially those dependent on the bus. This could leave appointment slots unfilled and doctors having to work late. The system was reported as not working well on Friday afternoons when people had to be seen before the weekend.

The fourth NT practice, a large urban practice with a personal list system, appointed two additional nurses for the trial. Initially, triaging took place all day, although demand was found to be lower in the afternoons, and some days triaging took place only in the mornings. GPs changed their surgery times to later in the day to allow triaged patients who needed an appointment time to get to the surgery.

Nurse-led telephone triage was a challenge for practices with a personal list system at which patients needing an appointment were triaged into the next available slot.

I think the things which we’ve found awkward are that we are a practice where we encourage in favour of personal lists and we agreed to this, we understand it’s part of the trial, so it’s not a surprise, but nevertheless, you know, we’ve had days when say four of us are here and each of us are seeing each other’s patients. Now I’m seeing someone of [GP name] and he’s seen some of mine, which is not very helpful . . . And that’s annoying. I think we wouldn’t want that to continue. That would tend to grate and would break down a good doctor–patient relationship after a while.

GP 04, practice 4 (NT)

Implementation of nurse-led triage: key themes

• Two practices triaged only in the morning, whereas the other two also did so in the afternoons.

• The number of nurses triaging at the same time varied from one to three.

• Two of the practices provided extra nursing capacity for the trial. Two did not.

• Timing of triage in relation to surgery time could be a difficulty, and doctors and nurses had to adjust their hours to accommodate this.

• Triaging into ‘next available slot’ is a challenge for personal lists.

General practitioner-led triage practices

Although the ESTEEM intervention manual prescribed many aspects of how GPT was to be implemented, it was essential that practices had some flexibility to design the intervention based on their local context. Consequently, there was marked variation in the ways in which GPT was implemented between the practices. There were two main organisational structures for GPT: a duty triage doctor system, for which GPs took it in turns to provide a dedicated triage service for patients, and an integrated system, for which triage (usually of doctors’ own patients) was combined with normal surgery. However, practices seemed to have evolved systems that suited them, often involving a combination of the two organisational structures, especially in practices operating personal lists. Two practices (practices 5 and 7) combined a duty doctor system with presurgery triage, in which non-duty doctors picked their own patients off the triage list and called them back before the duty doctor took over triaging. When an appointment was necessary, patients were triaged to an appointment with the triaging duty doctor, to their ‘own’ doctor or to the first available appointment, depending on the practice. This is arguably a third organisational pathway for GPT, being neither purely a duty system (because all doctors triaged their own patients rather than just one doctor triaging all patients on the list) nor a combined system (being performed outside surgery time).

Both systems presented issues of workload disparity:

• The duty triage system meant that doctors on triage duty on busy mornings, especially Mondays, struggled to get through the triage list.

• Disparities in the integrated system resulted from extra demand falling on certain doctors. For example, some female doctors faced extra demand from female patients with certain conditions; some doctors were ‘more popular’ than others, whereas other GPs had a ‘more challenging’ or ‘demanding’ patient list.

Our difference in terms of numbers of phone calls is immense. If you work it out per session per year, some doctors are . . . the number of phone calls you do per session per year may be in the region of 250 phone calls per year. Some of us do 2000 phone calls per session per year . . . I was taken on as an extra partner here so naturally anybody who wasn’t happy with their own GP immediately came to me. So by definition I had a motley crew.

GP 06, practice 7 (GPT)

In practice 5, a small rural practice with a personal list system, all the doctors triaged their own patients for an hour and a half in the mornings followed by their surgery, at which point a duty triage doctor system came into operation. Although this worked well, the four minutes per patient scheduled for triage in this practice caused some doctors to get behind with the triage list and, consequently, their surgery.

Receptionists in this practice also had extra appointments to offer, so that walk-in patients and those unsuitable for triage could be given appointments; this was not the case in all practices. Advance appointments were also available (following a failed attempt to follow the influential Productive Primary Care ‘Advanced Access’-based model of ‘do today’s work today’, which had impressed many of the practices). Surgery times were adjusted to accommodate triaged patients. There was still a flaw in the system, in that people phoning in the late end of the morning session could not be phoned back until 3 pm because triage sessions were immediately followed by surgeries, then lunch, then home visits. In addition to the long wait for ring-back that this created, if people were phoned at 3 pm there was sometimes no time for them to get to the surgery for their appointment, especially if they travelled by bus. Doctors did stay later in the evening to accommodate this but still found that their appointment slots were all gone before afternoon triage was finished.

Practice 6, a large urban practice with no personal list system, operated a duty triage system with one doctor on duty at a time on a rota system. The duty doctor later saw, where possible, the patients he/she had triaged, which operationalised one of the main perceived advantages of GPT – that the consultation has been started on the phone, which shortens the appointment time, and the patient’s history does not have to be taken more than once. It also addressed the continuity problems that can be inherent in triage. However, practice 6 had used a uniquely effective system for same-day appointment requests prior to the trial, in which patients were triaged face to face by the nurse practitioner. GPT, according to the ESTEEM protocol, did not allow for patients wanting urgent appointments to be seen by the nurse practitioner. This increased the doctors’ workload at this practice so that extra hours had to be worked and locums employed.

Practice 8 is a small urban practice with personal lists but very poor staff continuity. Their triage system was a duty doctor one, although this practice chose to include other things on the ‘ring-back’ list, such as queries and requests for results, resulting in ‘pages and pages’ of ring-backs to do, described as a particular challenge for the doctor on duty on Mondays.

Implementation of general practitioner-led triage: key themes

• There were two main systems: a duty triage doctor system, in which doctors triaged patients but did not see them, and an integrated system, in which triage was combined with surgery.

• Two personal list practices adapted the duty doctor system so that all doctors triaged their own patients from the list before surgery and the duty system took over.

• Both duty doctor and integrated systems produced workload disparities between GPs.

• The duty doctor system meant that surgery appointments were lost while doctors were triaging, increasing workload for the other doctors.

• The integrated system in which triage was combined with surgery produced work overload and the risk of running late.

• Timing of triage in relation to appointments required adjustment of working hours.

• Some practices triaged patients needing appointments to the triage duty doctor, some to the patient’s own doctor and some to the next available appointment slot.

• The system where triaged patients were not given appointments with their own doctor was a challenge for some doctors in practices with personal lists.

What worked well for whom? The benefits and advantages of triage

The positive aspects of triage reported by practice staff have been organised into the four main themes: (1) effects on work and workload, (2) benefits for individuals, (3) advantages for the practice as a whole and (4) benefits for patients as perceived by staff. A further theme is patients’ positive experiences and views (5).

What worked less well and for whom: challenges of triage

Challenges for the practice

Despite the fact that practices were remunerated for participation in the trial, there was concern expressed that performing triage was costing them money. Apart from the increase in telephone bills, one doctor argued that in NT patients appeared to be effectively consulting twice for the same condition and had their history taken by two different people, and that this had cost implications for the practice. (It is worth noting that this doctor was from a NT practice whose motivation to participate in the trial was described by staff as financial, and which chose not to use the trial payment to provide any additional nursing capacity.)

Um, well, my inclination is to feel that we have frequently ended up with a patient in touch twice for the same problem, once by the nurse and once by me. And there isn’t value added to my consultation by the nurse having spoken to the patient already beforehand and therefore we pay for two consultations rather than one.

GP 03, practice 1 (NT)

Another doctor, from a GPT practice, cited the 51 appointment slots lost per week to duty triage as a cost to the practice, as locums had to be employed and extra sessions worked to fill the gap. The increase in the telephone bill was another cost cited.

Staff in five of the eight intervention practices claimed that there had been no reduction in appointments (practices 1, 2, 5, 6 and 7). Triage was felt to be a lot of effort for very little gain, causing organisational problems and, in one practice in particular, dissent and dissatisfaction among staff and patients alike. Routine advance appointments disappeared as appointments were blocked out for triaged patients and this was a recurring dissatisfaction, reported to be a major cause of patients’ complaints. In the two NT practices that had not provided additional nurse time (practices 1 and 3) the knock-on effects of lost nurse appointments were causing difficulties for both practice and patients. Chronic disease management in particular had been neglected, which would impact on practices achieving their Quality and Outcomes Framework targets (NHS Employers 201465), with potential loss of income and reputation. Practice 3 had to have a gap from ESTEEM while nurses caught up.

Another difficulty was the issue of negotiating the timing of surgeries in relation to the timing of triage. Often these took place at the same time of day, and triage appointments were wasted when there was no time for patients to get into the practice in time, and then surgeries ran over time in an attempt to fit people in.

In two of the NT practices some of the doctors were not happy with NT as a model. The CDSS was felt by some to be too ‘tick boxy’, and ‘algorithmic medicine’ was in any case considered an affront to good patient care. It was felt that nurses did not have diagnostic or triaging skills and that taking a patient’s history was a highly skilled proficiency acquired over years of training and practice by doctors but not nurses.

Finally, somewhat perversely there was a concern that if triage was successful the improved access it provided would not so much manage demand as increase it.

And there is a concern in people that firstly, by triage, you’re going to increase demand because the more you answer demand the more people will demand. So that’s one concern.

GP 06, practice 7 (GPT)

Challenges for the practice: key themes

• There was concern about the cost of triage.

• In NT practices patients were assessed twice.

• In some GPT practices locums had to be employed to cover surgery time lost to duty doctor triage.

• The telephone bill increased.

• There was no perceived reduction in appointments in some practices.

• The triage system caused dissent and dissatisfaction among staff.

• Patients perceived triage as a barrier to access.

• Some doctors in NT practices were not happy with NT as a model.

• CDSS was seen by some as inappropriate care.

• There was concern that demand would increase as access improved.

What is seen to work in telephone triage?

Telephone triage delivered by GPs or nurses was seen to be effective for reducing both appointments and consultation times, and therefore for rationalising resources. Telephone triage delivered by nurses can work well. It can be perfectly effective and acceptable to both practice staff and patients, and can be felt to reduce doctors’ appointments. In this process evaluation of the eight intervention practices studied, the practice whose telephone triage was perceived to significantly reduce doctors’ appointments more than any of the others was a NT practice. NT was felt to potentially create a sustainable long-term reduction in doctors’ appointments by teaching self-management skills to patients and the conditions in which a doctor’s appointment is and is not necessary. NT is likely to be more effective when performed by a nurse practitioner or a primary care nurse, i.e. a nurse with experience of history-taking and diagnosis, and who can prescribe, but these are not necessary preconditions for success. It may be more likely to effectively reduce doctors’ appointments when nurses can prescribe certain medications for commonly presented conditions. Using CDSS can be a useful training tool for nurses learning triage, although some nurses are already experienced at taking histories, diagnosis and assessing appropriate treatment options, and some nurses feel it increases the amount of time taken to triage.

Triage training and experience can encourage more effective triage by both nurses and doctors. Nurses are less likely to have experience in history-taking, diagnosis and treatment, and are likely to require a longer induction period.

If there is already a culture of triage and telephone consultation in the practice the effectiveness of the introduction of a more widely applied triage system will be increased and expedited, as both clinicians and patients are accustomed to a remotely delivered service. Initial resources need to be devoted to setting up an effective triage system that will produce resource savings when established. Staffing needs to be appropriate. Good effective communication within the practice, with formal and informal opportunities for airing and sharing problems and successes, predisposes the practice to more successful introduction of triage.

A preparatory assessment of needs and capacity predisposes practices to more successful introduction of triage, and a run-in period to practise the intervention is likely to be helpful, as it was during the ESTEEM trial. Commitment to the change by all stakeholders is essential, and should be established before triage is set up, and a champion to drive it forward and address problems is helpful. Triage can improve access, and may be more acceptable at times when access has been poor in the past, although previous good access may create easier and more trusting relations between staff and patients. The opportunity to hear the experience of others who have successfully introduced the innovation can inspire and motivate staff to drive the change, but it is probably important that this is delivered by people without an overt commitment to a particular model that may not be the most appropriate for the practice.

What is seen not to work in telephone triage?

Many of the aspects of triage which were perceived not to work were a function of the trial rather than triage per se. An example is randomisation to an arm that is perceived as less effective than the system in place before the trial, for example when effective face-to-face NT was replaced by less effective GPT. Randomisation is not likely to be an effective way of introducing a major organisational change. Another example is when practices were already effectively managing their demand for same-day appointments so that there was no perceived need for change. It is axiomatic that models of triage that do not evolve from the organisational culture and framework of a particular practice, and particularly from a perceived need for change, are less likely to work, at least in the short term. In some practices the timing demands of the trial enforced the introduction of triage at times that were felt to be inappropriate, such as during a ‘flu’ epidemic. Implementing triage is also more problematic in practices with authoritarian hierarchies and poor staff communication, in which there are no conditions for formally or informally discussing concerns and experiences.

In relation to whether or not NT is seen to work, it is worth noting that some nurses are frequently nervous about triage, which may lie outside their comfort zone of experience and training, especially if they have no diagnostic experience and rely on CDSS. Nurses embark on telephone triage from a very different starting point from doctors, and embedding NT into practice is likely to take longer than perhaps a GPT system might, and require additional preparation, support and resourcing. Launching straight into a NT system in the absence of this preparation is unlikely to be as successful as a carefully managed change. NT is less likely to be successful if practices do not support nurses by ensuring that there is sufficient nurse capacity to undertake this new role, as well as performing their usual roles (such as chronic disease management, vaccination programmes and management of dressings). Nurses need to be proficient in IT, which was not always the case in this trial. Although it was regarded as a useful training tool by some nurses who had no triage experience, the CDSS that was used (which was designed for the different demands of out-of-hours primary care services) was judged by some nurses to be unsuitable for use in daytime primary care triage in a number of ways. Using CDSS may lead some nurses to feel that their judgement is undermined with regard to when an appointment is necessary. This may increase the numbers of patients advised to attend for a face-to-face appointment, as CDSS ‘errs on the side of caution’ and may also cause a strain on practice relationships. It was seen by some to be ‘long-winded’ and seen as increasing the time taken to triage.

Doctors without experience and training in triage can lack confidence and take too much time to triage, and doctors with no experience in or desire to practice telephone consultation are less likely to be efficient at triage or to have a commitment to its success. For a few doctors, telephone triage is an insult to their model of good and safe care, and as such can represent an existential challenge to their identity as good doctors. In addition, in the ESTEEM trial, the triage model initially promoted by Productive Primary Care (and sometimes confused with the ESTEEM protocol) was not wholly compatible with triage within ESTEEM, and this may have confounded and confused the delivery of triage in the trial. Superimposing a predesigned (and quite extreme) model of triage – such as the Advanced Access model – onto an organisation without careful consideration of specific aspects of that organisation is unlikely to work, and practices that tried to do this were forced to abandon the attempt. In the Productive Primary Care model receptionists do not have appointments to offer, and advance appointments cannot be made. Both staff and patients at practices who tried to implement this, as well as those in which loss of advance appointments was an adverse effect of the need to prioritise appointments for the triage system, found this uncomfortable. It can spoil good practice or patient relationships and can be embarrassing and disempowering for receptionists. It excludes patients who, for whatever reason, find telephones inaccessible. It is a barrier to continuity, as patients can no longer book ahead with their own doctor who knows their history, thus undermining the relationship of trust between doctor and patient. The Advanced Access system was advocated as a response to the Department of Health target that patients should be able to see a doctor within 48 hours66 but was modified, as recommended by Department of Health policy in 2004, following a public confrontation with the Prime Minister by a disgruntled patient in 2006. 67 Pope et al. 67 highlight a number of publications which identify problems with the acceptability of Advanced Access to both patients and clinicians, related to its adverse impact on doctors’ workload, continuity of care and patients.

There is no one approach to performing GPT which is appropriate for all or most practices. Effectiveness depends on a number of contingencies, for example whether there is a usual doctor system, individual doctors’ confidence and competence, whether doctors have prior triage experience and/or experience of telephone consultations, and whether their model of good patient care and safe practice encompasses a commitment to seeing patients face to face. Telephone triage is more acceptable to some types of patients than others, so practice catchment and population will influence effectiveness. Success will also relate to how effective a system for providing same-day appointments exists before a triage system is introduced. GPT can increase workload and exacerbate workload disparities between doctors in a practice.

Ring-back issues are a constant challenge in telephone triage. Delay in ring-back, patients not being able to receive calls, patients who cannot be reached and poor mobile phone signals can make telephone triage less effective, frustrate both staff and patients, and waste time. Practices in which triage was organised in a way that meant patients were waiting for long periods, sometimes hours, for ring-back were doing their patients a disservice and undermining the success of telephone triage as an efficient system. Strategies for timely ring-back should be prioritised. Similarly, noise, either in the practice or in the patient’s home, can compromise the safety and effectiveness of telephone triage. Practices need to provide a quiet space for triaging, and receptionists could advise patients waiting for ring-back that they will need to be in a place of privacy and quiet. There are safety and confidentiality issues in ringing people when these conditions cannot be fulfilled. Patients should be assured of privacy and confidentiality when they are rung back; there was concern among patients about who might be ‘listening in’.

There are certain patient groups for whom triage is inappropriate, for example some people with mental illness, substance abuse issues, learning disabilities, dementia, hearing disabilities or who are very old, who were excluded from ESTEEM. Policies need to be in place to exempt such patients from a triage system.

Numbers of contacts and disposition of patients

In UC, a substantial majority of patients (87%) received a GP face-to-face contact, with no further primary outcome contacts on the index day. Other patient pathways were represented by substantially smaller proportions of patients.

In contrast, patients in the triage arms had more diverse patterns of patient management when compared with UC, possibly reflecting a more flexible approach to patient assessment and management in the triage arms.

Where GPT was implemented, 46% of patients received only the GP triage contact on the index day, and 36% received a GP face-to-face consultation on the index day following the initial GP triage contact; a smaller proportion (9%) of patients had a nurse face-to-face consultation on the index day following their initial GP triage contact. A small proportion (6%) of patients received a GP face-to-face consultation on the index day instead of being managed under the triage system.

Where NT was implemented, 22% of patients received only the nurse triage contact on the index day, and 56% of patients received a GP face-to-face consultation following the initial nurse triage contact; 9% of patients received a nurse face-to-face contact following nurse triage. For a small proportion of patients (9%), their first contact on the index day was a GP face-to-face consultation instead of being managed under the triage system.

Distribution of clinician time in practice

Considering workload from the perspective of the GP, the introduction of GPT was associated with a 55% reduction in the rate of GP face-to-face contacts on the index day (compared with UC), although there was a 49% increase in the rate of GP overall telephone and face-to-face contacts combined. Across the whole 28-day follow-up period, the reduction in the rate of GP face-to-face consultations was smaller (39%), reflecting the deferral of some workload from the index day to the follow-up period, whereas there was a 38% increase in the rate of GP telephone and face-to-face contacts combined.

Introduction of NT was associated with a 31% reduction in the rate of GP face-to-face consultations on the index day (compared with UC), and a reduction of 28% in the rate of GP telephone and face-to-face contacts combined. Across the whole 28-day follow-up period, there was a 20% reduction in the rate of GP face-to-face consultations, with a 16% reduction in the rate of GP telephone and face-to-face contacts combined.

In considering these observations regarding GP contacts, it is worth noting that the introduction of GPT involved GPs undertaking the work of triage in addition to the (reduced) number of face-to-face contacts; in contrast, where NT was introduced, nurses, not GPs, delivered the triage element – delivering the resulting ‘gain’ to GPs in reduced numbers of GP face-to-face consultations.

Taking account of both the number and type of contacts in the practice (whether with a GP or nurse, face to face or telephone) and their associated durations provided additional insight into clinician workload, which were somewhat different from those based solely on the number of contacts. In UC, the estimated duration of patient–clinician contact time within the practice on the index day (based on the first two contacts within the practice) was 9.6 minutes, compared with 10.3 minutes when GPT was implemented, and 14.8 minutes where NT was implemented. The distribution of the time by clinician was, however, markedly different across the three arms.

In UC, the substantial majority (9.1 minutes) of the 9.6 minutes estimated contact time on the index day comprised contact with the GP. Where GPT had been implemented, the estimated GP contribution to the overall workload was 9.0 out of the 10.3 minutes, with nurse contact accounting for 1.3 minutes, the latter usually resulting from nurse face-to-face contact. Where NT had been implemented, of the total 14.8 minutes of estimated contact time provided on the index day, overall, around half of this was with GPs (7.7 minutes), with the remainder (7.1 minutes) with nurses.