A multifaceted intervention to reduce antibiotic prescribing among CHIldren with acute COugh and respiratory tract infection: the CHICO cluster RCT

Antibiotic prescribing

Respiratory tract infections (RTIs) in children present a major resource implication for primary health-care services internationally for five reasons. First, they are extremely common and costly to service providers, families and employers. 1,2 Second, there is clinical uncertainty in primary care regarding the diagnosis and best management of RTIs, as reflected by the variation in the use of antibiotics in primary care for RTIs between nations,3 general practitioner (GP) practices4 and clinicians. 5 Third, antibiotic prescribing by primary care clinicians in the United Kingdom (UK) remains significantly higher than in some of our European neighbours. 6 Fourth, the overuse and misuse of existing antibiotics, combined with the slowing in the development of new antibiotics, are associated with the development and proliferation of antimicrobial resistance between7 and within8 nations as well as individuals. 6,9,10 Finally, the use of antibiotics leads to the subsequent ‘medicalisation of illness’ in which patients believe they should consult for similar symptoms in the future. 11 A number of key publications have highlighted the need for more research to define the appropriate use of antibiotics and healthcare resources for RTIs if the public health disaster of ineffective antibiotics for serious infections is to be averted. 12–14

Findings from the TARGET programme

The design of the CHIldren with Cough (CHICO) trial’s intervention was borne out of evidence from the TARGET programme’s earlier work streams. This has involved data synthesis of qualitative and quantitative evidence,15–19 a qualitative investigation to understand both parents’ information needs and influences on clinical decisions surrounding antibiotic prescribing, quantitative evidence in terms of a large multicentre, prospective cohort study (over 8300 children) to derive and internally validate a clinical rule to predict hospitalisation of children with RTIs,20,21 and the development of an intervention informed by a critical synthesis of all findings. 22 This culminated in a feasibility cluster randomised controlled trial (RCT) study to measure the acceptability of using symptoms, signs and demographic characteristics to predict hospital admittance in children presenting to primary care with RTI. 23,24 Findings across the TARGET programme were synthesised using Greene and Kreuter’s Precede-Proceed model which integrates across several behavioural theories into a unified model. 25

Qualitative work from the National Institute for Health and Care Research (NIHR) TARGET programme for Applied Research in 2016 identified clinician uncertainty as a major driver of antibiotic prescribing. Primary care clinicians acknowledge that they prescribe antibiotics for a range of medical and non-medical reasons, particularly in children, who are seen as vulnerable26 and whose clinical state can change rapidly. Many clinicians report that they prescribe antibiotics just in case to mitigate perceived risk of future hospital admission and complications. 19,26 Our earlier programme work indicated that improved identification of children at very low risk of future hospitalisation could increase confidence to withhold antibiotics in low-risk groups. 15,16 Clinical prediction rules are designed to reduce clinical uncertainty in an outcome (such as a child’s risk of hospitalisation) by assessing the strength of association between the risk of it occurring and baseline characteristics (e.g. sociodemographic characteristics or symptoms and signs of illness).

Clinical prediction rule for hospitalisation

The TARGET cohort study aimed to identify symptoms, signs and demographic characteristics that may predict hospitalisation and poor prognosis of a child. In particular, such an algorithm could potentially identify a large group of children at very low risk of hospitalisation and therefore are potentially unlikely to require antibiotics.

In line with expectations, just under 1% of the children were admitted to hospital up to 30 days after their consultation with the primary care clinician. We found seven characteristics independently associated with increased risk of hospitalisation for their acute cough or RTI during the subsequent 30 days. These are described by the STARWAVe mnemonic: Short illness duration (parent reported ≤ 3 days), Temperature (parent reported severe in previous 24 hours or ≥ 37.8° on examination), Age of patient (< 2 years), Recession (intercostal or subcostal on examination), Wheeze (on listing to chest with stethoscope), Asthma (currently diagnosed) and Vomiting (parent reported moderate or severe in the 24 hours prior to consultation). The area under the receiver operating characteristic (ROC) curve for the coefficient-based algorithm was 0.82 [95% confidence interval (CI) 0.77 to 0.87].

The prediction rule identifies children at very low, medium and high risk of future hospitalisation with advice given on how the clinician can use this information in conjunction with their own clinical judgement to decide the best course of action for each child. Assigning one point per predictive characteristic, a points-based algorithm was used to quantify absolute probabilities to three groups: (1) ‘very low’ (0.3%, 95% CI 0.2% to 0.4%) scoring 0 or 1 point; (2) ‘normal’ (1.5%, 1.0% to 1.9%) scoring 2 or 3 points and (3) ‘high’ (11.8%, 7.3% to 16.2%) scoring ≥ 4 points. The rule can potentially be effective in both reducing the overall prescription of antibiotics by increasing clinician confidence that they are not needed ‘just in case’ in the very low-risk group (just under 70% of children are in this group), as well as better identify those children in need of close monitoring (2% of children are in the high-risk group). Children in the medium-risk group (29%) have a similar risk of future hospitalisation as all children combined, so management should follow current National Institute for Health and Care Excellence (NICE) guidelines,27 which state that clinicians should decide on the use of immediate, delayed or no antibiotics based on their assessment of the child’s illness severity.

The interaction within the consultation

Our finding from our qualitative reviews was that clinicians could misinterpret parent’s communication about their concerns or ideas regarding their child’s illness as pressure for antibiotics and in some cases this led to unnecessary or unwanted antibiotics being prescribed. 26 Clinician communication was focused on differentiating minor and more serious illnesses, with the message (both implicit and explicit) that viral illnesses were minor while those that were ‘serious’ were treated with antibiotics. Clinicians and parents were often talking at cross purposes about the seriousness of the illness; parents emphasising the severity of the symptoms to demonstrate the impact on child health and to justify the consultation; clinicians seeking to justify a no-antibiotic treatment decision by minimising the problem.

The findings of our qualitative study suggested that parents want better information on the symptoms and signs of serious RTIs and when to consult,28 along with more useful advice on home management of symptoms. 29 Parents did not want to know the absolute risk of hospitalisation for their child but they did want advice and information specific to their child. 29 When parents did consult, clinician explanations of diagnosis and treatment recommendations were not well understood by parents, and they remained unclear about how to manage a RTI and when to consult. 27–29 Clinicians reported that most often they gave a simple viral diagnosis, communicating that the illness was self-limiting and did not need antibiotic treatment. 29 However, if the child’s illness appeared severe to the parent, or the parent was concerned about particular symptoms or impacts which were not addressed by the clinician, the parent reported viewing a simple viral diagnosis as inadequate. Parents’ concerns encompassed things that fell outside a simple biomedical model for RTIs, including both child health and psychosocial impacts, but reported that clinicians rarely addressed these.

The feasibility study and an ‘efficient’ design

In the feasibility trial, the intervention group antibiotic prescribing rate at consultation was 25% (compared to 37% in our earlier cohort study); however, among the control group, the overall prescribing rate was even lower at 16%. The paradoxical effects found in the feasibility study were largely explained by a post-randomisation recruitment differential, with possible Hawthorne effects. 24 In the intervention arm, there was a significantly higher recruitment rate, difference in clinician type (proportionally more practice nurses recruiting), the children were significantly younger and importantly the intervention children were more unwell at baseline (significantly higher respiratory rate, significantly higher wheeze prevalence and significantly higher parent and clinician global illness severity scores). Learning from these lessons, we proposed a more ‘efficient’ study design that would not only mitigate recruitment differential but would also be resource efficient, using an intervention designed to be replicable for the National Health Service (NHS). Rather than recruit patients and collect consent during consultation we conducted the CHICO RCT at the practice level (as a cluster RCT) using Clinical Commission Groups (CCGs) and the national NIHR Clinical Research Network (CRN) to recruit practices. Rather than trawl through the practice notes to collect primary outcome data we utilised routinely collected data (dispensing data collected by NHS Business Services Authority (NHSBSA) ePACT230 and hospitalisation rates for RTI collected by CCGs). We also took a ‘lighter touch’ to data collection using short baseline and follow-up questionnaires filled in by the practice champion (GP, nurse, practice manager or pharmacist) and embedded the intervention within the practice system rather than as a stand-alone tool. This both reduced the time it takes to open and close the tool and had the advantage of utilising data already within the system and incorporating data entered as part of the trial into the practice system, thus, not duplicating clinical work. The barriers and facilitators to this efficient design approach are reported in Chapter 6.

Aim

The aim of the CHICO RCT is to reduce antibiotic prescribing among children presenting with cough or RTI without increasing hospital admission for this condition. Objectives pertain to the practice level, as no data was collected on individual participants.

Primary objectives

To determine:

1. Whether the CHICO intervention decreases the number of paediatric formulation antibiotic suspension items dispensed for RTIs to children presenting with acute cough and RTI to primary care (superiority comparison).

2. Whether the CHICO intervention results in no change in hospital admissions for children with a hospital diagnosis of RTI (non-inferiority comparison).

Secondary objectives

To determine:

1. Whether the CHICO intervention results in no change in the accident and emergency (A&E) attendance rates of children with a hospital diagnosis of RTI (superiority comparison).

2. The costs to the NHS for using the CHICO intervention.

3. Whether any intervention effect is modified by the proportion of locums used.

4. Whether any intervention effect is modified by the practices’ prior antibiotic prescribing rate.

5. Whether any intervention effect differs between GP and nurse prescribers.

6. Whether any intervention effect differs between practices with 1 versus 2+ sites.

7. Whether any intervention effect differs within child age groups.

8. Whether any intervention effect differs between practices and over time.

9. Whether the embedded CHICO intervention is acceptable to, and used by, primary care clinicians (GPs and nurses) in consultations with carers and their children and how does this vary between practices.

Inclusion criteria

General practitioner practices in England using the EMIS^® electronic patient record system to house the intervention, where the local CCG has agreed to provide primary outcome data and the practice consented to take part. Children aged 0–9 years presenting with a cough or RTI to the participating practice were flagged for eligibility in CHICO via a pop-up on the EMIS^® consultation screen (for practices in the intervention arm of the study). No identifiable information was collected at participant level for the study. The number of times the intervention was used at practice level, over a 12-month period, was collected via an EMIS^® search.

Exclusion criteria

Practices were excluded if they were participating in any antimicrobial stewardship activities during the study period involving concurrent intervention studies, where there was a potential to confound or modify the effects of our intervention. Practices involved in the CHICO feasibility study or were merging or planning to merge with another practice were also excluded.

Children aged 10–14 were considered for inclusion but at this age an increasing number are given antibiotics in tablet form and given the much lower consultation rate in older children the research team decided to exclude this group from the study population.

Study population

The study population was children aged 0–9 years presenting with acute cough and RTI. The primary outcomes were antibiotic dispensing rates (amoxicillin and macrolide items) and hospitalisation rates for RTI over a 12-month period at each practice divided by the number of 0–9-year-olds registered at that practice.

Overview

The intervention consists of (1) eliciting explicit carer concerns during consultation, (2) a clinician-focused algorithm to predict risk of hospitalisation for children with acute cough and RTI in the following 30 days and (3) a carer-focused personalised printout recording decisions made at the consultation and safety-netting information. 22 The algorithm was to be used as one tool among many the clinicians already have, to decide whether the child needs antibiotics. In the training package for clinicians it was emphasised that the primary purpose of the algorithm was not so much to identify the small proportion of children (2%) at higher risk of hospitalisation but rather the much larger proportion of children (69%), who have a very low risk of hospitalisation.

Installation

Practices randomised to the intervention were sent instructions including screenshots on how to install the intervention application on the EMIS^® system. E-mail support was offered to the practice champion to help implement this and encourage appropriate use of the tool. Requests to access a practice EMIS^® system remotely to assist installation, subject to appropriate data security clearance, were also used if the installation ran into any problems.

Detail of intervention workflow during a consultation

• Step 1: Soft Pop-up. Small CHICO pops up when child aged 0–9 years presents; clinician may click to open CHICO input screen.

• Step 2: CHICO Trigger. Specific EMIS^® codes automatically trigger CHICO input screen.

• Step 3: CHICO Input screen: closed questions (clinical prediction rule) and elicitation of parent concerns.

• Step 4: CHICO Output. Step 4a – Additional hover text.

• Step 5: CHICO Letter + printout. Populate EMIS^® record, generate Microsoft Word document (via mail merge) for printing.

Once imported, the intervention was embedded within the primary care information system. The steps above outline how the intervention worked in practice. When a child was in the age range of 0–9 years, the healthcare professional received a soft pop-up on their screen asking if the child was presenting with RTI. The pop-up gave the option of opening the CHICO Intervention (see Appendix 2, Figures 23 and 24). The Intervention screen would also open if the healthcare professional input a RTI-specific EMIS^® code during the consultation. An embedded form was presented to record the presence or absence of particular symptoms and signs. Two of the seven predictors (age of patient and history of asthma) were already available for automatic entry. The other five predictors (short illness duration, temperature, intercostal or subcostal recession on examination, wheeze and any vomiting) were recorded during the consultation. Clinical details that are required to complete the STARWAVe algorithm were documented in a standardised EMIS^® proforma and the electronic medical record, then identified whether the consulting patient was at elevated, average or very low risk of hospitalisation using a 7-point scoring system. The health professional was also prompted to elicit explicit concerns of the carer. The output page appeared as a small pop-up on the screen (see Table 1), requiring no action from the healthcare professional but informed them of the child’s risk of hospitalisation.

TABLE 1 - Children with cough risk outputs

CHICO result	Pop-up text
Low-risk group	Very reassuring CHICO score: 0 or 1 CHICO predictors: > 99.6% of children will recover from this illness with home care. Consider a no- or delayed-antibiotic prescribing strategy. CHICO leaflet and letter covers common concerns and safety netting advice.
Average-risk group	Reassuring CHICO score: 2 or 3 CHICO predictors: > 98% of children will recover from this illness with home care. Consider no- or delayed-antibiotic prescribing strategy. CHICO leaflet and letter covers common concerns and safety-netting advice.
Elevated-risk group	Safety-netting needed: 4+ CHICO predictors: This is more than average, but > 87% of children will still recover from this illness with home care. Highlight SAFETY NETTING advice in CHICO leaflet.

When the Intervention input page had been saved the health professional had the option to print a short, personalised letter for the carer. This letter captured the following information:

1. patient’s name and age;

2. pre-specified brief description of the function of the consultation;

3. parent/carer’s concerns; manually typed in by the healthcare professional;

4. healthcare professional’s advice regarding raised parent’s concerns;

5. name of healthcare professional.

This personalised letter (project website upload) was provided to the carer alongside the ‘Caring for CHICO’ safety-netting leaflet (see Figure 2) providing further information regarding common carer concerns. The leaflet was based on one which was co-designed with parents and drew on our earlier work on the natural course of a typical RTI illness, typical duration of cough, caring for a child with a cough and safety-netting advice about when to re-consult. A slightly amended version was used in this intervention and made available in print or electronic format. 44

FIGURE 2.

Children with cough safety-netting leaflet.

Monitoring intervention use

An internal dashboard system was used for reporting frequency of use of the intervention by clinicians to practices during the 12-month participation period; where a ‘use’ of the intervention was defined as entering five or more symptom codes into the template as part of the consultation. Practices were provided with searches (EMIS^® queries) which practice champions were requested to run monthly and feed back to the study team.

Encouraging intervention use

To accomplish this, the monthly searches returned by the practice champions were checked by the study team and relayed back to the practice if the intervention had not been used. In addition, practice champions were to review the search results and remind clinicians to use the intervention. CCGs were asked to support the study at both CCG levels and to endorse the use of the intervention within practices using local engagement processes and Anti-Microbial Stewardship (AMS) activities. Working with the AMS teams already established among practices and CCG pharmacists we selected CCG AMS leads to promote the use of the intervention at practice level.

Fidelity of the intervention

The consistency and reliability (fidelity) measures focused on intervention exposure and the quality of the intervention delivered (using the process interviews as part of the qualitative investigation). Using our light-touch approach to data collection meant it was challenging to measure fidelity.

Capturing intervention usage

Monthly searches collected data on intervention usage, defined as 5+ symptom codes being entered into the CHICO template (see Appendix 2). There may have been consultations that were missed (false negatives) and some that were included with fewer than five symptoms (false positives) but these were difficult to quantify. In addition, in the follow-up questionnaire sent after 12 months participation in CHICO, there were questions about the use of the intervention for practices in the intervention arm of the study. Additional questions were added, concerning change in use during the pandemic, from April 2020 (project web upload).

Primary outcome measures

There were two primary outcomes:

1. The rate of amoxicillin and macrolide items dispensed, calculated using the number of items dispensed to 0–9-year-olds and the number of children aged 0–9 registered at each practice over a 12-month period (testing for superiority). The number of items dispensed, for all indications, was used as a proxy measure due to the limitations of routine data.

2. The rate of hospitalisations for RTI, calculated using the number of hospitalisations for RTI among children aged 0–9 years and the number of children aged 0–9 registered at each practice over a 12-month period (testing for non-inferiority).

Secondary outcome measures

1. A&E attendance rates for RTI, calculated using the number of A&E attendances for RTI among children aged 0–9 years and the number of children aged 0–9 registered at each practice over a 12-month period.

2. An exploration into the usage of the intervention, in terms of both usage over a 12-month period and seasonality, and the effects it has on primary outcome P1.

3. A between-arm comparison of mean NHS costs in a cost-consequence analysis (health economics).

4. Acceptability of the intervention and variation in use determined by qualitative interviews with the clinicians.

Subgroup analyses (potential effect modifiers for P1)

1. Comparison of primary outcome P1 stratified by categorisation of proportion of locums (median) used over the 12 months the practice is in the study.

2. Comparison of primary outcome P1 stratified by categorisation of proportion of nurses (median) used over the 12 months the practice is in the study. This was originally planned as a dichotomisation of practices into those with GP prescribers only and those with nurse or other prescribers as well. However, given that the majority of practices had nurse prescribers, it seemed sensible to look at nurses as a proportion of staff.

3. Comparison of primary outcome P1 stratified by categorisation of practice dispensing rates taken from the 12 months prior to the data collection from each practice.

4. Comparison of primary outcome P1 dichotomised by those practices with one site only and those with multiple sites.

5. Comparison of primary outcome P1 dichotomised by those practices with follow-up periods before the COVID-19 pandemic and those with follow-up months on or after March 2020.

6. Comparison of primary outcome P1 dichotomised by those practices with a high level of deprivation versus those with a low level of deprivation.

Expression of interest

As part of the recruitment of CCGs to the study, a form was completed by interested CCGs denoting their willingness to participate. One of our co-investigators (EB) is a national project lead for healthcare-acquired infections and antimicrobial resistance at NHS England and co-ordinated expressions of interest from CCGs prior to the start of the trial. Eligible practices were contacted via the CCGs and CRNs to take part in the trial. When GP practices were invited to participate, they were advised on the general principles of the trial, namely that the research will investigate methods to optimise the management of childhood RTI as well to agree to provide a practice champion as our primary contact and willingness of a member of staff to take part in qualitative interviews. If they were randomised to the intervention arm, the practices would need to agree to install the CHICO intervention on EMIS^® and for the practice champion to monitor its use and send monthly updates. If practices wished to take part in the study, they completed an expression of interest form confirming agreement.

Randomisation data

When CCGs agreed to take part in the CHICO trial, the latest 12 months of data available, for amoxicillin and macrolide prescribing, was collected for all practices in their area. The latest month of list size data was used as the denominator to calculate the rate of dispensing over the 12-month period. This was then used to divide the data into those with a HIGH or LOW dispensing rate for stratification purposes in the randomisation process. In a similar way, practices within the CCG were split into those with a HIGH or LOW list size, relative to other practices within their CCG. These HIGH/LOW categorisations were not shared with the trial team until a practice agreed to take part and were ready to be randomised.

Baseline questionnaire

A brief questionnaire was sent to each practice to collect data on the characteristics of all practices prior to randomisation. Data were collected on the numbers of staff at the practice (by job type), questions regarding triaging of RTI conditions and postcode (for calculating the Index of Multiple Deprivation).

Routine data

For each practice, baseline data and follow-up data after 12 months were collected along with data for the two primary outcomes (dispensing and hospitalisations) and the secondary outcome (A&E attendances). The items of amoxicillin and macrolides dispensed, each month, were collected for the 12 months prior to randomisation and the 12 months after the implementation period. A short ‘implementation period’ (usually around 1 month) was allowed prior to the 12-month data collection period, giving time for the practices to install the intervention and encourage staff to use it. Any data collected during this period were not used in the analysis. For example, if a practice was randomised on the 10th of January their typical month 1 of follow-up would be February, as routine data were available for each calendar month. The data were obtained from NHSBSA ePACT2 system. 30

For the same 24-month period, data were collected on hospitalisations and A&E attendances, via a secure NHS e-mail inbox. These data were collected directly from each CCG. Data analysts from each CCG were asked to provide the number of hospitalisations for RTI, the number of hospitalisations with a ‘missing’ diagnosis and the total number of hospitalisations. Similar data were collected on A&E attendances. List size data, per month and 5-year epoch, were obtained from the NHS digital website. 45

Intervention usage

Practices were provided with an EMIS^® search, which looked at the number of times the intervention template was opened. This was collected for each of the 12 months the intervention practices were in the study. Qualitative interviews with clinicians (GPs and practice nurses) and other practice staff (managers, pharmacists) and CCG staff (medicines managers) explored the use of the intervention, how it was embedded into practice and whether it was used appropriately.

Follow-up questionnaire

Like the baseline questionnaire, practices were asked to complete a follow-up questionnaire 12 months later. This included additional data on items such as the number of locums used during the 12 months of follow-up as well as questions around COVID-19 and how it had affected their triaging. Intervention practices were also asked what they thought of the intervention and if they would want to use it again.

Safety data (serious adverse events)

Intervention and control practices were asked to report SAEs immediately on the practice becoming aware of the event. The CHICO team sent a reminder e-mail, quarterly during follow-up, to remind practices to report SAEs. An additional reminder was sent 3 months after the practice had completed their time in the study. As the trial was collecting data on A&E attendances and hospitalisations routinely, only the following events required reporting.

Fidelity data

The fidelity measures focused on intervention exposure and the quality of the intervention delivered (using the process interviews as part of the qualitative investigation).

Summary of baseline data and flow of participants

A Consolidated Standards of Reporting Trials (CONSORT) diagram was used to present the reasons why CCGs and practices were not recruited (due to ineligibility or declining). It was also used to present the number of withdrawals and the number used in the analysis, by arm.

Descriptive statistics were used to summarise characteristics of practices and compare them between groups in a baseline table. Categorical variables were summarised using frequencies and proportions while continuous variables were summarised using medians and interquartile ranges (IQRs; given their skewed nature). Had any differences, between the groups, been in excess of 0.5 standard deviations (SDs)/IQRs or 10% or more they would have been controlled for in sensitivity analyses. However, this was not the case.

Primary outcome analysis

The co-primary outcomes were the rate of amoxicillin and macrolide items (antibiotics) dispensed, for all indications, and the rate of hospitalisations for RTI. The denominator for each of these outcomes was the mean number of children aged 0–9 years registered at the practice over the 12-month follow-up period. In the analysis plan, the team pre-specified that the median list size would be used; however, given the number of practices that merged it seemed more accurate to calculate the mean. All analyses were carried out in Stata 17.0 and the results were described in terms of ‘strength of evidence’ rather than significance. No formal methods of imputation were carried out as the primary outcome was obtainable for over 99% of practices.

Mixed models were used to account for the within and between CCG level variation, incorporating the latter as a random effect. A random-effects Poisson regression model was used to analyse both of these co-primary outcomes. This has the advantage of incorporating person/years follow-up (number of children at a practice) and examining clustering by practice. The dispensing record of the practices in the 12 months prior to randomisation was used as a minimisation variable and thus balance the dispensing records at baseline. This was adjusted for, in the primary analysis of dispensing rates, to resolve any residual difference. While the dispensing outcome will be testing for superiority, the hospitalisation outcome will be testing for non-inferiority where a difference of no more than 1% higher in the intervention arm is reasonable to suggest non-inferiority. Therefore, emphasis was placed on the CI when assessing this outcome.

Secondary outcome analysis

The rate of A&E attendances for RTI was calculated in the same way as the rate of hospitalisations, although it was a test for superiority rather than non-inferiority.

Intervention usage was explored across the seasons and across the 12 months of follow-up. Its impact on dispensing was explored and the correlation was presented using scatter plots and Pearson’s correlation coefficient.

Sensitivity analyses

Subgroup analyses

Subgroup analyses were carried out for dispensing rates (P1). Formal tests of interaction between the potential effect modifiers and treatment pathway were carried out to test whether the treatment effect differed between the different subgroups. These effect modifiers are listed in Subgroup analyses (potential effect modifiers for P1). These variables were dichotomised to aid presentation and interpretation, taking the median value where the variable was continuous. Models with and without an interaction term were compared using a likelihood ratio test and this p-value was presented.

Stop/go assessment

There were four stop/go criteria (see Table 2).

Our progression criteria at the pilot stage were based on:

1. the percentage of practices recruited against the initial practice target of 60;

2. the percentage of GP practices with a named practice champion;

3. the percentage of intervention GPs/nurses using the intervention at least once;

4. the percentage of antibiotic dispensing data we can obtain for each practice.

Internal pilot areas of assessment

In addition to stop/go criteria, the internal pilot was used to assess areas where we could improve recruitment, communication with practice champions and use of the intervention. This included:

1. the number of eligible practices within the 20 CCGs already approached and whether at this stage we needed to approach further CCGs to increase the number of practices in the study;

2. the recruitment rate of eligible practices within the four CCGs in the pilot phase, again as an indicator of whether we need to approach further CCGs;

3. the recruitment of practice champions, a focus on their role and how we maximise strategies to encourage use of the intervention;

4. the efficiency of embedding the intervention in practice systems and resolution of any barriers or delays;

5. the number of times the intervention is used between practices and over time;

6. the timeliness of the primary outcome data and consistency of format between CCGs;

7. the timeliness of the secondary outcome data and consistency of format between CCGs.

Users of the intervention

Using the EMIS^® user mnemonic (a unique username for each healthcare practitioner using system), the CHICO template recorded 1399 individual users of the intervention. The median number of users per practice (n = 131) was 9 (IQR 3–16). This figure remained the same when looking at practices providing 12 months of data (n = 115); median 9 (IQR 3–17). Figure 6 shows the job roles of the intervention users. Of the 1399 users identified, the job role type was available for 1341 of them. Of these, 994 (74%) were GPs, 187 (14%) were nurses, 77 (6%) were office staff, 40 (3%) were clinicians, 34 (3%) were locum GPs and 7 (1%) were pharmacists. There was also one consultant and one phlebotomist.

FIGURE 6.

Job roles of those using the intervention.

There were four individual users who used the intervention more than 100 times; two GPs, one nurse and one clinician. Overall, the GPs and nurses used the intervention the most, with a median of five uses per GP/nurse over the 12-month period (see Figure 7). Office staff (e.g. analysts and medical secretaries) used it the least, averaging one use per member of staff.

FIGURE 7.

Uses of the intervention, per job role.

Usage of the intervention over the 12 months of participation in the study

Practices with 12 months of intervention usage data available allowed for a detailed inspection of trends over months 1–12 of follow-up. Panel A of Figure 8 shows usage over the 12-month period for all 115 practices. Due to COVID-19, it is difficult to determine what a typical pattern of intervention usage may look like. Table 9 provides the median use per month based on the individual practices presented in Figure 8. Panel B of Figure 8 and row 2 of Table 9 show the data for the 29 practices that completed follow-up prior to March 2020 and were, therefore, unaffected by COVID-19 lockdowns. Generally, the majority of use was at the start of follow-up, in months 1 (median = 22) and 2 (median = 18). However, for these 29 practices who completed follow-up prior to the pandemic, month 1 had to be between November 2018 and March 2019. Therefore, the increased usage in months 1 and 2 may coincide with the seasons, rather than indicate an increased usage at the start of follow-up. Panel C shows the practices that had 1–11 months of follow-up data on, or after, March 2020 (n = 57). Given the variety of months affected by COVID-19, it is difficult to spot trends or patterns in the intervention usage data.

FIGURE 8.

Intervention usage, by follow-up month for (a) all practices, (b) pre-COVID-19 (completed before March 2020), (c) partial COVID-19 (≥ 1 month from March 2020), (d) all COVID-19 (complete follow-up from March 2020).

For practices with complete follow-up on or after March 2020 (n = 29), the median usage was 0 for all months and panel D of Figure 8 shows how the maximum usage recorded over that period was 20.

Usage of the intervention across the seasons

If we combine all practices and look at the data across calendar months, we can see the seasonal effects on intervention usage. In Figure 9, we can see the seasonal variation between October 2018 and February 2020, with increased usage during the winter months and fewer uses during the summer months. This suggests that usage did correlate with the number of children presenting with RTIs. However, from March 2020 the effects of COVID-19 lockdowns are clear, with very little usage of the intervention. There is a small increase in May, June and July 2021, when restrictions were relaxed and RTIs were more prevalent in the population.

FIGURE 9.

Intervention usage over time, with each practice (n = 115) contributing 12 months of data.

Looking at usage per calendar month, with all years combined, the seasonal variation is clear (see Figure 10 and Table 10). Looking specifically at practices that completed follow-up before COVID-19 (n = 29), median usage was highest in November (19), December (14) and January (15). This was also evident in practices with at least 1 month on, or after, March 2020 (n = 57). For practices with all 12 months of follow-up data on, or after March 2020 (n = 29), median usage bucked the usual seasonal trends, with summer months providing the highest number of uses.

FIGURE 10.

Intervention usage, by calendar month for (a) all practices, (b) pre-COVID-19 (completed before March 2020), (c) partial COVID-19 (≥ 1 month from March 2020), (d) all COVID-19 (complete follow-up from March 2020).

Compliance with the intervention

The definition of compliance was pre-defined in the statistical analysis plan as the number of uses of the intervention divided by the list size of 0–9-year-olds. Figure 11 shows the relationship between these two variables. For the per-protocol analysis, the CHICO trial team pre-specified that a practice would be compliant if the intervention was used in at least 5% of children (0.05*list size). However, this definition may be impacted by COVID-19. The green line indicates the boundary line for practices considered compliant (dots on or above the line).

FIGURE 11.

Assessing compliance with the intervention.

We had planned to look at the proportion of prescribing staff that were using the intervention at least once. As described previously (see Baseline data), the reporting of the number of staff at each practice was quite poor.

Plotting the number of staff (x-axis) versus the number of individual users of the intervention (y-axis), it is clear that, in many cases, the number of users exceeds the number of prescribing staff at the practice (see Figure 12). Given the underreporting, the ‘number of staff’ figures were not utilised when looking at compliance with the intervention.

FIGURE 12.

Number of individual users compared to the number of prescribing staff.

Relationship between compliance and change in dispensing

Figure 13 looks at the relationship between the level of compliance (annual intervention usage/list size) and the change in dispensing (follow-up rate – baseline rate). Panel A shows the relationship for all intervention practices with compliance data (n = 129).

FIGURE 13.

Correlation between compliance and change in dispensing in (a) all intervention practices with compliance data and (b) restricted to those with follow-up prior to COVID-19.

The line of best fit suggested that increased compliance led to increased antibiotic usage; Pearson’s correlation coefficient 0.241, p = 0.006. However, as described in Usage of the intervention across the seasons, practices that had follow-up during COVID-19 often reported 0 compliance, yet had a decrease in dispensing due to lockdowns. Therefore, panel B is perhaps a better representation of the relationship between intervention usage and change in dispensing rate. In this panel, the focus is on those practices that completed follow-up prior to COVID-19, n = 31. The relationship here is in the opposite direction, with increased intervention usage leading to a decrease in dispensing; Pearson’s correlation coefficient −0.326, p = 0.074. After removing the outlier (compliance = 0.42), the correlation observed was weakened to −0.222, p = 0.238.

Impact of COVID-19 on dispensing

Figure 14 combines all baseline and follow-up data for all practices, in both arms, and shows the seasonal effects on dispensing. Between October 2017 and February 2020, the dispensing rates reached highs of over 0.35 in the winter and approximately 0.10 in the summer. The effects of COVID-19, and its associated lockdowns, on dispensing are clear from March–July 2020 when it dropped to 0.05. When restrictions were relaxed in late spring/summer of 2021, there was an unseasonably high dispensing rate. Focusing specifically on the control practices (reflecting usual care) comparing the 12 months (March 2019–February 2020) prior to the pandemic to the first 12 months (March 2020–February 2021) during the pandemic (taking account of seasonal variability), the dispensing rates halved from 20.55 items per 100 children (95% CI 19.64 to 21.46) to 9.84 items per 100 children (95% CI 9.18 to 10.50). A similar reduction was observed in the intervention arm.

FIGURE 14.

Mean dispensing rate (95% CI) during the CHICO baseline and follow-up periods. Each practice contributed 24 months of follow-up data.

In the sample size calculation, for the dispensing rate outcome, the team had hypothesised a 4% absolute rate reduction, from 0.33 to 0.29 (equivalent to 4 prescriptions per 100 patients, per year). Given the low dispensing rates during COVID-19, it would have been difficult to distinguish any differences between the arms of the trial between March 2020 and April 2021.

Sensitivity analyses for the dispensing primary outcome

There were six pre-specified sensitivity analyses for the dispensing rate primary outcome (see Table 12) and five post hoc analyses that were added during follow-up data collection or after presenting the results to the team. Firstly, a pre-specified per-protocol analysis was carried out that excluded ‘non-compliant’ practices from the intervention arm. Practices were considered non-compliant if the number of uses of the intervention divided by the list size of 0–9-year-olds was less than 5% (n = 50). Intervention practices were also excluded where the intervention usage was unknown (n = 15) or where an intervention practice merged with another practice that had previously taken part in the control arm (n = 1). In total, there were 66 exclusions from this analysis leaving 78 intervention practices to compare with the 149 in the control arm.

The per-protocol analysis produced strong evidence against the null hypothesis of equal dispensing in both arms of the study, with a higher rate of dispensing in the intervention arm (restricted to compliers). However, when investigated further, a lot of the practices that were non-compliant had joined in the latter half of the study, when COVID-19 had lowered all dispensing rates Figure 16. Therefore, this sensitivity analysis was not considered to be suitable as it falsely inflates the intervention arm dispensing rate as equivalent practices in the control arm were still included.

FIGURE 16.

Baseline and follow-up dispensing rates across the intervention arm compliers and non-compliers. Marker labels indicate the number of months on or after March 2020 (COVID-19).

Therefore, a post hoc CACE analysis was included to allow for compliance to be incorporated but avoid bias caused by only removing practices from one arm of the study. This analysis provided a similar point estimate as the per protocol but with large CIs and therefore no evidence of a difference.

A pre-specified sensitivity analysis had attempted to account for COVID-19 effects, by including a variable in the model which provided the number of years of follow-up that were on, or after, March 2020 (1–12). This analysis produced very similar results as the primary outcome. A post hoc sensitivity analysis was carried out which only utilised follow-up month data prior to COVID-19 and excluded follow-up months after February 2020. The exposure (list size) was then scaled down to the number of months of available data. This provided evidence of a difference, in favour of the intervention [0.192 vs. 0.204, RR = 0.967 (95% CI 0.946 to 0.989). p = 0.003]; assuming 1000 children (0–9-year-olds) per practice this equates to 192 items (intervention) compared to 204 items (controls) per practice or a 5.9% reduction. However, interpretation of this finding is limited given the lack of power and the post hoc nature of this observation.

As previously described in Internal pilot review, there were 48 practices that were recruited as part of a CHICO internal pilot. These practices were removed in a sensitivity analysis as the intervention was adapted for the main trial. Excluding these practices provided an IRR of 1.026 (95% CI 1.005 to 1.048), which provided weak evidence to suggest that the intervention may have led to higher dispensing rates.

Another sensitivity analysis incorporated dispensed amoxicillin/macrolide items with ‘missing’ age. This produced equally higher rates of dispensing in both arms, providing the same conclusion as the primary analysis. We felt that a post hoc analysis may have helped to distinguish whether there was any difference in amoxicillin dispensing only. This analysis did provide a slightly lower dispensing rate in the intervention arm but there was no statistical evidence of a difference.

A difference was seen between the arms when splitting the primary outcome into two separate epochs; 0–4 and 5–9. In the 0–4 group, there was evidence to suggest an increase in dispensing in the intervention arm. Conversely, in the 5–9-year-old group, there was evidence to suggest a decrease in dispensing rate in the intervention arm. This may be because clinical staff, using the intervention, had more confidence when using the intervention in the older age group but this is conjecture.

Another sensitivity analysis replaced the CCG random effects with PCN random effects. The 293 practices were part of 174 unique PCNs, with between 1 and 6 practices in each. Adding PCN as a random effect gave an IRR of 0.942 (95% CI 0.916 to 0.969), providing evidence of lower dispensing in the intervention practices, compared with controls.

The final sensitivity analysis included a covariate to indicate the number of months between baseline and follow-up (implementation period). This was 1 month for 253/293 (86%) of practices, 2 months for 14 (5%) of practices (10 intervention: 4 control) and between 3 and 21 months for the remaining 26 (9%) practices (all intervention). This adjusted analysis provided evidence of a difference, in favour of the control arm.

Subgroup analyses for the dispensing primary outcome

There were six pre-specified subgroup analyses for the dispensing primary outcome,46 which utilised practice-level data to split the data into two subgroups and look for an interaction between the subgroup and trial arm effects, on dispensing rates (see Table 13).

In Table 13, when assessing the impact of the proportion of locums at the practice, there was some evidence to suggest that practices with more locums had a treatment effect in favour of the intervention. There was strong evidence of an interaction between trial arm and proportion of nurses. For those practices with a low proportion of nurses (< 17% of staff), there was a 0.15 dispensing rate in the intervention arm and a 0.17 dispensing rate in the control arm. In practices with a higher proportion of nurses (≥ 17%) the opposite was true; 0.16 and 0.14, respectively. This suggests that the intervention was successful at decreasing dispensing rates in practices with a lower proportion of nurses. Equally, it suggested that the intervention increased the dispensing rate in practices with a high proportion of nurses. A similar interaction was found when looking at the number of sites a practice has. In those with more than one site, the intervention had a negative impact on dispensing, suggesting that the intervention training may be difficult to implement across multiple sites.

There was also some evidence to suggest that the intervention was more successful in practices with lower levels of deprivation and had a negative impact on those that had high levels of deprivation.

There was no evidence of an interaction when looking at past dispensing habits or separating practices into those affected/not affected by COVID-19.

Impact of COVID-19 on hospitalisations for respiratory tract infection

In a similar way to dispensing rates, there was seasonal variation in hospitalisation for RTI across all CHICO practices (see Figure 17). Between October 2017 and March 2020, the hospitalisation rates, for RTI, reached highs of nearly 0.06 in the winter and approximately 0.02 in the summer. The effects of COVID-19, and its associated lockdowns, on hospitalisations are clear from April 2020 onwards when it dropped to < 0.01. When restrictions were relaxed in late spring/summer of 2021, hospitalisations climbed back towards normal levels; apart from July 2021 where they were unseasonably high.

FIGURE 17.

Mean hospitalisation rate (95% CI) during the CHICO baseline and follow-up periods. Each practice contributed 24 months of data.

Hospitalisation rate comparison

Hospitalisation rates appeared to be very similar between the arms, over the 3-year follow-up period (see Figure 18).

FIGURE 18.

Mean hospitalisation rate (95% CI) during the CHICO follow-up period (January 2019–August 2021), by arm. October 2018–December 2018 and September 2021 have been excluded due to low numbers. Each practice contributed up to 12 months of follow-up data.

We pre-specified a non-inferiority margin of 0.01 for this outcome. Therefore, when comparing intervention with the control the IRR upper CI needed to be less than 1.01 to conclude non-inferiority. The rate of hospitalisations were 0.019 and 0.021 for the intervention and control arms, respectively (see Table 14). The IRR was 0.952 (95% CI 0.905, 1.003). As 1.003 lies below the 1.01 non-inferiority margin the intervention was considered non-inferior to the control.

Three pre-specified sensitivity analyses were carried out to test the robustness of the non-inferiority result. As West Hampshire did not provide accurate annual figures, their data were removed in a post hoc sensitivity analysis. First, a proportion of the hospitalisations with a missing diagnosis were included as RTI diagnoses. This provided very similar results to the primary outcome; IRR 0.950 (95% CI 0.903 to 1.000). Second, all hospitalisations with a missing diagnosis were included as RTI diagnoses, akin to a worst-case scenario. While there were still more hospitalisations in the control arm, the CI for this sensitivity analysis was (0.920, 1.016), slightly above the non-inferiority margin. Including a variable from 1 to 12 months, as with the dispensing outcome, to account for the number of months affected by COVID-19 provided results that agreed with the non-inferiority conclusion. We observed the same effect following the removal of West Hampshire practices.

Impact of COVID-19 on accident and emergency attendances for respiratory tract infection

In a similar way to dispensing and hospitalisations, there was obvious seasonal variation in A&E attendances for RTI across all CHICO practices (see Figure 19). Between October 2017 and March 2020, the A&E attendance rates, for RTI, reached highs of nearly 13% in the winter and approximately 4% in the summer. The effects of COVID-19, and its associated lockdowns, on A&E attendances is clear from April 2020 onwards when it dropped to 2%. When restrictions were relaxed in late spring/summer of 2021, they became unseasonably high.

FIGURE 19.

Mean A&E attendance rate (95% CI) during the CHICO baseline and follow-up periods. Each practice contributed 24 months of data.

Accident and emergency attendance rate comparison

As with the dispensing and hospitalisation rates the A&E attendance rates appeared to be very similar between the arms, over the 3-year follow-up period (see Figure 20).

FIGURE 20.

Mean A&E attendance rate (95% CI) during the CHICO follow-up period (January 2019–August 2021), by arm. October 2018–December 2018 and September 2021 have been excluded due to low numbers. Each practice contributed up to 12 months of follow-up data.

The rate of A&E attendances was 0.049 and 0.045 for the intervention and control arms, respectively (see Table 15). The IRR was 1.013 (95% CI 0.980, 1.047; p = 0.437), providing no evidence of a difference between the groups.

As with the hospitalisation outcome, various sensitivity analyses were carried out to account for the various misgivings in the data. As with hospitalisations, there were A&E attendances with missing diagnoses. Including a proportion of these as RTI-related gave very similar results to the main secondary outcome; however including all of them led to a higher number of A&E attendances in the intervention arm. This was true for both hospitalisations and A&E attendances suggesting that there may have been a higher proportion of ‘missing’ diagnoses in the intervention arm, compared with the control. Including a variable from 1 to 12, to account for the number of months affected by COVID-19, provided results that agreed with the secondary outcomes results. As did the removal of West Hampshire practices.

Recruitment and sampling

Clinicians involved in implementing the CHICO intervention were invited to take part in semistructured interviews. To capture maximum variation in views and experience and reflect a range of practices (those with high and low patient lists and antibiotic prescribing rates, serving areas of high and low socioeconomic deprivation) and clinicians (GPs and practice nurses with a range of years’ experience) were purposively sampled. The socioeconomic status of practices was estimated using the Index of Multiple Deprivation decile47 based on the practice post code. The sample size was determined by data saturation, such that no new themes were emerging from the data by the end of data collection. 50 Potential participants were contacted by the qualitative researcher (CLCL) via e-mail and invited to take part in the interview. Clinicians from 56 practices were invited to participate. Participants were asked to provide their audio-recorded verbal consent to take part immediately before the interview.

Interviews

The interviews were all conducted by telephone by the qualitative researcher (CLCL), who is an experienced social scientist. The first set of interviews was conducted during the trial pilot phase and findings were fed back to the Trial Management Group (TMG) to optimise the intervention implementation and guided changes during the remainder of the study. The second phase of interviews was conducted after practices had been using the intervention for 12 months. A flexible topic guide was used to assist questioning during interviews but allow participants to introduce and discuss topics not anticipated by the researchers (see Box 1).

Topic guides were modified as necessary throughout the course of the study to reflect findings as they emerged. With informed consent from participants, interviews were audio recorded using a digital voice recorder, transcribed using a professional transcription service and anonymised to protect confidentiality. Interviews lasted between 15 and 37 minutes, with an average time of 25 minutes.

Analysis of the interviews

Anonymised transcripts were checked for accuracy and then imported into NVivo Pro (version 10/11)51 data analysis software to aid the management and analysis of the data. Analysis began shortly after data collection started and was ongoing and iterative. Ongoing analysis informed further data collection; for instance, analytic insights from data gathered in earlier interviews helped identify any changes that needed to be made to the topic guide during later interviews, for example, to explore new or underexplored topics raised in earlier interviews. Thematic analysis52 was used to scrutinise the data to identify and analyse patterns and themes of salience for participants across and within the data set. Transcripts were coded inductively to establish an initial analysis framework and three researchers (CLCL, JH and CC) coded a subset of three transcripts; any discrepancies were discussed to ensure a coding consensus and maximise rigour. 53 The remaining transcripts were analysed (by CLCL) using the agreed analysis framework. The four NPT constructs were used to further develop themes across the data sets. To ensure that findings were trustworthy and credible, preliminary findings were discussed with the multidisciplinary trial management team.

Participants

Twenty-six clinicians (20 GPs and 6 practice nurses) were interviewed from across 24 practices and 13 CCGs (see Table 16). Findings are presented for each of the NPT constructs, illustrated with anonymised verbatim quotations.

Coherence (sense making)

Clinicians welcomed the CHICO intervention as it aimed to help with ongoing issues being faced by practices such as the prevalence of children presenting with a cough and perceived parent concerns with not receiving antibiotics.

It was something that we were particularly interested in doing anyway. We do see lots of children, obviously, with coughs and colds and some parents generally are concerned. Some parents do understand that infections can be viral, but some do also expect an antibiotic if it’s had a chesty cough for a certain period of time.

(GP 14)

Just a couple of weeks ago before CHICO was installed, they [nurses] were having numerous complaints from parents saying they felt cheated out of antibiotics.

(GP 5)

The intervention also aligned with existing strategies and efforts to reduce unnecessary antibiotic prescribing. Clinicians were already familiar with similar protocols and templates commonly used in primary care and believed the intervention would fit within their usual practice.

I thought it was good using that kind of direction that we were moving anyway to prescribe less antibiotics. So it was just something that was in time and in tune with what we needed to do.

(GP 21)

It seemed like it would be a simple intervention to use that could be used, you know just within a consultation so we always like that … It’s almost part of what we would normally do.

(GP 11)

Cognitive participation (buy-in)

Clinicians felt they were well prepared for using the intervention and liked the training and materials. They found the training guides useful to refer to throughout the study and liked the ability to practice using a training patient. Although a website that held background information on the study and published literature on the intervention development was available, most clinicians did not feel the need to visit it.

I think [we were] really well prepared. The training results are really good but having a test patient was really good.

(GP 11)

The leaflets that you gave out and the information and the CHICO guides were given, were quite self-explanatory and all seemed quite straightforward.

(Nurse 17)

Collective action (using children with cough in practice)

Most clinicians liked the intervention and thought it was quick and easy to use and used it as a supportive aid within consultations and described it as a safety net. It was a way of reassuring themselves and parents of the appropriateness of some treatment decisions.

I thought it was good, it was easy to use and yes, I thought it was a good idea.

(Nurse 20)

I think if you are not prescribing antibiotics, then it’s a good safety netting tool. So, you can confidently say to the patient that you don’t feel antibiotics are indicated.

(GP 19)

It’s very reassuring for the professional and of course when you’ve printed out the leaflet that this is the scoring we have done; it is very reassuring for parents as well.

(GP 16)

Reflexive monitoring (appraisal of CHICO)

When appraising the CHICO intervention and making recommendations for future implementation, participants suggested expanding the template to encompass more information. This could help overcome issues with having to record information in multiple places and switching screens.

Maybe a box, history of presenting complaints, so if there was other information – because I think if you’re filling in a template, especially when we’re busy in the winter, it be good if we could record all the information in that template and then not have to go back into the notes to record things that we think is important to record.

(GP 19)

But if you’re using a template, you want it to capture all the information. So, adding those two, heart rate, respiratory rate and oxygen saturations would help it to be more complete.

(GP 13)

Clinicians also recommended adapting the intervention to be more conducive to remote consultations. This could include informing parents about how to assess symptoms and having parent-reported criteria rather than having the clinician-assessed criteria. However, some clinicians worried about relying on parent-reported symptoms as these could be less accurate and more subjective.

Adapting the template to remote consultation will be a good thing … have parent observed criteria rather than clinician observed.

(GP 23)

I think the tool is little bit reliant on clinical aspect as well which you may not have so it’s difficult to judge on chest signs and symptoms and respiratory distress and that sort of thing, wheeze, based on a conversation with a parent and even temperature. They may not have a temperature probe so you may not be able to get particular aspects of it but then some bits you will be able to get. But if it can be tweaked to amend for things that may not happen on remote working then that may obviously help.

(GP 25)

And asking if they would use the intervention in the future

I would have no problem with starting to use it again now because I feel you know, now we’re gonna start getting back to normal and coughs will just be coughs and colds and it would be really useful to have that back again.

(Nurse 24)

Summary of the findings and implications

The use of NPT allowed for examination of issues with both the design of the intervention and its implementation. Interview data indicated that while the NPT constructs of coherence and cognitive participation were fulfilled, this was not the case for cognitive participation, with negative consequences for implementation.

The qualitative findings help explain why no difference was found between the control and intervention groups. They confirmed that clinicians used the intervention at the start of the trial, but usage waned over time. Clinicians initially welcomed CHICO in theory but in practice, it proved difficult to align the intervention flow with that of the clinician’s usual consultation practice. Most clinicians liked the algorithm template and found it straightforward to use, without adding any more time to consultations. However, having to close the patient’s record before the end of the consultation to complete the intervention process did not always align with clinicians’ usual processes and was problematic. The COVID-19 pandemic also impacted use of the intervention due to changes to practice pathways, increased use of remote consultations and reduced numbers of children presenting with RTIs.

While some clinicians reported that the intervention influenced prescribing decisions and found it most useful in ‘borderline cases’, others reported that they used it as a supportive aid during consultations rather than a tool to change prescribing behaviour. CHICO helped elicit parent concerns and reassure clinicians and parents of the appropriateness of some treatment decisions. Clinicians particularly liked the safety-netting parental advice leaflet, as it helped explain treatment decisions and home care with parents and this was seen to be the most useful intervention component. To increase use of the intervention, findings suggest it may need to be adapted for use within remote consultations and to fit better with clinicians’ consultation flow.

Strengths and limitations of the qualitative study

Strengths include interviewing both GPs and nurses who used the intervention from a diverse range of practices. The use of NPT to inform data collection and analysis enabled a focus on issues with both the intervention design and the way it was implemented in practice.

Limitations include possible memory bias as most clinicians were interviewed towards the end of the trial, which may have been some time after they had last used the intervention. In addition, we did not interview those clinicians who have never used the tool, and this could have provided useful insights into the barriers of using the tool. Insight into whether and when clinicians would choose to use or ignore the tool is also missing, which limits understanding of its use in different situations. A further limitation that parents of children that CHICO was used with were not interviewed which should be taken into consideration when interpreting the results. However, clinicians did discuss how the intervention was received by parents, which provided some useful insight into parents’ views.

As we were examining the experience of using the intervention, we did not interview people who did not use the intervention. However, as interviews were conducted with those who used the intervention early on but then their use reduced over time, we have provided insight into why clinicians may not engage with the intervention.

Conclusions

Clinicians found the CHICO intervention useful, and it helped discussions with parents about concerns and treatment decisions. Some clinicians believed the intervention influenced their prescribing behaviour and found it most useful in ‘borderline cases’. However, others believed that it supported rather than changed their prescribing decisions and did not change their prescribing behaviour. Clinicians usage of the intervention reduced over time and this was further impacted by the COVID-19 pandemic and subsequent increase in remote consultations and reduction in RTIs.

To increase use of the intervention, findings suggest it may need to be adapted to align more with clinician’s consultation flow and allow use during remote consultations.

Costs of the intervention

The costs of the intervention comprised the non-research related costs involved at the practice level that arose from integrating the intervention into local computers, and training costs borne at the practice level. We consider possible other intervention-related costs that may arise in the event of a large-scale (e.g. national) deployment in Chapter 7 Discussion, but for the between-arm cost analysis we restricted our focus to those costs incurred in the trial itself.

To reflect experiences at some but not all practices in the trial, we made the conservative assumption that time will be required to manage and troubleshoot the integration of the algorithm at the practice level. We assumed that this would be undertaken by a practice manager working for 1 hour in total. This amounts to a cost of £54 per practice in 2020/21 prices, corresponding to a salary of Band 6 on the Agenda for Change (AFC) pay scale. 55

For training time involved in using the intervention, we assumed that it would take some time for prescribing clinicians to understand the rationale for the intervention and to become familiar with its use. This is also a conservative assumption since the ‘implementation’ period of the trial allowed time for practices to install the intervention and to encourage its use by staff. We assumed that the time involved would be equivalent to that of a single consultation. The median number of GP prescribers of amoxicillin and macrolides per practice in the trial was 4. Given a cost per consultation of £39 in 2020/21 prices, we therefore calculated practice-level training costs as 4*£39 = £156.

The costs of the intervention were therefore estimated as £210 per practice. In sensitivity analysis, to test our conservative assumptions on GP training and manager integration of the intervention, we re-ran our main results attributing no costs to the intervention itself.

Costs of amoxicillin and macrolides

Data on both the volume and cost of prescriptions at each practice were taken from the BNFePACT2 (Electronic Prescribing Analysis and Cost) system. The volume data were specific to each practice, collected routinely every month and reflected dispensed amoxicillin and macrolide prescriptions given to children aged 0–9 years, for each practice, and obtained from ePACT2.

Data on the net ingredient cost of amoxicillin and macrolides (see Appendix 6, Table 20) were also obtained from the ePACT system. The cost data were not specific to each practice. Instead, they represented cost of all amoxicillin and macrolides dispensed to this age group in England during the 2020/21 financial year. Using these cost data, therefore, assumes that the costs and composition of prescriptions – reflecting the relative mix of specific medications as well as their formulation, dose and presentation – of all GP practices in England are similar to those of practices included in the trial.

The use of net ingredient cost reflects a basic cost of the drug concerned. Given the unavailability of details on formulation, presentation, dose and other information, it does not account for container costs, dispensing costs and net fees or net prescription charges income.

Costs of accident and emergency attendances

As noted in Chapter 3, the collection of data on A&E attendances is routinely undertaken each month by each CCG. We used these data to identify the volume of attendances. These were costed using NHS Reference Costs 2019/2056 and inflated to 2020/21 prices. The inflation index is used as the average of the last 5 years (2016/17–2020/21 financial years) of the NHS cost inflation index. 55

We did not have information on whether an attendance led to a subsequent admission, the type of investigation conducted or the nature of treatment received while attending. Reference costs also does not distinguish between attendances related to paediatric or adult populations. We, therefore, created an average unit cost, weighted by the number of attendances reported for all admissions. However, we excluded dental admissions, and those where the patient was reported as dead on arrival. The unit cost in 2020/21 prices was £186.

Costs of hospital admissions

Volumes of per-practice hospital admissions related to RTI were also obtained from data routinely collected by CCGs. These were also costed using NHS Reference Costs 2019/20, inflated to 2020/21 prices as for A&E attendance using NHS cost inflation index as described above. 54

The data available to the trial indicated that an admission had taken place, but did not otherwise identify the specific diagnosis associated with an admission nor whether the admission was associated with non-elective long or short stay, or day care. We therefore costed admissions as follows. Within each of non-elective long-stay, non-elective short-stay and day care episodes, we calculated an average of all unit costs that relate to acute respiratory symptoms. We weighted each unit cost by the respective volume of finished consultant episodes. Finished consultant episodes are completed episodes of treatment received by a patient under the care of a single consultant, and are a fundamental way of tracking activity in the NHS. These unit costs were then weighted by the total finished consultant episodes of each of non-elective long-stay care, non-elective short-stay care and day care to give a total average unit cost for LRTI-related hospital admissions.

Clinical Commissioning Groups were asked to identify hospital admissions for children for a broad list of conditions that may have been associated with RTIs. We developed estimates of unit costs that specifically focused on admissions pertaining to these types of infection.

A currency, in NHS reference costs, is a unit of healthcare activity such as a finished consultant episode. The currency codes used and their associated descriptions in, for example, the costing of non-elective long-stay care are summarised (see Appendix 6, Table 21). For the primary economic analysis, we estimated the cost of a hospital admission as £1209. We also conducted a sensitivity analysis in which we repeated this calculation of unit costs, but only including those service descriptions that specifically mention RTIs. In this case, the unit cost was estimated as £1165.

Analysis and sensitivity analysis

The comparison of between-arm costs used a two-way mixed-effect linear regression that accounted for the nesting of practices in CCG clusters. The analysis used this model for mean between-arm cost differences given the sample size of the trial and the desirable coverage properties for these kinds of parametric estimators even when within-arm cost data happen to be skewed. 57,58 The primary analysis model regressed total costs on arm and covariates for list size and dispensing rate, both of which were used for minimisation at randomisation. We conducted sensitivity analyses that repeated this regression but (a) from a per-protocol perspective, following the same analysis steps described in Chapter 2 Methods but applied to the between-arm comparison of costs (b) using unit costs for hospital admissions where the currency description of care episodes specifically mentions ‘RTI’ as in Table 18 below (c) excluding intervention costs from the between-arm comparison.

Data completeness and description of cost data

We used data from 293 of 294 randomised practices; the 294th practice merged with another trial practice after randomisation and was analysed as part of the larger, merged entity. Data were complete on all resource items used to undertake the between-arm cost comparison.

The volume of resource use in each arm is shown in Table 17. All cost data reported below are in 2020/21 prices. All practices reported some costs, ranging between £188 (for a practice in the control arm) and a maximum cost of £452,029 (for a practice in the intervention arm).

Costs were similar between arms. Mean (£39,826) and median (£29,162) unadjusted per-practice NHS costs during the 12 months of trial follow-up were somewhat lower in the intervention arm than in the control arm (mean £38,246, median £28,216). The distribution of per-practice costs was also similar between arms (see Figure 22).

FIGURE 22.

Distribution of total costs in control and intervention arms.

There was no evidence of a between-arm difference when comparing cost categories (see Table 18).

Note that the costs of the intervention itself are modest compared to other cost categories, especially that of hospitalisation and A&E attendance.

Primary economic analysis

The primary analysis model regressed total costs on trial allocation and on covariates for list size and dispensing rate (see Table 19), adjusting for variance in 47 CCG clusters.

Although mean NHS costs were lower in the intervention arm, primarily because of the impact of lower hospitalisation costs, all results were consistent with a null effect of the intervention on NHS costs.

This conclusion did not change in any of the sensitivity analyses. Hospitalisation was the largest cost element, and the results of analysis in which a narrower definition of unit costs attributable to admissions was used were similar to the primary analysis. Costs attributable to the intervention itself included training time as well as potential integration and troubleshooting time. Our sensitivity analysis in which we assumed zero costs of the intervention, a possible consequence of long-term use and familiarity of the intervention, was similar to the other between-arm cost comparisons. As noted in Chapter 3, the pre-specified per-protocol analysis is likely to be biased by the effect of non-compliance among practices that joined in the second half of the study, during which period the COVID-19 pandemic had reduced dispensing rates.

Strengths of the economic evaluation

The design of the trial permitted the efficient analysis of very large volumes of data, encompassing millions of dispensed items, and tens of thousands of intervention usages, A&E attendance and hospital admissions. The reliance on routine data for all elements of the economic evaluation met the protocol’s objective of assessing the costs of the intervention to the NHS, while reducing the overall costs of the trial.

Limitations of the economic evaluation

The CHICO feasibility trial confirmed the difficulty of obtaining comprehensive, useable quality-of-life data in this young patient population. This reflected both the absence of quality-of-life instruments designed for young children (especially those aged 0–4 years), issues in interpreting quality-of-life measures in this population and the completeness of data collected from parents on behalf of participating children. By design, we did not measure the quality of life of children in this trial. This meant that the present analysis was limited to a comparison of costs, rather than a full economic evaluation.

Without patient-specific information, it was necessary to make assumptions about the type and costs of care received for those children attending A&E and those admitted to hospital. We used volume-weighted unit costs to account for uses of this service. This was likely to cause material bias for between-arm cost comparisons only if the composition of care (with respect to complications, comorbidities and any attendant complexity) was markedly different in one arm or another.

Our unit cost data on dispensed amoxicillin and macrolides also involved a compositional assumption. We used unit costs drawn from all dispensed amoxicillin and macrolides in England in the 2020–21 financial year. We, therefore, assumed that the practices included in the trial had the same costs as all other practices in England. We also lacked data on the precise formulation and presentation of dispensed items, but it is not clear that including this information would have had a noticeable impact on between-arm comparisons.

The analysis reported here was – by design – limited to a within-trial evaluation of the impact of the intervention on NHS costs during the 12-month period of trial follow-up. Distal impacts of the intervention on antimicrobial resistance and its associated costs over the long term, a critical motivation for undertaking this work, were not accounted for in the within-trial cost comparison. We also did not intend to evaluate the wider costs to the NHS of a full national roll-out, not least because this would be to presuppose both the efficacy outcome of the trial as well as the interest and capacity of systems providers to integrate the intervention.

Nevertheless, the research team did undertake very preliminary, high-level assessments of how costs might be impacted by a wider deployment of the intervention. Based on discussions with CCG ‘Research and Development’ staff, it appears that costs related to licensing, maintenance and support would be modest, especially when considered on a per-practice basis. The level of these costs would also depend on the specific way in which providers implemented and provided the intervention to practices. Nevertheless, there appear to be few grounds on which a wider use of the intervention would drastically increase NHS costs relative to the experiences observed in the CHICO trial.

Conclusion

There was no evidence of a difference in mean NHS costs in those practice randomised to use the intervention compared to those that did not. This conclusion held under various sensitivity analyses, including a per-protocol analysis.

Facilitators to recruitment

The 15 CRNs were helpful with the recruitment of practices; presenting our study at CRN meetings and using the national CRN lead for support helped facilitate a co-ordinated national approach. Some CRNs contact practices that opt-in to conduct research while others contact all the practices in their area. Feedback from 11 of the 15 CRNs suggested around half the practices had joined the Research Sites Initiative scheme (NIHR-funded scheme to enable research delivery) and were often the first to be contacted. Recruitment via CCGs had a wider reach of practices than via CRNs, although CCG participation in recruitment was more variable depending on capacity. Working with these two networks provided a large geographical spread across England (see Appendix 5) and provided us with over a quarter of the practices (26%) in the most deprived socioeconomic quintile.

Using quarterly study newsletters for practices, CRNs and CCGs with league tables monitoring levels of recruitment improved responses.

Barriers to recruitment

Recruitment took longer than expected, partly because we had to recruit CCGs first (who provide hospitalisation data for practices). It was difficult to know which individuals to contact and how to do so (some CCGs did not appear to be public-facing), response times were often slow, their role in research was often misunderstood, staff changes hindered communication and a number of CCGs merged during the study period. Some CCGs were averse to getting involved in research or cited lack of capacity as a reason to be excluded from the research.

I mean in reality they [CCG] don’t play a major part in – and never actually have a major role in research. It’s not a core business of a CCG like it would be in a provider. So we don’t have a research department, which is probably why the CHICO trial ended up at my door because it was about prescribing.

[P4 CCG]

At the start of recruitment, October 2018, there were 211 CCGs across England. By the end of follow-up, September 2021, there were only 106 CCGs. While this did reduce the number of CCG contacts required to obtain the data, it sometimes resulted in change of staff who were not familiar with the trial or its requirements.

The level of engagement was much better with CRNs although access to practices was not straightforward. Limiting the contact to practices that want to opt-in to research via some of the CRNs missed the opportunity of letting research-naïve practices know about light-touch efficient design studies. Although other CRNs contacted all practices.

I would say that about a third of the practices were what I would call research naïve or inexperienced, green as in they hadn’t really had a sort of established relationship with us in the past.

[P2 CRN]

Of the 294 practices we recruited, we are aware of at least 22 practices (7%) who merged during their baseline/follow-up annual data capture. We excluded practices who anticipated a merger with another practice but had no control over this once the practice was randomised, especially in the rare instances when the merging practices were randomised to different arms of the trial. The length of time from expression of interest from practices to randomisation was longer than expected due to the delayed site agreements returned from the practices.

Facilitators to collecting routine data

Using aggregated data avoids the need for the consulting physician to solicit individual consent, reduces or eliminates the risk of selection bias and removes the task of accessing individual patient notes. Routine data are often collected monthly so data completeness and quality can be monitored throughout the collection period, which was particularly important in this trial to scrutinise and report any sudden changes in hospital admission rates to the DMC. The routinely collected data can be imported to data management software; thus, there is less likelihood of data entry errors and missing data are rare. The cost of research staff and time taken to collect the primary outcome data was much reduced compared to a traditional trial. In total 11/294 practices asked to be withdrawn from the study (3.7%) and 14/294 (4.8%) were lost to follow-up from the study; the main reasons cited being lack of capacity and prioritising the COVID-19 pandemic. A further advantage of collecting routine data is that withdrawal of a practice does not necessarily mean a loss of the primary outcome data for that practice (if the data are already in the public domain).

Barriers to collecting routine data

A potential problem with aggregate data collected from a third party is where data are suppressed, owing to a low number of events, although in this study data were collected over a 12-month period largely avoiding this problem. Some liaison was needed between the trial team and the CCGs to know exactly what data were available and in what format this could be presented needing additional time to import and utilise data from different formats. Dispensed prescription data, practice list size data and hospital admission data are reliable as they feed into financial transactions; however, data reporting A&E attendance were less so due to a limited coding set. Many A&E attendances do not have a diagnosis coded; therefore, the number of attendances attributable to RTI is likely to be inaccurate.

Facilitators to integrating the intervention

Installation of the intervention was relatively straightforward for EMIS^®. Practice systems allow for user-friendly manipulation so interventions can be integrated and interfaced with system data already collected; thus, negating any additional clinical workload. Screen pop-ups can also be used to notify clinicians of eligible patients for the study.

Barriers to integrating the intervention

Familiarity with the EMIS^® system varied between practices; thus the level of support required from the research team to help install the intervention and download usage also varied. Provision to help install or use third-party algorithms is not offered by system providers. EMIS^® upgrades to the system during the trial meant that rewriting installation instructions, resources and testing had to be carried out and fed back to the practices. For instance, the READ codes used to identify clinical terms within consultations were upgraded to SNOMED codes by EMIS^® in the early months of 2020, which meant the algorithm had to be amended so the intervention would function correctly. We found that our funded provision of IT support throughout the study was crucial to the smooth running of the integrated intervention. Although pop-ups can be used, some clinicians found them irritating and switched them off while other practices had so many pop-ups (especially during the COVID-19 pandemic) that the CHICO pop-up was often obscured.

Facilitators to a light-touch approach

The light-touch approach reduced the time needed during consultation to record information, the trawling of patient notes and provided a more objective data resource downloaded from the system rather than from individual input. Practice champions, familiar with practice systems, played an important role in obtaining the required data and training clinicians in the use of the intervention. Baseline questionnaires were received from 294/294 practices (100%), follow-up questionnaires were received from 265/294 practices (90%) while intervention usage was collected from 116/144 of the intervention practices (81%), indicating the data burden was not too onerous. Training consisted of a brief conversation between the practice champion and clinicians, signposting to further information available if needed and a 1-month run-in period before data collection where clinicians could familiarise themselves with the intervention. Feedback suggested the clinicians were happy in terms of training to use the intervention tool.

Barriers to a light-touch approach

Conversely, this approach reduced the level of interpretation that can be gleaned from the data. Removing patient-level recruitment results in a loss of denominator data, in our case not knowing how many individual children consulted for RTI and cough. As a proxy, we used the number of children registered at each practice. Given the diversity of the practices included in the trial, we are assuming that those children taking part in the trial were no different from children in the general population, but we have no way of testing this assumption. We also lost detail that may be taken from the consultation of whether antibiotics were prescribed immediately or delayed relying instead on the number of antibiotics being dispensed. The lack of contact with control practices (apart from reminders to provide SAE reports) also ran the risk of practices wanting to withdraw from the study, especially with changes of staff. Several strategies were needed to obtain follow-up paperwork back from practices, including help from the CRN network and the managed recovery team, newsletters and time and resources spent trying to contact the practices at varying times and days. Consideration also needs to be given to the ethical implications of not seeking consent from individual patients. We asked sites to display posters informing patients the research was taking place and no concerns were raised but we did not proactively seek a response on this issue.

Impact of the COVID-19 pandemic on hospitalisation rates

The effects of COVID-19, and its associated lockdowns on hospitalisation admission were clear. From April 2020, the rates dropped and only started to climb back towards normal levels in the summer of 2021 when the rates were unseasonably high. A similar pattern was observed for A&E attendance. Despite these uncommon fluctuations, the rates for attendance and admission were similar for both arms of the trial.

Intervention usage over time

The intervention was used nearly 12,000 times in the trial with a median average of 70 uses per practice. It was difficult to gauge usage over the 12-month follow-up period, given the seasonal influence and impact of the pandemic, although the data suggest usage fell after the first few months and slightly picked up towards the end of follow-up. The qualitative interviews confirmed that clinicians used the intervention at the start of the trial, but usage waned over time. We did not collect data on how many RTI consultations were conducted in the trial. A rough estimate can be made from the dispensing data and intervention practice characteristics if we assume the dispensed items were all for RTI consultations and 50% of the children consulting were given these items. The dispensing rate in the intervention arm was 15.5 items per 100 children, given the median number of 0–9-year-olds was 975 children in 144 intervention practices, this yields 21,762 dispensed items and 43,524 consultations for RTI. This is purely speculative but would suggest the 11,944 uses of the intervention was around 1 in every 4 consultations. The qualitative interviews suggested some clinicians found the intervention most useful in ‘borderline cases’, which suggests clinicians may have been selective in terms of when they used the tool and partly explains the low usage.

Acceptance of the intervention

Generally the intervention was easy to use and few thought it unhelpful; nearly three-quarters suggested they would use the intervention again, if it became available. The qualitative interviews suggested the clinicians liked the algorithm template and found it straightforward to use, without adding any more time to consultations. Clinicians particularly liked the safety-netting parental advice leaflet, as it helped explain treatment decisions and home care with parents and this was seen to be the most useful intervention component. While some clinicians reported that the intervention influenced prescribing decisions, others reported that they used it as a supportive aid during consultations rather than a tool to change prescribing behaviour. Some clinicians reported the intervention interrupted the consultation flow, having to close the patient’s record before the end of the consultation to complete the intervention process, which did not always align with clinicians’ usual processes.

Importing the template

• Go to Setup → Data Entry → New Template Maintenance.

• Select Import Templates.

• Select the file called ‘SystmOneTemplates CHICO child cough RCT 151021.xml’ (or similar).

• The New Template window is displayed. Select the category called ‘CHICO’ if it already exists. Otherwise create and select a ‘New Category’ called ‘CHICO’. Select Ok to complete the template import.

Publishing the template

• Select ‘Unpublished Templates’ on the left-hand side of the New Template Maintenance screen. Right click the ‘CHICO child cough RCT’ template and select ‘Publish Template’ from the pop-up menu.

• Select ‘Publish Locally’ from the available options. Select Ok.

Importing the protocols

• Go to Setup → Workflow Support → Protocols.

• Select Import Protocol.

• Select the file called ‘SystmOneProtocol CHICO patient alert protocol 151021.xml’ (or similar).

• Select the category called ‘CHICO’ if it already exists or create and select a ‘New Category’ called ‘CHICO’. Select Ok.

• A confirmation creation message is displayed. Select Ok.

• You may see the following information message in which case select Ok.

• Now import the protocol file called ‘SystmOneProtocol CHICO template launch + outcomes protocol 151021.xml’ (or similar) in the same way as described above.

Editing the ‘launch ± outcomes’ Protocol

When the ‘launch + outcomes’ Protocol is imported, links from it to the clinical template and letter templates are lost. The Protocol therefore must be amended to reinstate the links.

• Select ‘Unpublished’ on the left-hand side of the Protocols screen. Right click the Protocol called ‘CHICO template launch + outcomes protocol’ and select ‘Amend Protocol’ from the pop-up menu.

• Select the tab called Design.

Three steps in the design flowchart need to be amended:

• The Action step that identifies the clinical template.

• The two steps that identify the letter templates.

Linking the clinical template

• Double left-click the Action step near the top of the design.

• In the Select Quick Action window, search for ‘template’ and select the ‘CHICO child cough RCT’ template. Select Ok.

• Select Pause Protocol.

• Check that this part of the design now looks like this.

Linking the two Letter Templates

• Towards the bottom of the Design, there are two steps with unknown letter templates. Double left-click the one on the left.

• In the Select Quick Action window, search for ‘letter’ and select New Letter → Specific Word Letter Template. Select Ok.

• Search for ‘chico’ and select ‘CHICOLetterAntibiotics’. Select Ok.

• Select Pause Protocol.

• Repeat the above process for the other unknown template but this time selecting the ‘CHICOLetterNoAntibiotics’ template.

• Check that this part of the design now looks like this making sure that the CHICOLetterAntibiotics is on the left and CHICOLetterNoAntibiotics is on the right.

• Select Ok in the top-left corner of the Amend Protocol window to save the amended protocol design.

Publishing the protocols

• Select ‘Unpublished’ on the left-hand side of the Protocols screen.

• Right click the Protocol called ‘CHICO patient alert protocol’ and select ‘Publish Protocol’ from the pop-up menu.

• Select ‘Publish locally’. Select Ok.

• A confirmation message is displayed. Select Yes.

• Repeat the above publication sequence for the Protocol called ‘CHICO template launch + outcomes’.