Enhanced feedback interventions to promote evidence-based blood transfusion guidance and reduce unnecessary use of blood components: the AFFINITIE research programme including two cluster factorial RCTs

Behaviour change techniques

Documents contained on average 8 of the 93 BCTs in Taxonomy v1 (range 3–14) (see Report Supplementary Material 6). They included, on average, half (5.6, 51%) of the 11 BCTs associated with control theory (range 2–7). The most frequent, identified in at least one document from each of the three audits, corresponded to the first half of the control theory loop: ‘goal-setting’ (i.e. audit standards), ‘feedback on behaviour’ and/or ‘outcomes of behaviour’ and ‘discrepancy between behaviour and goal’. Only one BCT, ‘action-planning’, from the second ‘adaptive response’ half of control theory was identified from all three audits (see Report Supplementary Material 6). There were no identified BCTs encouraging review of audit standards, setting of localised goals or ongoing self-monitoring.

Behavioural specificity

Overall, behavioural specificity of audit standards, feedback and recommendations for change was low. Although the action (e.g. measuring pre-transfusion haemoglobin) and target group (e.g. neonates) were often stated, the actor (e.g. consultant haematologists), context (e.g. ‘paediatric ward’) and time (e.g. within 3 days of transfusion, preferably same day) were rarely stated (see Report Supplementary Material 7). Across documents, on average only half (range 0–100%) of the feedback related to behaviours specified in audit standards, indicating that there was a high volume of extraneous feedback.

Evidence-based characteristics

We identified only two of the eight evidence-based feedback characteristics (see Report Supplementary Material 8). Feedback was always provided electronically, in writing, and, rarely, also graphically or using other modalities. The stated feedback provider was always a regulatory body, rather than a respected peer. Feedback was delivered only once, typically 12 months after data collection. Average national performance in relation to audit standards was 72% (range 64–86%), suggesting that baseline performance for targeted behaviours was reasonably high, with limited room for improvement. Although the feedback provided was not necessarily punitive, supportive BCTs such as ‘social reward’ and ‘social support’ were delivered infrequently.

Although the two evidence-based characteristics ‘include explicit goals and action plans’, and ‘use multiple comparators’ were identified from all three audits, these were not fully operationalised. Documents included goals, and some recommendations for change, but these were not explicit, as evidenced by the limited AACTT specification. All audits provided comparative feedback on peer performance (nationally and/or regionally). However, no audit compared practice against achievable benchmarks of care (see Report Supplementary Material 8).

On this basis, six recommendations for enhancing the content of feedback reports were identified and agreed with multidisciplinary stakeholders:

1. include at least one BCT corresponding to each step of control theory

2. ‘be specific’ – phrase audit standards, feedback and recommendations in terms of who/what/where/when

3. ‘be relevant’ – include only feedback related to audit standards and take a graded-entry approach by producing feedback reports with varying levels of detail, from key findings to supplementary reports

4. include multiple comparators

5. include positive messages recognising good practice

6. improve feedback document presentation (e.g. provide feedback visual format, such as graphs, make writing legible, use a consistent layout, personalise feedback).

Enhanced prototype reports and the enhancement guidance manual are available in Report Supplementary Material 9a–c and Report Supplementary Material 10a–b.

Who receives feedback?

Report Supplementary Material 11 depicts the dissemination pathway of feedback reports for each hospital. Dissemination was a key barrier to implementing feedback across all hospitals. Feedback was often initially received by the hospital transfusion team, but then not disseminated more widely to clinical staff from other specialties who prescribe transfusions or to more junior staff. The extent to which feedback from the NCA was discussed at hospital transfusion committee meetings also varied. 15

What do hospitals do with feedback and what are the barriers and enablers?

There was considerable variation in how feedback was received, shared and responded to across hospitals. The key barriers to and enablers of feedback across all cases fell into eight theoretical domains (see Report Supplementary Material 12):

1. social influences – not sharing and discussing feedback with colleagues; lack of influence over practice change and support from peers; desire for feedback to be delivered from a familiar, respected colleague; the view that comparison against national averages is not useful for identifying areas for improvement

2. behavioural regulation – not setting goals and action plans as a team; need for support materials and tools to facilitate planning locally; having to amend feedback to make it relevant locally

3. social professional role/identity – lack of clarity about who is responsible for A&F

4. knowledge – variable and limited awareness of transfusion NCA

5. motivation – competing priorities and audit fatigue

6. environmental context and resources – lack of staff and resources to conduct re-audits and/or implement change

7. beliefs about consequences – A&F does change practice

8. memory attention decision-making – not recalling the feedback materials; noticing new information only when it is different or clinically relevant. 15

The prototype enhanced follow-on intervention included 19 BCTs targeting these barriers and enablers. Report Supplementary Material 13 presents the full list of BCTs and associated TDF domains. These BCTs were delivered through a paper-based ‘toolkit’ to support staff in planning their response to feedback. Tools included a dissemination cascade to facilitate the sharing of feedback reports, a fishbone problem-solving worksheet, goal-setting and action plan templates, and ‘QuickAudit’ local re-monitoring template (see Report Supplementary Material 14). The intervention involved a 1-hour training visit during which a member of the research team met with each hospital transfusion team to talk them through the toolkit (i.e. its purpose and how to use it).

Enhanced content

Enhanced content comprised feedback documents with content written to specifically deliver behaviourally specified feedback and the relevant theoretically consistent BCTs. These were delivered by the NCA programme via written and graphic feedback presented in multiple feedback documents and presentations.

Enhanced follow-on

Enhanced follow-on made use of targeted dissemination of feedback to relevant staff with discussion and agreement of action plans. Follow-on support comprised practical guidance for clinical teams on how to operationalise the process of responding to feedback, including materials for clinical teams to facilitate discussion and agreement of locally relevant goals and action plans based on feedback.

Screening and recruitment

Prior to cluster randomisation (15 October 2015), we screened 189 NHS trusts and health boards. A total of 152 (80.4%) were included in the audit, and 135 of these (88.8%) were randomised. Reasons for exclusion, when given, were mainly mergers or ineligibility, so the included clusters were typical of the UK as a whole. The baseline audit comprised a total of 2714 patients, 40% of target; 66 clusters were allocated to standard content, 69 were allocated to enhanced content, 67 were allocated to standard follow-on, and 68 were allocated enhanced follow-on. An average cluster size of 20 patients was observed, with a coefficient of variation of 0.7 (Figure 3).

FIGURE 3.

Trial 1: NHS trust/health board and patient CONSORT flow diagram.

Over half of the clusters received the enhanced (62%) and standard reports (56%). Receipt of the toolkit was lower (31% overall), with more clusters allocated to standard content receiving the toolkit (36%) than those allocated to enhanced content (26%). A total of 23 out of 135 clusters (17%) were lost to follow-up, this proportion being higher among those allocated to enhanced content (22% vs. 12%) and enhanced follow-on (21% vs. 13%). Overall, two clusters had no cases to audit, eight clusters provided audit cases but did so too late, and 13 declined to take part. Among the 112 clusters taking part in the follow-up audit, there were a total of 2222 audit cases (32% of target). Again, an average cluster size of 20 patients and a coefficient of variation of 0.7 were observed. We included all 112 clusters and 2222 patients in our primary analyses. Table 2 shows baseline patient-level characteristics. (Other baseline summaries are given in Appendix 2, Table 10.) These were generally well balanced, as would be expected given that patients were ascertained prior to randomisation. Notably, 30% surgical procedures were for a fractured neck of femur and, as anticipated, these patients would not typically be able to attend a preoperative clinic. In total, 249 (9%) received a preoperative, 363 (13%) an intraoperative and 2560 (94%) at least one postoperative transfusion.

The health-care professional making the decision to transfuse was usually the consultant or other doctor, with this being more commonly a junior doctor postoperatively.

Appendix 2, Table 11, shows that patient-level characteristics in the follow-up audit were generally well balanced. Recruitment bias following cluster randomisation was therefore minimal, with imbalances likely to result from missing clusters. The characteristics of patients at baseline and at follow-up are similar.

Outcomes at baseline

Patient-level outcomes at baseline are provided by content and follow-on in Appendix 2, Table 12. The proportion of missing data for the primary outcome was 7%, well balanced by follow-on but less well balanced by content. Around 80% of patients had a pre- or postoperative transfusion outside guidelines: 82% of patients received a preoperative and 78% a postoperative transfusion outside guidelines. On average, 2.2 units of blood were transfused per patient by content and follow-on.

A large number of patients were excluded from the audit standards: 31% from standard 1, 91% from standard 2, 95% from standard 3, 91% from standard 4 and 6% from standard 8. Feedback given to clusters was therefore limited to specific patient groups, limiting the ability of clusters to make changes at scale. A significant group of patients could not be classified for standard 1 (28%) and standard 8 (27%). Approximately 26% of patients had a transfusion classified as not meeting standard 1, 7% had one classified as not meeting standard 2, 2% had one classified as not meeting standard 3, 6% had one classified as not meeting standard 4 and 64% had one classified as not meeting standard 8. Thus, the greatest room for improvement was for standard 8, and missing data were substantial. As expected, patient outcomes at baseline were well balanced across content and follow-on.

Outcomes at follow-up

Patient-level outcomes at follow-up are provided by enhanced content and follow-on groups in Table 3 (other outcomes are summarised in Appendix 2, Table 13). The proportion of missing data for the primary outcome was 11%, balanced across randomised groups. Approximately 73% of patients had a pre- or postoperative transfusion outside guidelines; again, the proportions were similar between randomised groups. Overall, 62% of patients receiving a preoperative transfusion and 72% receiving a postoperative transfusion received one outside guidelines.

The proportions were similar across randomised groups in postoperative transfusions, and these contributed to a greater extent to the primary outcome than preoperative transfusions. On average, 2.1 units of blood were transfused per patient, similar across randomised groups. Similar numbers of patients were excluded from audit standards at follow-up. Again, a significant group of patients could not be classified for standard 1 (22%) or standard 8 (33%). Approximately 27% of patients had a transfusion classified as not meeting standard 1, 6% had one classified as not meeting standard 2, 2% had one classified as not meeting standard 3, 5% had one classified as not meeting standard 4 and 57% had one classified as not meeting standard 8. The greatest difference observed between randomised groups was for standard 8, but the number of missing data was substantial.

Primary outcomes

Table 4 shows the primary outcome results (with the main sensitivity analysis in Appendix 2, Table 14). Across 100 imputations, the unadjusted proportion of acceptable transfusions was 18% for those allocated to standard content and to enhanced content, and the adjusted risk difference was –1% (95% CI –7% to 4%). There was no evidence of a clinically or statistically significant effect. The unadjusted proportion of acceptable transfusions was 18% for those allocated to standard or enhanced follow-on, and the adjusted risk difference was 1% (95% CI –5% to 6%), again providing no evidence of a statistically significant effect. For the follow-on intervention, a clinically important effect size of 5% (in the presence of an interaction) was not ruled out. The interaction between content and follow-on is given for information. The conclusions were unchanged regardless of the method adopted for handling the missing data. The estimates were also similar, indicating that this result is robust.

Secondary outcomes

Appendix 2, Table 17, describes the cluster-level volume of RBCs transfused (BSMS) across baseline and follow-up (and in 3-month periods). Overall, seven and six clusters, respectively, were lost to follow-up. Data were skewed; interquartile ranges (IQRs) are similar across randomised groups at baseline and follow-up. There is an indication of a steady overall reduction in blood use over this period. Appendix 2, Tables 12 and 13 summarises the patient-level volume of RBCs transfused (NCA), in each case separately for preoperative and postoperative transfusions. Imbalances are observed at baseline by randomised group, expected to be related to imbalances in missing clusters.

Appendix 2, Table 16, describes the cluster-level total number of SHOT incidents and number of relevant errors. There were no missing SHOT data, as all 135 clusters provided data, summarised across baseline and follow-up (and in 3-month intervals). Interpretation is limited because of sparse data relating to the number of relevant errors. The median number of relevant events across the baseline and follow-up periods was zero across the randomised groups. IQRs were the same across baseline and follow-up. Therefore, there is no evidence that the trial interventions increased or decreased secondary outcomes reported to BSMS, NCA or SHOT.

Appendix 2, Table 15, reports supportive outcomes.

Study design and setting

The trial 2 methods were similar to those in trial 1. We conducted a second 2 × 2 factorial, cross-sectional, cluster-randomised controlled trial, embedded in 2016/17 NCABT, of RBC and platelet transfusions in haematology patients.

Cluster eligibility

Cluster eligibility was assessed separately for trial 2 using the same criteria as for trial 1. Where possible, cluster definitions remained the same across trials; however, in some cases, different hospital sites within clusters signed up to the audits.

Randomisation

As previously, the trial statistician undertook re-randomisation at a single point following receipt of the NCA baseline audit database. We independently randomised trusts or health boards on a 1 : 1 : 1 : 1 basis, as before balancing for trust size and regional transfusion committee, but also for the previous treatment allocation where clusters had been entered into both trials.

Interventions

The interventions remained the same across the trials, although the content of the feedback reports was tailored to the audit topic.

Monitoring intervention adherence

As before, the assessment of fidelity was based on a fidelity framework proposed by the NIH Behaviour Change Consortium. 22

Data collection

The required data and collection time points for trial 2 mirrored those in Table 1, but baseline was from July 2015 to June 2016 and follow-up was from July 2016 to June 2017.

Outcomes

Transfusion may occur for RBCs or platelets; all patients will have had one or more transfusions. The primary outcome, measured at the patient level and taken from the NCA follow-up audit, was to compare how many RBC and/or platelet transfusions were acceptable with how many transfusions were carried out outside guidelines. The clinical algorithm (see Appendix 1) was agreed by an independent panel based on clinical relevance and baseline compliance. No clinical judgement was required at a patient level to categorise transfusions.

Secondary outcomes comprised:

• total volume of RBCs transfused (units at trust level, from BSMS summed over blood groups and clinical specialties)

• total volume of RBCs transfused (units at patient level, from NCA)

• total volume of platelets transfused (units at patient level, from NCA)

• total number of incidents reported to SHOT (count at trust level, from SHOT summed over clinical specialties and events, near-misses and ‘right blood right patient’ incidents)

• number of definitely, probably or possibly preventable incidents reported to SHOT within clinical specialties targeted by the audit (count at trust level, from SHOT summed over events, near-misses and ‘right blood right patient’ incidents).

Supportive outcomes comprised:

• RBC transfusion (acceptable/outside guidelines)

• platelet transfusion (acceptable/outside guidelines)

• individual NCA audit standard met (1, 2, 3, 6 and 7; yes/no)

• total volume of RBCs issued (units at trust level, from BSMS data)

• total volume of RBCs transfused (units at trust level, from BSMS data).

As in trial 1, all other outcomes (i.e. intermediate NCA outcomes, number of relevant near-miss or ‘right blood right patient’ SHOT incidents, total number of unpredictable SHOT incidents, total number of possibly preventable SHOT incidents, BSMS total volume of RBCs wasted) reported are exploratory.

Sample size

As previously, we required 17–68 patients (mean 45) from 152 clusters in order to detect a minimally important reduction of 6% (to 14%) in the presence of, at most, a small antagonistic statistical interaction (i.e. main effects of 5%) with 80% power using logistic regression models, with a random-intercept for cluster, and a two-sided 2.5% significance level, for each comparison.

Statistical analysis

No interim analyses were planned or conducted. A similar analysis strategy was adopted for trial 2, except that, when adjusting for design factors, previous allocation was added.

Screening and recruitment

Prior to cluster randomisation (6 July 2016), we screened 187 NHS trusts and health boards, covering the whole UK. A total of 141 (75.4%) were included in the audit, and 135 of those (95.7%) were randomised (although one was randomised in error, so 134 were entered). Reasons for exclusion were mainly ineligibility or declining to take part in the audit, so included clusters were typical of the audit as a whole. The baseline audit comprised a total of 4372 patients, 64% of target; 68 clusters were allocated to standard content, 66 were allocated to enhanced content, 67 were allocated to standard follow-on and 67 were allocated to enhanced follow-on. An average cluster size of 33 patients was observed, with a coefficient of variation of 0.5 (Figure 4).

FIGURE 4.

Trial 2: NHS trust/health board and patient CONSORT flow diagram.

Overall, half of the clusters received the enhanced (50%) and standard reports (47%). Receipt of the toolkit was lower (30% overall), with the proportion of clusters receiving the toolkit lower among those allocated to standard content (24%) than among those allocated to enhanced content (36%). In total, 12 out of 135 clusters (9%) were lost to follow-up; similar proportions had been allocated to standard and enhanced content (10% vs. 8%) but a higher proportion had been allocated to standard follow-on than to enhanced follow-on (12% vs 6%). Overall, four clusters had no cases to audit and eight clusters declined to take part in the follow-up audit. Among the 123 clusters taking part in the follow-up audit, there were a total of 3886 audit cases. We included all 123 clusters and 3859 patients (56% of target) in our primary analyses; 27 patients were excluded because they had received a platelet transfusion only for therapeutic reasons, making them ineligible. An average cluster size of 32 patients and a coefficient of variation of 0.5 were observed at analysis.

Appendix 2, Table 18 shows the baseline patient-level characteristics. (Other baseline summaries are given in Appendix 2, Table 19.) These are generally well balanced, given that they were ascertained prior to randomisation. Overall, 1387 (31%) patients received a RBC and platelet transfusion, 2781 (63%) patients received only a RBC transfusion and 271 (6%) patients received only a platelet transfusion. For this reason, the majority categorised were RBC transfusions for medical or chronic anaemia.

Patient-level characteristics in the follow-up audit are given in Appendix 2, Table 20. These are as balanced as they were at baseline, providing no indication of recruitment bias after cluster randomisation. Characteristics of patients at baseline and follow-up are also comparable, except for the overall proportion of patients receiving a RBC transfusion undergoing a chronic transfusion programme (33% vs. 25% at baseline).

Outcomes at baseline

Patient-level outcomes at baseline are given by content and follow-on group in Appendix 2, Table 21, together with detailed definitions. The proportion of missing data for the primary outcome was 12%, balanced by content and follow-on. Around 26% of patients received a RBC or platelet transfusion outside guidelines. Overall, 11% of patients receiving a RBC transfusion and 45% receiving a platelet transfusion received one outside guidelines. On average, 2.0 units of RBCs and 1.0 units of platelets were transfused per patient across content and follow-on groups.

As in trial 1, a large number of patients were excluded from the audit standards, 6% for standard 1, 75% for standard 2, 99% for standard 3, 90% for standard 6 and 71% for standard 7, owing to the specific subset of patients considered in each standard. As such, the feedback given to clusters was focused on specific patient groups, thereby limiting the ability of clusters to make changes that would apply to all patients. Few patients could not be classified (0% for standards 1, 3 and 7; 2% for standard 2; and 7% for standard 6). Of those eligible, 6% of patients had a transfusion classified as not meeting standard 1, 73% as not meeting standard 2, 47% as not meeting standard 3, 42% as not meeting standard 6 and 38% as not meeting standard 7. Thus, the greatest improvement was required in standards 2, 3, 6 and 7, which focused on specific patient groups. As expected, patient outcomes at baseline were well balanced across content and follow-on groups.

Outcomes at follow-up

Patient-level outcomes at follow-up are given by content and follow-on in Table 5 (other outcome summaries are given in Appendix 2). The proportion of missing data for the primary outcome was 9%, balanced by randomised groups. About 25% of patients received a RBC or platelet transfusion outside guidelines, and the proportions were similar between randomised groups. Overall, 11% of patients receiving a RBC transfusion and 45% receiving a platelet transfusion received one outside guidelines. Again, proportions were similar between randomised groups. On average, 2.0 units of RBCs and 1.0 units of platelets were transfused per patient, the same across randomised groups.

Similar numbers of patients were excluded from each audit standard at follow-up. Again, only a few patients could not be classified, with the exception of standard 3, for which 12% could not be classified. Of those eligible, 6% of patients had a transfusion classified as not meeting standard 1, 72% as not meeting standard 2, 21% as not meeting standard 3, 42% as not meeting standard 6 and 36% as not meeting standard 7. Differences between enhanced and standard content were indicated for standards 2, 3, 6 and 7, favouring enhanced content for standards 3 and 7 and favouring standard content for standards 2 and 6. Differences between enhanced and standard follow-on were minimal across all the standards.

Primary outcomes

Table 6 shows primary outcome results (sensitivity analyses are given in Appendix 2, Table 23). Across 100 imputations, the unadjusted proportion of acceptable transfusions was 74% for those allocated to standard content and 71% for those allocated to enhanced content; the adjusted risk difference was –4% (95% CI –9% to 2%). There was no evidence of a clinically or statistically significant effect. The unadjusted proportion of acceptable transfusions was 74% for those allocated to standard follow-on and 72% for those allocated to enhanced follow-on; the adjusted risk difference was –1% (95% CI –6% to 5%), again indicating no evidence of a statistically or clinically important effect. The interaction between content and follow-on is again given for information. The conclusions were unchanged regardless of the method adopted for handling the missing data. The estimates are also similar, indicating that this result is robust.

Secondary outcomes

Cluster-level volume of RBCs transfused (BSMS) is summarised descriptively across baseline and follow-up (and in 3-month periods) in Appendix 2, Table 26. Overall, three clusters were lost to follow-up. Data were again skewed; IQRs are again similar across randomised groups at baseline and follow-up. Patient-level volume of blood components transfused (NCA) is summarised descriptively in Appendix 2, Tables 21 and 22, in each case separately for RBC and platelet transfusions. The conclusions are consistent. Cluster-level total number of SHOT incidents and number of relevant errors are summarised descriptively in Appendix 2, Table 25. There were no missing SHOT data, as all 134 clusters provided data, summarised across baseline and follow-up (and in 3-month intervals). Again, sparse data relating to the number of relevant errors limit the interpretation of these. The median number of relevant events across randomised groups was similar at baseline and follow-up; IQRs were also similar. As in trial 1, there is no evidence that the trial interventions changed secondary outcomes reported to BSMS, NCA or SHOT.

Appendix 2, Table 24 reports supportive outcomes.

Limitations

Four main factors affected the interpretation of the cluster-randomised trials. First, the number of clusters participating in each audit and in the trials, as well as the number of patient records audited in each cluster, was smaller than projected. This compromised statistical power, increasing uncertainty around our estimates of intervention effects. However, the results appeared sufficiently consistent across the trials, the two interventions, and the range of primary and secondary outcomes for us to conclude that it is highly unlikely that the enhanced interventions had any important effects. Second, randomised clusters lost to follow-up were not included in analyses. Further sensitivity analyses are required to explore the impact of this. Third, the audit data contributing to trial outcome measures were more complex and required more review and cleaning than anticipated. Fourth, the audit standards that required most improvement were relevant to only a subset of patients included in the audits. This limited the ability of the trial to detect change in practice at scale, as it diluted the underlying effects of the interventions.

Design overview and model

The analysis was conducted using decision-analytic modelling from the perspective of the NHS. For each of trials 1 and 2 we compared the costs and the outcomes of the ‘enhanced content’ with ‘usual content’ arms, and the ‘enhanced follow-on support’ with ‘usual follow-on support’ arms (Figure 5). We explored uncertainty around the parameters used in the model using sensitivity analysis.

FIGURE 5.

Cost-effectiveness analysis model for trials 1 and 2.

The primary outcome of the trials was the proportion of transfusions given that were acceptable. For ease of understanding the incremental cost-effectiveness ratios (ICERs) presented in this analysis, we multiplied those proportions by 100 so that they were percentages. Therefore, the primary outcome of this analysis was the percentage of acceptable transfusions. Secondary outcomes comprised the volume of blood transfused and the number of SHOT-reportable incidents. The time horizon for this analysis was 1 year, during which two rounds of audits took place for each trial. We applied no discounting.

Unit of analysis

The unit of analysis for this evaluation was per site, reflecting the fact that the intervention was delivered at secondary care sites, and that many of the cost data available were for local, site-level costs. This represented a change from the original protocol, in which the unit of analysis was ‘per patient.’

Analysis overview and changes to protocol

The analysis presented follows the original proposal, with the following amendments:

• The unit of analysis was originally ‘per patient’ but was changed to ‘per site’, as the costs were collected at trust and whole intervention level, and the interventions were delivered at the site level.

• The number of transfusion-related adverse events used in the economic analysis was taken from the results of the statistical analysis and was not modelled in the long-term, reflecting the rest of the analyses’ time horizon.

• The analysis was amended to undertake four two-arm comparisons, rather than two four-arm comparisons, to be appropriate to the overall study design.

• The protocol intended quality-adjusted life-years (QALYs) to be estimated from SHOT events. However, the number of these events was small and there was no significant difference in SHOT events between the comparison groups, meaning that a cost–utility analysis would not yield meaningful results.

Costing exercise methodology

The costs of the intervention were collected from a combination of top-down (gross-costing) and bottom-up (microcosting) methods, depending on the type and the quality of data available. An extensive costing exercise was undertaken to estimate the resources required to deliver the A&F interventions and the standard practice audits. The resource components included in this exercise were collecting the audit data, developing and delivering the audit reports, delivering the follow-on-support programmes, additional NHS activity in response to receiving the audit reports and the volume of blood transfused. The costs associated with the volume of blood transfused were far higher than the other costs combined. There was a large degree of uncertainty around these cost estimates, particularly around the amount of time NHS staff spent collecting and inputting audit data and using the online toolkit, and the volume of blood transfused per site. This uncertainty is explored in the sensitivity analyses. The costing exercise methodology and results are detailed in Report Supplementary Material 18.

Unit costs

The cost used for purchasing a unit of blood components was £128.99 and £185.86 for RBCs and platelets, respectively. 23 We applied a cost of £51.32 for transfusing 1 unit of blood components. This was obtained using the value given for subsequent units of blood transfused in the costing statement of the National Institute for Health and Care Excellence (NICE) guidelines for blood transfusion and management,24 and inflating the difference for 2017/18.

Therefore, the unit cost of one unit of RBCs and platelets was £175.78 and £229.51, respectively. Unit costs of staff time were drawn from the NHS pay scale 2017/18,25 the British Medical Association pay scales for junior doctors 2017/18,26 and the British Medical Association pay scales for consultants 2017/18. 27 Non-clinical staff costs were drawn from the health-care or research organisation employing the relevant staff member (i.e. NHSBT; University of Leeds; or City, University of London).

Outcome parameters

Outcome data were obtained from the outputs of the trial analyses (see Tables 4 and 6). Where necessary, these outcomes were converted from mean-per-cluster to mean-per-site to conform to the unit of analysis for this study. Table 7 presents the parameters used in the analysis for the primary and secondary outcomes for trial 1, and Appendix 3, Table 28, describes those for trial 2.

Cost-effectiveness analysis methodology

We calculated ICERs to measure the cost-effectiveness for each comparative analysis for trials 1 and 2. The ICER is interpreted as the additional cost required to produce one unit of benefit. For the primary outcome, benefit is defined as one additional percentage of blood transfused acceptably. For SHOT it is one fewer SHOT-reportable event. We treated the volume of blood used as a resource consumed and not as a health consequence, and thus included it only on the cost side of the ICER calculations.

The ICER calculation shown in Appendix 3, Tables 27 and 28, demonstrates how the different cost components were included in the calculation of the ICER for the primary outcome.

Sensitivity analysis methodology

To explore the uncertainty around the parameters used in our model, we conducted probabilistic sensitivity analysis (PSA) and deterministic sensitivity analysis (DSA). Each parameter in the DSA was varied by 20% above and below the mean value. For the PSA all cost parameters were characterised using a gamma distribution, as was the volume of blood transfused and the number of SHOT events. The proportion of acceptable transfusions was characterised using a beta distribution.

We ran 1000 iterations for each PSA comparison and presented the results in terms of mean incremental difference and associated uncertainty intervals (UIs) on a cost-effectiveness plane. A cost-effectiveness acceptability curve (CEAC) was plotted for each analysis. We found that PSA iterations fell into all four quadrants of the cost-effectiveness plane. To calculate the ICER relative to willingness-to-pay (WTP) thresholds for the CEAC, we took the following approach:

• All iterations in the north-west quadrant were never considered to be cost-effective.

• All iterations in the south-east quadrant were always considered to be cost-effective.

• Iterations in the north-east quadrant had to fall below the WTP threshold to be considered cost-effective.

• Iterations in the south-west quadrant had to be above the WTP threshold to be considered cost-effective.

Budget impact and cost-neutral analyses

We calculated the effect of each intervention on all 185 participating sites in trial 1 and 194 sites in trial 2 to estimate the effect on the NHS budget at a national level. We then adopted a cost-neutral framework by asking ‘How many units of blood would need to be prevented from being transfused in order for the interventions to be cost neutral (i.e. to break even)?’. For this analysis we evaluated the cost of each intervention in terms of the equivalent cost in units of RBCs. We did this by calculating the incremental costs of each pair of interventions (enhanced minus standard) for each trial, and dividing the result by the cost of a unit of RBCs. We used sensitivity analysis to explore the impact that the intervention costs and unit costs of RBCs had on the results.

Intervention costs: budget impact analysis

Table 8 presents the cost of each arm for both trials. The highest costs across all arms were those of the blood transfused. The lowest costs were those of the feedback interventions.

Excluding the cost of blood transfusions, in trial 1 the incremental cost of the enhanced compared with the standard content intervention was £219 per site, and the incremental cost of the enhanced compared with the standard follow-on support intervention was £18 per site. For trial 2 these figures were £248 and –£198, respectively. Owing to the cost of additional NHS activity, enhanced follow-on support cost less per site than standard support.

Probabilistic sensitivity analysis: primary outcome

For each of the following analyses there is considerable uncertainty in the estimates produced, indicated by the wide uncertainty intervals, which include zero.

Trial 1: percentage of transfusions acceptable (enhanced versus standard content)

The PSA results suggest that the mean incremental cost of the enhanced versus standard intervention was £129,449 per site (95% UI –£1,092,049 to £1,350,946 per site). The mean incremental change in percentage of acceptable transfusions was –0.81% (95% UI –3.1% to 1.4%). This indicates that the enhanced intervention was more costly and less effective than the standard intervention.

Figure 6 shows the simulated outputs shown on a cost-effectiveness plane (CEP).

FIGURE 6.

Simulated outputs on the cost-effectiveness plane for trial 1 enhanced vs. standard content for the primary outcome. Each blue diamond represents one individual simulated output of the model.

Figure 7 shows the probability of the enhanced intervention being cost-effective at increasing WTP thresholds. The slight downwards-trending CEAC is due to more iterations falling in the south-west quadrant than in the northeast quadrant (32% vs. 14%, respectively), and as the WTP threshold increases more iterations in the south-west are excluded from being cost-effective than iterations in the north-east are included as being cost-effective.

FIGURE 7.

The CEAC for trial 1 enhanced vs. standard content for the primary outcome. The navy line represents the probability that the intervention is cost-effective at ascending WTP thresholds.

Trial 1: percentage of transfusions acceptable (enhanced versus standard follow-on support)

The PSA results suggest that the mean incremental cost of the enhanced versus standard intervention was £64,319 per site (95% UI –£1,003,034 to £1,131,673 per site). The mean incremental change in percentage of acceptable transfusions was –0.11% (95% UI –2.38% to 2.16%). Therefore, the enhanced intervention was more costly and less effective than the standard intervention.

Figure 8 shows the simulated outputs on a cost-effectiveness plane (CEP).

FIGURE 8.

A CEP for enhanced vs. standard follow-on support for percentage of acceptable transfusions.

Figure 9 shows the CEAC holding fairly constant at 45% probable cost-effectiveness. This is because the number of iterations in the south-west quadrant that are found to not be cost-effective is roughly similar to the number in the north-east quadrant that are found to be cost-effective and this remains the case as the WTP threshold increases. See Appendix 3 for further results tables on trial 2.

FIGURE 9.

The CEAC for enhanced vs. standard follow-on support for percentage of acceptable transfusions.

Deterministic sensitivity analysis results

Varying all of the intervention components had a negligible effect on the model, with the exception of the cost of blood transfused. Varying the cost per site of RBCs and platelets had a large effect on the model, as did varying the percentage of appropriate transfusions and the number of SHOT events. The detailed results of the DSA are presented in Report Supplementary Material 19.

Cost-neutral analysis

For trial 1, the mean incremental cost of the enhanced content intervention compared with the standard content intervention was £219 per site and of enhanced versus standard follow-on support was £18 per site. Hence, the interventions would need to reduce the volume of unacceptable blood transfusions by 1.2 and 0.1 units of RBCs per site, respectively, to be cost neutral.

In trial 2, the mean incremental cost of the enhanced content intervention compared with the standard content intervention was £248 per site and the mean incremental cost of enhanced compared with standard follow-on support was –£198 per site. Hence, the interventions would need to reduce the volume of unacceptable blood transfusions by 1.4 and 0.0 units of RBCs per site, respectively, to be cost neutral. The incremental costs and equivalent units of RBCs are presented in Table 9 (comparable results for trial 2 are shown in Appendix 3, Table 29).

Quantitative assessment

Web analytics data were used to assess initial receipt. The NCA web page on which interventions were delivered was programmed to record the number of times each feedback report (enhanced and standard) was downloaded throughout the intervention period (8 months). For enhanced follow-on, the toolkit was programmed to record usage patterns (i.e. number of logins, page views, adding/deleting characters). Data were collected for all clusters participating in each trial (trial 1, n = 135; trial 2, n = 134). For clusters with multiple sites, site-level data were aggregated to cluster level. Web analytics data were summarised using descriptive statistics and compared across interventions and trials.

Qualitative assessment

In-depth semistructured qualitative interviews were conducted with 35 participants from 21 clusters in trial 1 and with 20 participants from 14 clusters in trial 2. Participants were hospital staff in the hospital transfusion team and/or the clinical specialty being audited in each trial (surgery and haematology; see Appendix 4). Participants were recruited via a study information sheet that was sent to the NCA listed contact at each site, who was asked to forward the invitation to potentially eligible staff. Interviews were conducted by trained researchers over the telephone approximately 6 months after intervention delivery. These interviews were semistructured and included questions to explore how hospitals had responded to feedback, particularly extent of receipt (i.e. whether or not the participant recalled receiving the intervention materials, how much of the feedback materials they read and ease of understanding), enactment of the aforementioned target behaviours, and factors facilitating and hindering this. The interview topic guide is available in Report Supplementary Material 20. Interviews were transcribed, anonymised and analysed using inductive thematic synthesis. 34 Themes were compared across interventions, trial arms and trials. Analysis was primarily conducted by a researcher not involved in designing and delivering the enhanced interventions. However, interpretation and proposed themes were discussed with the process evaluation research team, some of whom had been involved in the intervention design.

Enhanced content

For enhanced content, 100% of the intended enhancements were present in at least one enhanced report in both trial 1 and trial 2 (see Report Supplementary Material 21). This provided evidence that it was feasible to deliver, with good fidelity, the proposed enhancements to the design and content of feedback.

Enhanced follow-on

For enhanced follow-on, 100% of the 17 intended BCTs were identified in the final version of the web-based toolkit (see Report Supplementary Material 13). This version was used in both trials. An initial telephone support call was delivered to 89% of clusters randomised to receive enhanced follow-on in trial 1 (n = 71), and to 90% of clusters in trial 2 (n = 77). Transcripts of trial 1 telephone support sessions contained on average 96% of manual-specified BCTs, whereas trial 2 sessions contained on average 86% of manual-specified BCTs (see Report Supplementary Material 22).

Web analytics

In trial 1, most clusters downloaded the enhanced feedback reports (summary key and full findings versions) at least once (Figure 10) [57 (82.6%) clusters receiving enhanced content and 52 clusters (78.8%) receiving standard feedback]. This percentage difference was not statistically significant (χ²₍₁₎ = 0.317; p = 0.574).

FIGURE 10.

Percentages of clusters downloading enhanced and standard feedback reports in trial 1.

In trial 1, most clusters (59; 86.8%) logged in to the toolkit at least once (see Figure 3). This was lower in trial 2 (49; 73.1%). In trial 1, the highest number of logins occurred in months 1 and 2, with the number subsequently decreasing over the intervention period (Figure 11), except in month 6, when there was a spike in logins that coincided with data collection for interviews.

FIGURE 11.

Number of logins in to the enhanced follow-on toolkit in trial 1.

Almost all toolkit enactment metrics (page views, downloads, items edited/added/deleted) had median scores of 0, indicating low engagement with the toolkit. Metrics with median scores of ≤ 1 included page views for the introduction page (median = 1), dissemination cascade (0.5) and selecting standards tool (0.5). Full descriptive statistics on all enactment metrics across both trials are available in Report Supplementary Material 23.

Interviews

Themes and supporting quotations related to receipt, enactment and context across interventions are presented in Report Supplementary Material 24–26 for trial 1 and Report Supplementary Material 27 for trial 2.

Receipt

For enhanced content, receipt was high: most participants interviewed in trial 1 recalled receiving feedback about their performance related to the elective surgery audit. Participants reported that the enhanced reports were clear, well written, comprehensible and structured in a logical order, with helpful visual representations of performance data that enabled them to readily identify their hospital’s performance. Many participants expressed a preference for the enhanced reports over standard reports. Standard content receipt was mixed: all participants in the standard content arms recalled receiving feedback reports from this audit. However, many participants reported having great difficulties with reading and interpreting feedback from standard reports. Although not all participants experienced such problems, some reported receiving summary versions produced by their hospital’s transfusion practitioner. Although the recommendations for change in both standard and enhanced reports were considered clear, some felt that the recommendations were unrealistic, for example those around the use of cell salvage machines when there was a lack of available equipment and trained clinicians (see Report Supplementary Material 24).

By contrast, receipt of enhanced follow-on was low: only six participants recalled receiving the toolkit. Many did not comment further on their understanding of and engagement with the intervention. Those who did said that they required support to use the toolkit as they struggled to use it otherwise (see Report Supplementary Material 24); however, no participants called the telephone support line that was offered as part of enhanced follow-on.

Enactment

In both trials, all participants reported enacting key behaviours targeted by the interventions, with minimal variation across trial arms and interventions. All participants shared feedback materials with colleagues, typically at hospital transfusion committee meetings. There were concerns over feedback not being disseminated to front-line staff (e.g. junior doctors). Shorter summary enhanced reports were more likely to be disseminated widely. All participants reported conducting gap analyses to set localised goals based on where their performance was poorest relative to audit standards and other hospitals. Action plans were in turn developed by participants in all intervention groups, and often shared with relevant colleagues. Some hospitals did not report ongoing monitoring of practice locally. However, some re-monitored weekly or monthly using a mix of formal and informal audits. Although enactment of target behaviours was evident, participants did not explicitly refer to using the enhanced reports or toolkit to facilitate these processes (see Report Supplementary Material 25). It is, therefore, not possible to attribute enactment to the enhanced interventions.

Web analytics

The NCA website tracked downloads of all documents in the document library for each site, including non-trial-related documents such as reports from previous audits and data collection tools for other ongoing or planned audits. Figure 12 shows the number of non-trial documents downloaded by all clusters during trial 1. This large number of downloads suggests that hospitals were often engaging with materials from other transfusion audits, representing potential competing priorities that could detract from engagement with the trial audits.

FIGURE 12.

Number of downloads of non-trial documents for 8 months following intervention delivery.

Interviews

Several contextual factors were identified during the interviews with participants from all trial arms that may have undermined perceptions of, and response to, the feedback (see Report Supplementary Material 26). Many of these related to the audit side of the NCA audit and feedback intervention. The enhanced interventions in AFFINITIE focused on the feedback component and did not intervene with or attempt to enhance the audit component, which was standardised across all trial arms (i.e. same audit standards and data collection procedures).

Many participants discussed how some audit standards were unrealistic and did not represent the complexity of blood transfusion practice, particularly for different types of surgery (trial 1). Audit standards were also perceived as inflexible and not accounting for clinical judgement and knowledge when decisions to transfuse were made. In trial 2, strong concerns were expressed about the lack of evidence underpinning certain audit standards. Participants argued that adherence to such standards would be inappropriate. Consequently, several participants reported low motivation to change their transfusion practice, despite performing poorly against these standards.

These issues were further compounded by perceived issues around the credibility of the audit data, because of findings based on a small number of patient cases, the inappropriate inclusion of certain patient groups (e.g. fractured neck of femur patients in elective surgery audit) and scepticism around how performance against standards was assessed. Some viewed the audit data collection process as complicated, burdensome and confusing, sometimes leading to inaccurate data being collected and resulting in erroneous performance feedback being delivered.

Many participants reported that colleagues had varying levels of interest in blood transfusion. Unsurprisingly, transfusion practitioners and haematologists viewed transfusion as a core part of their clinical role/specialty and, thus, more of a priority. However, for clinicians from other specialties (e.g. surgery, anaesthetics) transfusion was a relatively smaller part of their role and, thus, of lower priority. This translated to mixed willingness to change practice and transfusion practitioners feeling as though they had limited influence over the practice of their colleagues (see Report Supplementary Material 26).

One theme discussed by most participants was the influence of national initiatives on transfusion practice. In 2015, NICE published new guidelines for the assessment and management of blood transfusions. These guidelines were published before the feedback for the surgical audit in trial 1 had been delivered. When participants discussed how they were responding to the feedback, and what recommendations they were working on, most participants reported how this was in response to the NICE guidlines rather than the feedback received from the audit. NICE was generally held in high esteem by clinicians and seen as more credible. Interestingly, the feedback from the NCA was seen as evidence to support the changes that they were already seeking to make based on NICE guidelines, rather than the other way around (see Report Supplementary Material 26).