Prehospital continuous positive airway pressure for acute respiratory failure: the ACUTE feasibility RCT

Trial design

The ACUTE pilot trial was an individually randomised, parallel-group, external pilot trial to determine the feasibility and acceptability of a definitive trial to evaluate the clinical effectiveness and cost-effectiveness of prehospital CPAP compared with standard oxygen therapy for ARF. The pilot trial is reported in accordance with the Consolidated Standards of Reporting Trials (CONSORT) statement and the pilot and feasibility trials extension. 37,38

Trial oversight, ethics and governance

The trial and study documents relating to enrolled participants received ethics approval from the NHS Leeds East Research Ethics Committee (REC) (31 October 2016, reference number 16/YH/0406). All substantial protocol amendments were approved by the NHS Leeds East REC and the Health Research Authority (HRA) before implementation. The University of Sheffield provided sponsorship and monitoring oversight for the project. The chief investigator and trial manager performed day-to-day management of the trial, with support from a Trial Management Group (TMG) consisting of co-investigators. An independent Trial Steering Committee (TSC) and Data Monitoring and Ethics Committee (DMEC) oversaw the safety, conduct and progress of the trial. Funding was received from the National Institute for Health Research’s HTA programme (reference 15/08/40).

Setting

The pilot trial was conducted between 1 August 2017 and 31 July 2018 in the West Midlands Ambulance Service (WMAS), which serves a mixed urban and rural population of 5.6 million. It employs approximately 4000 staff across five divisions and operates from 15 ‘super-hubs’, each covering 5–10 community ambulance stations. The ACUTE trial recruitment took place across four ambulance hubs (Stoke, Stafford, Lichfield and Erdington) and their satellite community ambulance stations, covering a population of 1.5 million. Included hubs were chosen to provide a representative mixture of urban, semiurban and rural localities. Patients presenting to participating EMS ambulance stations were conveyed to the secondary- or tertiary-level hospital closest to the scene of incident. Included ambulance hubs, ambulance stations and hospitals are detailed in Appendix 2.

Participants and eligibility criteria

The trial population consisted of adults transported to hospital by emergency ambulance with ARF, regardless of suspected underlying aetiology. Potential recruits were identified by participating ambulance service clinicians (paramedics and ambulance technicians) after assessment of trial eligibility criteria at the scene of incident, during normal working practice. Prior management in primary care or by a rapid response ambulance clinician did not affect recruitment. ARF was defined as respiratory distress with peripheral oxygen saturation below British Thoracic Society (BTS) target levels (88% for patients with COPD, 94% for other conditions), despite supplemental oxygen (titrated low-flow oxygen for COPD, or titrated high-flow oxygen in other conditions). 14 Potential participants were excluded if any of the following criteria were met:

• hospital CPAP treatment available within 15 minutes of eligibility assessment

• aged < 18 years

• known to have terminal illness

• known pre-existing lack of capacity (confirmed by relatives, carers or documentary evidence, such as lasting power of attorney)

• documented not for resuscitation status

• acutely incapacitated patients with known valid advanced directive declining NIV or participation in research

• the patient has an oxygen alert card

• anticipated inability to apply CPAP (e.g. facial deformity)

• respiratory failure due to chest trauma

• contraindication to CPAP (suspected pneumothorax, respiratory arrest, epistaxis, vomiting or hypotension)

• previous enrolment in the ACUTE trial

• pregnancy

• patient unable to communicate with ambulance service clinicians

• patient with capacity declined consent for participation at the scene of the incident.

Enrolment and consent procedures

Consent procedures were designed in accordance with the Declaration of Helsinki,39 the UK Mental Capacity Act 200540 and Good Clinical Practice guidelines. 41 Potentially eligible participants identified by participating ambulance service clinicians were assessed for mental capacity and approached for enrolment in the trial; the clinicians were guided by a standardised script taking approximately 2–3 minutes to deliver. For patients with mental capacity, verbal consent was obtained for participation prior to enrolment. Eligible patients lacking mental capacity were enrolled in the trial without consent if the treating ambulance service clinician determined that it would be in their best interests. The advice of advanced directives, or appointees with lasting power of attorney, was followed for incapacitated patients, if present. In all cases, a research paramedic reviewed the participant in hospital as soon as possible after enrolment. For patients with mental capacity, the research paramedics provided verbal and written information regarding the trial, and sought written informed consent for further data collection and participation in the trial. In the event that a patient did not have capacity, advice was sought from a personal consultee about their further participation in the trial. 42,43 When a personal consultee was unavailable, a nominated consultee was approached for a consent waiver. 42,43 In the event that a patient lost mental capacity after providing written consent, the existing consent was considered to remain effective, unless a personal or nominated consultee indicated that the incapacitated patient should be withdrawn from the trial. Initially, incapacitated enrolled patients who later regained capacity within the 30-day follow-up period were personally approached by research paramedics to obtain informed consent to continue in the trial. Data collection then proceeded as per the patient’s wishes and details described above.

If a patient died prior to approach from a research paramedic, delayed written consent was not possible. In this event, all non-identifiable data collected prior to death were retained. It was initially planned to collect anonymised 30-day mortality data only. However, following advice from the TSC and DMEC, and after PPI group consultation, an application was made to collect anonymised data on other hospital management and clinical outcomes to provide sufficient information for a full assessment of the safety of the trial interventions. Permission for the collection of these anonymised data was subsequently provided by the HRA Confidentiality Advisory Group and a substantial protocol amendment was approved by the Leeds East REC (substantial amendment 4, 31 July 2018). The ACUTE trial consent procedures are summarised in Figure 1.

FIGURE 1.

Flow chart detailing consent procedures for the ACUTE pilot trial. AE, adverse event.

Interventions

Ambulance service clinicians (paramedics and ambulance technicians) volunteering to participate in the ACUTE trial and trained in trial procedures delivered trial treatments. Participants in the intervention arm were treated by CPAP with supplemental oxygen. Participants in the control arm received standard oxygen therapy. Treatment in both arms was targeted to BTS guidelines for peripheral oxygen saturations. 14 Target peripheral oxygen saturations were 88–92% for participants with known or suspected COPD and 94–98% for participants with other suspected causes of ARF. Ancillary condition-specific treatments were administered in both trial arms in accordance with standard JRCALC practice guidelines. 11

The ACUTE trial used the O-Two unit (O-Two Medical Technologies Inc., Brampton, ON, Canada), a lightweight, open, single-use, low-flow CPAP system, in the intervention arm. 44 This device consists of tubing, which is connected to an oxygen source (either a portable oxygen cylinder or the usual ambulance oxygen flow regulator), and an in-line CPAP unit connecting to a close-fitting face mask. The CPAP unit entrains ambient air to increase local mask pressure, providing resistance for the patient to breathe against. The level of CPAP is varied by altering the incoming oxygen flow rate. Thus, the concentration of inspired oxygen varies according to desired degree of CPAP, as the flow rate is altered. As an open system, with access to ambient air, the device allows unrestricted inspiratory flows and is unaffected by respiratory rate. The O-Two CPAP device is shown in Figure 2.

FIGURE 2.

The O-Two CPAP device. Reproduced with permission from the University of Sheffield, March 2019. O-two CPAP unit; O-Two Medical Technologies Inc.

Reproduced with permission from The University of Sheffield, March 2019. O_two CPAP unit; O-Two Medical Technologies Inc.

Continuous positive airway pressure treatment could be commenced at the site of initial clinical contact or after transfer to an ambulance. An appropriately sized mask was used, with CPAP started at 5 cmH₂O and then incrementally increased by 1 cmH₂O every 2–5 minutes to a maximum of 15cmH₂O, in accordance with BTS peripheral oxygen saturation targets,14 measured by standard pulse oximetry. If necessary, nebuliser treatments could be positioned between the face mask and the O-Two CPAP unit, and sublingual glycerine trinitrate could be given by lifting the face mask. CPAP was continued until arrival at hospital, unless it was not tolerated (e.g. patient request, claustrophobia, anxiety, significant agitation), the patient was unable to maintain their own airway, systolic blood pressure decreased to < 90 mmHg, vomiting occurred, epistaxis occurred, the consciousness level decreased and the patient did not respond to voice, the patient improved or a suspected pneumothorax was detected. In the event of non-compliance with CPAP, treatment with standard oxygen therapy was provided.

In the control arm, oxygen was delivered at normal atmospheric pressure from a compressed gas tank or portable oxygen cylinder, via a flow regulator, to the patient using nasal cannula, an air entrainment Venturi mask, a simple face mask or a non-rebreathing reservoir face mask. The exact choice of flow rate and oxygen delivery device was determined by ambulance service clinicians, depending on the patient’s condition and peripheral oxygen saturation levels.

On arrival at a hospital ED, staff were informed of the trial and current treatments. Patient care was then transferred from ambulance service clinicians to hospital clinicians in accordance to normal practice. Subsequent care followed hospital guidelines, as implemented by the hospital clinician. For the intervention group the hospital clinician could decide to continue CPAP using the O-Two unit, to switch to an in-hospital CPAP or NIV system or to discontinue CPAP altogether. Participants in the control group were able to receive in-hospital CPAP or NIV if indicated, based on assessment by the hospital clinician.

Training

The WMAS ambulance service clinicians volunteered to participate in the trial and were able to enrol participants once they had completed a programme of training. Training was offered by three approaches. First, a 1-day ACUTE trial teaching event was held, consisting of lectures and teaching stations. Second, research paramedics provided individual or small-group teaching, including demonstrations, hands-on familiarisation and scenario-based practice. Third, a series of online training videos could be studied remotely. All of these training methods covered identification of eligible patients, application of the inclusion and exclusion criteria, providing appropriate information, seeking consent, randomisation, delivery of CPAP, monitoring for adverse events (AEs) and data collection. Training specifically focused on trial exclusion criteria, particularly the identification of clinical conditions (e.g. pneumothorax or vomiting), for which administration of CPAP could be harmful. Research paramedics provided ongoing support and education as necessary, including training of any new ambulance service clinicians starting at trial ambulance hubs after recruitment had started.

Randomisation

Enrolled participants were individually allocated to CPAP or standard oxygen therapy in a 1 : 1 ratio, using equipment boxes and simple randomisation constrained by the maximum number of trial devices supplied to the WMAS for trial use (160 boxes: 80 containing CPAP masks, 80 containing standard oxygen masks). This number allowed for trial devices to be available to all trained ambulance service clinicians on duty at any given time, regardless of the number of participants previously recruited. The randomisation sequence was computer generated by an independent statistician not directly involved in the conduct of the trial. The allocation schedule was held centrally on a password-protected, access-restricted network drive. The trial statistician did not have access to the randomisation sequence until after data lock.

Allocation concealment

Continuous positive airway pressure devices and high-concentration oxygen therapy masks were packaged in identical, shrink-wrapped, tamper-proof, sealed trial equipment boxes, measuring 170 mm × 170 mm × 70 mm and weighing 0.52–0.54 kg. All boxes contained a brief letter for the receiving hospital, providing information about the trial and the treatment arm the patient had been allocated to. Intervention arm boxes also contained brief instructions for using CPAP. Stickers summarising trial eligibility criteria and enrolment processes were placed on the exterior of each box. The equipment box is presented in Figure 3.

FIGURE 3.

The ACUTE trial intervention and control arm equipment boxes.

Equipment boxes were assembled, numbered and checked for indistinguishability (including weighing) in an audited process, in accordance with the randomisation sequence at Sheffield Clinical Trials Research Unit (CTRU) by research assistants not directly involved in the conduct of the trial. Boxes were then transferred to a central WMAS storage and distribution centre, where they were held in an access-restricted research store. Boxes were supplied by the WMAS internal distribution team to participating ambulance hubs, and subsequently held locally, unordered by number, in a designated storage area.

Participating ambulance service clinicians selected a single box at the beginning of each shift, regardless of box number. At the end of the shift, boxes were returned to the ambulance hub equipment store. Boxes were signed in and out for each shift with personnel, ambulance and equipment box details recorded in a distribution log in each hub. Participating ambulance service clinicians identified potential participants with ARF when attending emergency 999 ambulance calls. Immediately after enrolment, paramedics opened the trial equipment box and provided treatment according to whether a CPAP device or high-concentration oxygen mask was supplied. It was not possible to reseal the box. Research paramedics monitored the location and condition of all boxes, allocation concealment (i.e. boxes had not been tampered with) and adherence to the allocation schedule on a weekly basis. A detailed audit log was completed recording this information.

Blinding

Because of the physical differences between the CPAP device and the standard oxygen mask, it was not possible to blind patients, ambulance service clinicians or hospital clinicians to the treatment arms. Research paramedics assessing outcomes were also not blinded. However, both feasibility and clinical end points were objective measures, minimising the possibility of information bias.

Data collection

A recruitment form [case report form (CRF) A], contained in each trial equipment box, was completed by ambulance service clinicians every time a patient was enrolled in the trial. This collected trial-specific information, including trial number, patient identifiers, eligibility criteria, suspected prehospital diagnosis, consent details, patient-reported and clinician-assessed VAS dyspnoea (1–10) score and prehospital clinical and treatment data. The information contained in CRF A was relayed to the central WMAS clinical support desk telephonically and stored centrally on a specific trial database. Routinely collected baseline characteristics, EMS timings, details of treatments provided and vital signs (including peripheral oxygen saturations) en route to hospital were extracted later from WMAS electronic patient records (EPRs) and entered into trial CRF B by research paramedics.

Baseline quality-of-life assessments were performed by research paramedics shortly after hospital admission, following confirmation of patient consent for participation in the trial. Patients, or their representatives, were asked to estimate their current health status, using the EQ-5D-5L. 45 At 30 days, research paramedics reviewed the hospital records to collate details of subsequent progress, inpatient treatments provided (including provision of hospital CPAP/NIV), length of hospital stay, use of critical care, any AEs and vital status at 30 days. These data were also recorded in CRF B.

Quality of life and resource use were assessed by questionnaire at 30 days following enrolment, either in person, if still in hospital, or by telephone or post if discharged. Vital status was checked using hospital and summary care records prior to approach. Participants were asked for their preferred method for data collection, either telephone or post. Initial non-responders were contacted again after a further 2 weeks. Key elements of health-care resource recorded included hospital services and general practitioner or community services. Participants were also asked to report any AEs in the 30-day follow-up questionnaire.

Following review of blinded outcome data during TMG meetings, an unexpectedly high mortality rate was noted. Supported by the TSC and DMEC, a post hoc descriptive analysis of the deceased patients was therefore planned (substantial amendment 4, 31 July 2018). Local trial collaborators were contacted by research paramedics and asked to review clinical records to determine if a ceiling-of-treatment decision was made during the patient’s admission. Anonymised data were coded and stored using Excel^® (Microsoft Corporation, Redmond, WA, USA). The assessments and follow-up for the ACUTE trial are summarised in Appendix 3.

Data management

All data were collected and retained in accordance with the UK Data Protection Act 1998,46 the European Union General Data Protection Regulation 2016/67947 and the University of Sheffield CTRU standard operating procedures. Trial data were extracted from the WMAS clinical support desk database, source documents and CRFs by research paramedics, and entered onto a secure CTRU data management system. Patient-identifiable data (names, date of birth and contact details) were collected and entered only when written informed consent was confirmed. Validation reports were run regularly to check the data for completeness, accuracy and consistency. Any data discrepancies were monitored and managed to resolution by research paramedics.

Outcomes

The following feasibility outcomes and targets were prespecified:

• recruitment rate per 100,000 persons per year (target eight, i.e. 120 across the 1.5 million population of the four WMAS hubs)

• proportion recruited in error and classified as minor or major non-compliances (target 0% and ≤ 10%)

• adherence to the allocation schedule (target ≥ 90%)

• adherence to treatment in the CPAP arm (target ≥ 75%)

• retention at 30 days (target ≥ 90%)

• data completeness (target ≥ 90%).

Secondary effectiveness outcomes were:

• proportion surviving to 30 days

• proportion undergoing endotracheal intubation by 30 days

• proportion admitted to critical care at any point up to 30 days

• mean and median lengths of hospital stay

• change in VAS dyspnoea score from initial presentation to immediately before ED arrival

• mean EQ-5D-5L score

• key elements of health-care resource use up to 30 days.

Safety reporting

Adverse changes in the health of ACUTE trial participants were defined, monitored, recorded and reported in accordance with CTRU standard operating procedures and HRA guidance for non-Clinical Trial of an Investigational Medicinal Product studies. Only AEs and serious adverse events (SAEs) related to trial interventions or procedures were recorded. The following SAEs were expected to occur following ARF and were recorded as outcomes: death, hospitalisation, intubation and ventilation, or admission to critical care. Expected related SAEs from CPAP administration were prespecified as pneumothorax, aspiration, hypercapnia, progressive respiratory failure or hypotension.

Adverse health changes were identified by ambulance service clinicians during delivery of trial treatments, local hospital trial collaborators, trial participant reports, research paramedics’ review of medical records at 30 days or by the trial manager from the 30-day follow-up questionnaire. Adverse health changes were assessed and classified by an appropriately qualified member of the ACUTE trial research team or a local clinician, and reported to the chief investigator, who reviewed all relevant documentation to determine causality, severity and expectedness to determine whether or not the AE was related and/or was unexpected. Reporting requirements depended on the seriousness of the adverse health change and categorisation as expected or unexpected: related AEs and expected related SAEs were regularly reported in aggregate to the TMG, TSC, DMEC, sponsor and REC. Unexpected SAEs related to trial interventions or procedures required expedited reporting to the sponsor, relevant NHS research and development department, REC and DMEC within 15 days of notification, using the HRA SAE form. Full details of safety reporting are provided in Appendix 4, Table 21 and Figure 19.

Sample size

The trial was an external pilot trial intended to explore the feasibility of conducting a future definitive trial. The sample size for a feasibility trial should be adequate to estimate the uncertain critical parameters needed to inform the design of the full randomised controlled trial (RCT) with sufficient precision. 48 Mortality at 30 days would be the primary end point of a definitive trial. Mortality under standard care was estimated at 12%, and, for a full trial, a 5% absolute reduction was postulated (i.e. to 7%) in the prehospital CPAP intervention arm. 26 Given the short follow-up period, loss to follow-up of < 5% at 30 days was envisaged.

A minimum sample size of 120 was proposed by Teare and colleagues49 for pilot studies with dichotomous outcomes to provide sufficient precision to measure binary parameters for use in the sample size calculation of the full trial. A pilot trial sample size of 120 would therefore allow mortality to be estimated to within a standard error of 2.7% for use in the sample size calculation of an eventual large-scale trial. This sample size also allowed estimation of feasibility outcomes with a precision of < 5%.

A previous evidence synthesis study3 estimated that the incidence of eligible cases ranges from 3.5 to 40.8 per 100,000 persons per year. The lowest estimates were based on actual patients treated with prehospital CPAP in services with limited ability to deliver treatment for all eligible patients and are likely to be underestimates. The highest estimates were based on audit data for in-hospital NIV use among emergency admissions and are likely to be overestimates. Assuming that there are 20 eligible cases per 100,000 persons per year and that 40% of these were recruited, we anticipate that 120 patients would be recruited from the trial’s source population of 1.5 million, over 1 year.

Statistical analyses

Participant recruitment and retention are described and summarised in a CONSORT flow diagram (see Figure 6). 38 Missing data are described for each variable by indicating the number and percentage of observations present, and reasons for missing data. Available-case analyses were performed with casewise omission in the event of missing data. Owing to the relatively small number of missing data, and the focus on feasibility rather than effectiveness, sensitivity analyses for missing data were not performed. Non-normally distributed continuous variables were summarised using the number of observations, median and interquartile range (IQR). Categorical variables were evaluated using the number of observations and percentages. All analyses were conducted in the R statistical package (The R Foundation for Statistical Computing, Vienna, Austria) in accordance with a prespecified statistical analysis plan and CONSORT principles.

The baseline characteristics, prehospital treatment, hospital management and AEs of enrolled participants are reported descriptively for the whole trial population, and separately per treatment arm, using an as-randomised analysis set (i.e. participants were analysed according to their randomisation, regardless of compliance or completeness of follow-up) with the denominator identified throughout. Feasibility outcomes are reported descriptively for the whole trial population, together with their 95% confidence interval (CI) (calculated using the Wilson score interval),50 using a full analysis set. Pilot trials are not hypothesis-testing studies, and safety, efficacy and effectiveness results should, therefore, be interpreted cautiously. 51,52 Consequently,only summary estimates of relative effectiveness outcomes are presented, overall and stratified by treatment arm, without 95% CIs or p-values. An intention-to-treat, full analysis set was used for clinical end points with complete follow-up (i.e. all randomised participants included in the group to which they were randomly assigned, regardless of their adherence with the entry criteria, the treatment they actually received and deviation from the protocol). A modified intention-to-treat, complete-case analysis was performed for other clinical end points with missing outcome data (i.e. randomised participants with complete outcome data included in the group to which they were randomly assigned, regardless of their adherence to the entry criteria, the treatment they actually received and deviation from the protocol). 53

Patient and public involvement

The public and patients were fully involved in the ACUTE trial from conception to dissemination. The research proposal was developed in partnership with a service user co-applicant and was reviewed in terms of feasibility and relevance by the Sheffield Emergency Care Forum, the Sheffield Cardiovascular Patient Panel and the Barnsley Patient Advisory Group. A PPI bursary from the Yorkshire and Humber Research Design Service was used to consult with a group of respiratory patients about their views on the research proposal, consent procedures and how best to involve patients throughout the project. Feedback from these groups led to a number of important changes, for example reducing the number of patient questionnaires. A service user advisory group was enlisted during the trial set-up period for collaboration throughout the project and helped develop patient-facing research materials. A member of the Sheffield Emergency Care Forum (MM) was a co-applicant who provided PPI advice on trial matters, attended TMG meetings, supported trial management, and contributed to the trial report and interpretation of results. Two further lay PPI representatives served separately on the TSC and the DMEC. Asthma UK and PPI co-applicant Margaret M Marsh helped write the Plain English summary. Findings from the ACUTE trial were reported to our partner PPI groups and presented at a University of Sheffield research engagement event.

Decision problem

The decision problem, informed by the clinical research question posed in the ACUTE pilot trial, was ‘Which is the most cost-effective treatment strategy for patients presenting to NHS ambulance services with ARF?’.

The aim of the ACUTE trial economic evaluation was, therefore, to evaluate the cost-effectiveness of prehospital CPAP compared with standard care for patients with ARF in the UK. Specific objectives were to:

• estimate the cost-effectiveness of prehospital CPAP compared with standard care for patients with ARF, in terms of the costs and QALYs gained by each treatment strategy

• identify the strategy that is most likely to be cost-effective for patients with ARF, defined as the most cost-effective strategy at a willingness-to-pay threshold of £20,000 per QALY gained

• evaluate the cost and value of undertaking further research by estimating the EVPI

• identify the critical areas of uncertainty in which future research would produce most benefit, by calculating the expected value of partial perfect information (EVPPI) for different parameters.

Interventions

Any potentially relevant prehospital treatments that could be feasibly be implemented in the NHS for ARF were considered. However, owing to the complexity of alternative forms of NIV, only CPAP was judged as being a practicable alternative to standard oxygen practice in UK ambulance services. Therefore, interventions comprised prehospital CPAP provided by ambulance service clinicians and standard oxygen therapy (i.e. without prehospital CPAP). Hospital management was assumed to be identical for both comparators.

Form of economic evaluation

A cost–utility economic evaluation was performed using a probabilistic decision-analytic model to synthesise available evidence and compare alternative management strategies. 54,55 Such models offer a framework to systematically, transparently and objectively collect all available information on a particular decision problem. Relevant evidence can then be synthesised and translated by modelling into estimates of costs and effects, along with an indication of the uncertainty surrounded these estimates. This allows identification of the most cost-effective treatment options and facilitates an assessment of the benefit of performing future research. The ability of decision-analysis models to consider all important aspects of a decision problem contrasts with trial-based economic evaluations, which are often limited by omission of relevant treatment options, exclusion of important external evidence, failure to capture long-term differences in economic outcomes and inclusion of non-representative populations or treatment regimens. 56

Model development and scope

A cohort model, structured as a decision tree, was used to estimate the cost–utility of alternative treatment approaches. 54 The decision tree structure was based on a previously published economic model used for a HTA programme-funded evidence synthesis project, with updated parameter values. 3 Base-case principles for economic evaluations outlined in the National Institute for Health and Care Excellence (NICE) Guide to the Methods of Technology Appraisal57 were followed. The economic perspective was the NHS in England and Wales, with only direct treatment costs included. The model employed a life time horizon.

Costs and consequences

Direct treatment and Personal Social Services costs were included. The price base was assumed to be 2018; valuations were in Great British pounds and unit costs were considered to be time divisible. When unit costs were valued prior to 2018, the Bank of England’s Consumer Price Index data were used to inflate costs to current value. 58 All costs were applied using present values; discounting was not performed.

The consequences of alternative management strategies were measured in QALYs to allow comparison within and across different disease areas. 59 QALYs were calculated by multiplying survival duration with an appropriate mean utility value.

Model structure

Differences between management options were accounted for by designating each chance node with a strategy-specific probability, and by assigning differing costs and utility values to the terminal nodes of each individual subtree branch. Expected costs and QALYs for each strategy were subsequently calculated by summation of the terminal node values, weighted by the conditional branch probabilities. 54,56

Patients with ARF are at increased risk of short-term mortality and intubation. 6 Therefore, the decision tree assigned a baseline probability of intubation or death within 30 days for the standard care arm. Log-ORs for mortality and intubations were used as effectiveness parameters in the model for prehospital CPAP, and applied to the baseline risks to give intervention arm probabilities. If the baseline risk is P, then µ is estimated as logit(P) = log[P/(1 – P)]. The absolute probabilities for the intervention (CPAP) are then estimated as:

in which d is the log-OR for an intervention relative to standard care.

Lifetime QALYs were accrued by patients who survived their initial ARF presentation. The model estimated prognosis by using a 30-day probability of death and probabilities of intubation depending on the type of treatment. Survivors (i.e. those who avoided the short-term 30-day mortality risk) accrued QALYs estimated from life expectancy and their utilities for this time period. It was assumed that the lifetime QALYs were the same for all survivors, irrespective of whether they were in the standard care or prehospital CPAP arm.

The cost of standard care was assumed as £0. This simplification was made as the analysis is based on incremental costs (i.e. it was assumed that all initial treatment costs are the same, regardless of whether or not the patient receives prehospital CPAP). 54 The zero costs for standard care relate only to prehospital and ED treatment and do not include hospitalisation costs, intubation costs or additional lifetime costs for survivors. This was deemed sensible by the clinical experts on the TMG, as it was assumed that the proportion of participants who would receive NIV in hospital were similar in both arms, irrespective of whether or not a participant received prehospital CPAP.

Thus, the only difference in initial treatment costs between standard care and prehospital CPAP was the additional cost of providing prehospital CPAP. This was determined by sharing out the initial and ongoing equipment and training costs among the number of patients who would benefit (i.e. the incidence of ARF patients eligible for CPAP), based on a 5-year depreciation period (i.e. assuming new prehospital CPAP equipment will be required in 5 years).

Subsequent hospital treatment costs were dependent on the probability of needing intubation. Lifetime health-care costs accrued among survivors according to their life expectancy. It was assumed that the lifetime costs were the same for all survivors, irrespective of whether they were in the standard care or prehospital CPAP arm. The model structure is shown in Figure 4.

FIGURE 4.

Structure of the ACUTE trial decision-analytic model.

Model population and setting

The population consisted of a hypothetical cohort of patients with ARF due to any cause and potentially suitable for CPAP treatment. The setting was a representative NHS ambulance service, such as WMAS. Although this cohort could include patients with heterogeneous aetiology for ARF, including acute cardiogenic pulmonary oedema/heart failure, COPD and pneumonia, for the purposes of modelling they were treated as a single group.

Parameterisation

The identification, appraisal and selection of evidence to determine model inputs have been previously described in detail,3 but were updated for this analysis. When possible, parameters, ideally valid evidence from systematic literature reviews or routine official data sources deemed to be at low risk of systematic error, were used from the previous HTA model. 3 When relevant and unbiased published evidence was unavailable, parameters were updated using ACUTE trial pilot data or expert opinion of clinical experts from the ACUTE TMG. Each model input was assigned an average or most likely value, and a probability distribution representing a credible range and the relative likelihood of possible values for the uncertainty in this estimate was defined. Distributional choices were carefully chosen based on theoretical considerations, logical constraints and the parameter estimation process. 60 As model inputs were derived from alternative sources, with no data available on the covariance structure, it was not possible to account for any correlation between costs and outcomes. 56 The sources, mean values and distributions for each parameter are described in detail with the economic model results in Chapter 3, Economic analysis results.

Cost-effectiveness analysis

The cost-effectiveness of the different interventions was estimated using both the ICER and the net monetary benefit (NMB) approaches. 61,62 The ICER measures the relative value of two strategies and is calculated as the mean incremental cost divided by the mean incremental benefits, computed by comparing to the next most effective alternative. A strategy is dominated when another strategy accrues more QALYs at a lower cost. The willingness-to-pay threshold (λ) is the amount of money that the decision-maker is willing to pay to gain 1 additional QALY. 63 The usual threshold for decision-making at NICE is considered to be £20,000 per QALY, as detailed in NICE HTA guidelines. 57 The NMB framework transforms cost-effectiveness results to a linear scale and simplifies interpretation of cost-effectiveness results for decision-makers. NMB is defined as the number of QALYs multiplied by a value for the QALYs (e.g. £20,000) minus the costs of obtaining them [i.e. NMB = (QALYs × λ) – cost]. The strategy with the highest expected incremental NMB is the most cost-effective. 54,55

To account for the uncertainty in model inputs, a probabilistic sensitivity analysis (PSA) was conducted using Monte Carlo simulation to randomly sample from the inverse cumulative distribution function of each model parameter’s probability density functions. 54,55,64 Multiple model runs were performed, each with a random draw from every parameter’s distribution, thus evaluating the full range of cost-effectiveness results possible, given current uncertainty on the true values of model inputs. Mean ICERs calculated from the average expected costs and effects over all model runs were computed and compared with cost-effectiveness thresholds to inform adoption decisions. The number of PSA iterations to produce a stable mean estimate of incremental cost–utility was determined by visual inspections of the mean cost per QALY plotted against the number of trial PSA simulations. 56

The incremental costs and QALYs of each model run were also depicted graphically on a cost-effectiveness plane. The cost-effectiveness plane shows the incremental costs (y-axis) and incremental QALYs (x-axis) compared with usual care. In this chart, if a model run for a strategy had exactly the same costs and QALYs as usual care, then the ‘sample’ for that model run would appear at the origin. Samples plotted to the right of the y-axis have more QALYs than usual care and samples plotted above the x-axis have more costs. Samples plotted to the right of a straight line with slope λ passing through the origin are cost-effective, whereas those plotted to the left are not. The mean NMB was also calculated for a defined threshold value of λ = £20,000. 54,55

A cost-effectiveness acceptability curve, plotting a relevant range of λ values against the probability that each intervention was the most cost-effective, was additionally graphed to summarise the uncertainty of PSA results. 65 The cost-effectiveness acceptability curve is derived from the joint density of individual incremental costs and incremental effects calculated for competing treatment options during a PSA and shows the proportion of model runs for which each strategy is cost-effective over a range of potential willingness-to-pay thresholds (i.e. λ). The probability of a treatment being the most cost-effective corresponds to the proportion of the joint density of incremental costs and effects that result in the highest NMB at a given λ.

Model uncertainty and sensitivity analyses

Uncertainty can arise in decision models due to variability, heterogeneity, parameter uncertainty and structural uncertainty. Variability arises from random differences between individuals with similar characteristics. 66 Heterogeneity refers to the differences in costs and effects explained by particular patient characteristics. Parameter uncertainty will occur as model inputs are estimates, with a probability distribution denoting the relative likelihood of alternative values. Finally, structural uncertainty relates to the assumptions imposed by the structure, scope or other methodological decisions taken during the modelling process.

As a cohort model was implemented with examination of mean values, an examination of variability is extraneous. Furthermore, a prehospital CPAP service is a population-level intervention, which will be implemented for any patient presenting with suspected ARF, meaning that examination of heterogeneity is superfluous. Parameter uncertainty (i.e. uncertainty surrounding the true value of a model input within the specified probability distribution) was fully explored in the PSA. 56,64

To examine structural uncertainty and ensure that the correct statistical form has been specified for the probability distribution of important parameters, a number of sensitivity analyses were planned. Two different scenario analyses were performed using different effectiveness parameters. In the base case, results from the ACUTE pilot trial were used. The ACUTE pilot trial was not designed to test effectiveness, but the clinical experts felt that it provided the only genuinely representative data for routine NHS practice. It was felt that an imprecise but representative estimate of effectiveness was more appropriate for the base case than a more precise estimate based on meta-analysis that included selected populations and practice that differed markedly from the UK. A scenario analysis was undertaken in which the effectiveness parameters were based on a network meta-analysis synthesising previously published experimental data, updated with results from the ACUTE trial using identical methods to those previously reported. 3 Included studies were predominantly from non-UK settings, and the pooled effect estimate may reflect the efficacy of CPAP achievable in more developed EMS systems.

The incidence of ARF was identified as an influential parameter in the original economic analysis and a further scenario analysis was planned to explore the effect of extreme estimates. 3 However, developments in CPAP technology dramatically reduced the cost of providing prehospital CPAP from that estimated in the previous analysis (see Chapter 3, Prehospital costs),44 thus removing the potential means by which the incidence of ARF could influence cost-effectiveness. Therefore, we did not proceed with this scenario analysis.

Expected value-of-information analysis

Using estimates of the probability of making the wrong decision, together with the ensuing opportunity costs of error, the expected opportunity loss surrounding a decision can be estimated. 54,56,67 A rational decision-maker, aiming to maximise health within a fixed budget, should be willing to spend up to this value for additional evidence to remove decision uncertainty, a figure termed the population expected value of perfect information (i.e. EVPI). This can be thought of as the maximum that the health-care system should be willing to pay for additional evidence to inform the decision in the future. 54,67

The population EVPI places an upper limit on the total value of additional research relating to a specific decision problem, but does not indicate where future research may be beneficial. The population EVPPI is the difference between expected value with perfect and current information about particular model inputs, for all future patients. There will be a large value in improving the precision of estimates of parameters, or groups of parameters, with high EVPPI, which may suggest using particular study designs (e.g. commissioning a cohort study to obtain estimates of disease incidence or a clinical trial to investigate effectiveness). 54,67,68

Individual-level expected value-of-information metrics were initially calculated for both the base-case and updated meta-analysis scenario analyses. Assumptions on ARF incidence (11,000 patients per year in England and Wales), discount rates (3.5%) and health technology lifespan (5 years) were then used to compute population-level statistics. EVPI for individual patients was calculated directly from the model PSA output using standard formulas. 54 Individual EVPPIs were estimated by using two-level Monte Carlo simulation techniques. 68 Ten parameters, reflecting targets for potential future research designs, were considered in EVPPI analyses: baseline mortality, baseline risks, relative effectiveness for mortality, relative effectiveness for intubation, cost of prehospital CPAP, cost of hospitalisation, cost of intubation, cost of long-term survival, life expectancy and lifetime quality of life.

Model implementation

The decision-analytic model was programmed in the R statistical package. Internal testing was performed throughout model development to ensure that mathematical calculations accurately represented model specifications and were correctly implemented. Debugging techniques included reimplementation in Excel, null and extreme input values; setting equal values across comparators; fixed distributions; and line-by-line checking of syntax. Model validation was performed by comparing model outputs with published estimates from the ARF literature,6,8 including the costs and life expectancy predicted from the model. 69–71

Recruitment

The Stoke ambulance service hub commenced recruitment as planned on the 1 August 2017. Owing to delayed research approvals, the remaining ambulance hubs became active later (Stafford, 17 August 2017; Lichfield, 11 September 2017; and Erdington, 6 October 2017). Across the hubs, 204 ambulance service clinicians, representing 13% of available staff, completed trial training. Seventy-three staff attended the face-to-face ACUTE trial training event and 131 completed the online educational package.

Over the recruitment period up to 30 August 2018, approximately 364 patients with ARF meeting the ACUTE trial eligibility criteria presented from the 1.5 million population of four participating WMAS hubs. Of those patients, 161 (44.2%) were attended by ACUTE trial-trained ambulance service clinicians and could potentially have been recruited. Of these 161 patients, 77 (47.8%) were enrolled in the trial, by 41 individual ambulance service clinicians, equating to a recruitment rate of 5.1 per 100,000 persons per year (95% CI 4.1 to 6.4 per 100,000 persons per year). The remaining 84 patients (52.2%) presented to participating ambulance service clinicians, but were not enrolled. Although the attending clinician had signed out an ACUTE trial equipment box, the patient was not enrolled in 18 of these cases, for unknown reasons. In the remaining 66 cases, the participating ambulance service clinician had failed to carry an ACUTE trial equipment box and was therefore unable to recruit an eligible patient. This occurred either because the clinician forgot to collect an equipment box or because they withdrew from participation in the study. No patients assessed to have capacity at the scene and invited to enrol in the trial declined verbal consent for participation.

The recruitment rate was broadly similar across the four participating hubs. Enrolment showed a marked seasonal variation, with an increased rate between October and February and fewer patients included over spring and summer months. Overall, recruitment did not meet the feasibility target of 120, with a much lower enrolment rate than target over the closing summer months of the trial. The target and actual recruitment figures are shown in Figure 5, corrected for the staged start in hub activity. Recruitment by the four hubs by month is shown in Table 1.

FIGURE 5.

Recruitment rate by month compared with the target rate.

Allocation concealment

Weekly audit of trial boxes was unremarkable until week 10 of recruitment when it was noted that some of the intervention arm boxes had begun to ‘rattle’. Further examination revealed that the CPAP mask had been packaged under tension and, over time, the cushion seal had slightly deflated. All equipment boxes were immediately returned to the CTRU, replaced using a revised technique that no longer placed the CPAP masks under pressure and redistributed to WMAS. The original randomisation schedule was used and box assembly otherwise followed the initially used protocol. For the remainder of the trial, no issues were noted during box audits. A post hoc allocation concealment study was subsequently conducted as described in Chapter 2, Allocation concealment study, to explore the robustness of randomisation using equipment boxes.

Participant flow

The majority of patients were able to provide verbal consent for participation in the trial (59/77, 76.6%); however, 18 patients (23.4%) were incapacitated at the scene of incident and enrolled after a best-interests assessment by the attending ambulance service clinician. All patients were transported to hospital. Informed consent for further participation was confirmed by research paramedics for 51 patients (66.2%, 46 individual patient consent, five personal consultee). This allowed retention of prehospital data, collection of hospital and clinical outcome information, and invitation to complete the 30-day quality-of-life and resource use follow-up questionnaire. None of these patients withdrew from the trial at a later date due to subsequent removal of consent, losing capacity and a consultee declining further participation, or regaining capacity and over-ruling preceding consultee consent.

Consent for further data collection was declined by nine patients (11.7%), resulting in retention of prehospital data and collection of anonymised 30-day mortality data only. Seventeen patients (22.1%) died before a research paramedic could approach them for consent. In accordance with research approvals, prehospital data were retained, and anonymised collection of hospital and clinical outcome information was conducted; however, 30-day questionnaire follow-up was not possible. ED and inpatient information were unavailable for two patients owing to absence of research approvals to access data, secondary to the reorganisation and merger of one hospital. Of the 51 participants who were consented for follow-up, 41 completed the questionnaire at 30 days (80.4%). Four participants died between consent and follow-up and six participants did not complete the questionnaire despite providing consent for follow-up.

Figure 6 presents the participant flow for the trial, with follow-up figures presented at each relevant time point. Table 2 summarises the consent, data collection and follow-up.

FIGURE 6.

Consolidated Standards of Reporting Trials (CONSORT) diagram of participants’ flow through the ACUTE pilot trial. DNAR, do not attempt resuscitation.

Baseline characteristics

A small number of data were missing for baseline characteristics, secondary to not being recorded in CRF A or routine EPRs, including clinician’s assessment of patient’s breathlessness (1/77), systolic blood pressure (7/77), diastolic blood pressure (7/77), peripheral oxygen saturations (1/77) and pulse rate (2/77). The final hospital diagnosis was missing for 12 patients (nine patients declining consent for data collection, one patient for whom no clear primary diagnosis was apparent and two patients for whom clinical records were unavailable).

Of the 77 recruited patients, slightly more participants were randomised to the CPAP intervention arm (42 cases), than to the standard oxygen control arm (35 cases). The trial population was elderly (median age 71 years), predominantly male (62.3%) and severely unwell (median VAS breathlessness score 9/10, median pulse 115 beats per minute, median respiratory rate 34 breaths per minute and median initial peripheral oxygen saturations of 78.5%). The most common final primary diagnoses were COPD (21/65, 32.3%) and LRTI (28/65, 43.1%). A minority of cases (4/65, 6.2%) had non-respiratory primary diagnosis, comprising abdominal aortic aneurysm, myocardial infarction, sepsis (not further specified) and liver failure (ascites). Secondary conditions accounting for ARF were diagnosed in 27 out of 65 patients (41.5%), with COPD (16/65) and LRTIs (14/65) being the most common concomitant conditions. Pre-randomisation characteristics were similar across trial arms, as summarised in Table 3.

Delivery of interventions

Continuous positive airway pressure was fully delivered as planned (i.e. administered until hospital arrival or discontinued as a result of patient improvement after successful treatment) in 73.8% (31/42) of intervention arm participants. CPAP was commenced in 90.5% of participants (38/42), with two participants refusing to wear the mask, one participant spontaneously improving and a fourth participant having a cardiac arrest prior to commencement. Of participants commencing CPAP, 31 (81.6%) continued with CPAP until they arrived at hospital (six did not tolerate CPAP and the remaining participant was transferred to a standard oxygen non-rebreather mask owing to non-improvement). CPAP administration in the intervention arm is detailed in Table 4.

Standard oxygen therapy was delivered using a range of devices, either individually or in combination, consisting of bag–valve–mask apparatus, high-flow non-breather masks, Venturi masks, nebuliser masks and, in one case, the patient’s own CPAP machine. High-flow non-breather mask used alone was the commonest treatment (10/35, 28.6%), with treatment with various combinations of non-breather mask, Venturi mask and nebuliser mask also being common. Oxygen therapy in the control arm is summarised in Table 5.

No participants underwent intubation during the prehospital interval, but 6 of 77 participants (7.8%, two in the CPAP arm and four in the control arm) required assisted bag–mask ventilation. Ancillary disease-specific treatments were administered to 61 of 77 (74.3%, 35 in the CPAP arm and 26 in the control arm) participants. Table 6 summarises the additional treatments administered by ambulance service clinicians.

Arterial blood gas results, sampled in the ED after arrival, were recorded for 45 participants. Data were unavailable for 14 cases (nine participants declined consent, two participants had absent research approvals, and notes were unavailable for three participants). For the remaining 18 cases, an arterial blood gas procedure was not documented in the clinical records. The median values indicated type II respiratory failure, with acidaemia secondary to uncompensated acute respiratory acidosis. ED NIV or CPAP was performed on a minority of participants [13/52 (25.0%) data were missing for 25 participants]. Tables 7 and 8 details ED investigation and management.

Feasibility outcomes

Feasibility outcomes, compared with the prespecified target, are summarised in Table 9.

Recruitment rate

The sample size of 77 enrolled participants resulted in a recruitment rate of 5.1 (95% CI 4.1 to 6.4) per 100,000 persons per year, falling short of the feasibility target of 8 per 100,000 persons per year (i.e. 120 patients).

Major or minor non-compliances

All participants were recruited appropriately, with no major protocol non-compliances. A single minor protocol non-compliance (1/77, 1.3%) was recorded when it was reported that a control arm standard oxygen mask did not work, necessitating substitution of another mask. However, when formally tested by the supplier, no abnormality was found. This compared favourably with feasibility targets for minor or major non-compliances of ≤ 10% and 0%, respectively.

Adherence to the allocation schedule

There was full adherence to the allocation schedule (feasibility target ≥ 90%). Treatment with CPAP was attempted in all participants enrolled in the intervention arm. All participants received appropriate standard oxygen management in the control arm, although one participant was enrolled who used their own CPAP machine.

Adherence to treatment in the continuous positive airway pressure arm

Continuous positive airway pressure was fully delivered as planned (i.e. administered until hospital arrival, or discontinued because of patient improvement after successful treatment) in 74% (31/42) of participants in the intervention arm, as described in Delivery of interventions. This compared favourably with the feasibility target of 75%.

Retention at 30 days and data completeness

Full data were available for key outcomes, including all feasibility end points and vital status at 30 days, which meets the feasibility targets of ≥ 90% retention at 30 days and ≥ 90% data completeness. A small number (< 1%) of prehospital data describing baseline patient characteristics were not available because of missing values. The difference in clinician-reported VAS dyspnoea scores was unavailable for one participant secondary to prehospital cardiac arrest (n = 77, 1.3%).

A larger proportion of hospital data were missing secondary to lack of consent for collection (n = 9, 11.7%), absence of local research approvals to access hospital data (n = 2, 2.6%) or unclear or absent information in the hospital clinical records [differing across variables, ranging from n = 0 (0%) for hospital length of stay to n = 25 (32.5%) for ED management]. Across the prespecified secondary hospital outcomes, data were missing for:

• endotracheal intubation by 30 days (15/77, 19.5%)

• admission to critical care at any point up to 30 days (12/77, 15.6%)

• mean and median lengths of hospital stay (11/77, 14.3%).

Baseline EQ-5D-5L score was unavailable for 26 participants (33.8%) in total: for nine (11.7%), there was a lack of consent for collection, and the remaining 17 (22.1%) died prior to research paramedic approach for measurement. Of participants alive at 30 days (n = 56), 30-day follow-up questionnaires examining quality of life and post-discharge health resource use were fully completed by 40 participants (71.4%), with partial completion by one participant of health resource use only. Consent for further data collection was declined by nine participants and a further seven participants did not complete the questionnaire, despite providing consent for follow-up.

Effectiveness outcomes

Secondary effectiveness outcomes are summarised in Table 10.

Proportion surviving to 30 days

Overall mortality of the trial population was higher than expected, with 27.3% (21/77) of participants dying by 30 days. 3 A total of 28.6% (12/42) of participants in the CPAP arm and 25.7% (9/35) in the standard oxygen arm (expected 12%) died by 30 days. Two participants in each arm died on the day of enrolment, with the remainder dying between 1 and 28 days post hospital admission. The median time to death among those who died was 4 days overall (2.5 days for CPAP arm and 10 days for the standard oxygen arm). Survival of enrolled participants, stratified by trial arm, is presented using a Kaplan–Meier graph in Figure 7.

FIGURE 7.

Kaplan–Meier plot showing post-randomisation survival. Line represents survival curve estimate. Shaded areas represent 95% CI.

An unplanned descriptive analysis of the deceased participants was undertaken to explore the circumstances for the unexpectedly high mortality rate. Data were not available for two cases owing to lack of research approvals to access data, and two participants died from non-cardiorespiratory conditions not amenable to NIV (ruptured abdominal aneurysm and liver failure). Of the remaining 17 cases, six participants (35.3%) received hospital NIV (5/17, 29.4%) or mechanical ventilation (1/17, 5.9%) and the other 11 participants had explicit ceiling-of-treatment decisions documented in hospital that excluded hospital NIV and critical care management (5/17, 29.4%), or appeared to have implicit limits on treatment level and died without NIV or critical care intervention (6/17, 35.3%).

Proportion undergoing endotracheal intubation by 30 days

The overall risk of intubation was low, occurring for only three of the 62 participants with follow-up data (4.8%). Two participants in the CPAP arm were intubated (n = 33, 6.1%), and one participant in the standard oxygen therapy arm was intubated (n = 29, 3.4%).

Proportion admitted to critical care at any point up to 30 days

A small proportion of participants were admitted to higher levels of care, with six participants receiving critical care-based management (n = 65, 9.2%). Four participants in the CPAP arm (n = 35, 11.4%) received intensive care unit care, with two of these also receiving high-dependency or respiratory support unit treatment. The median length of critical care stay for these participants was 6.5 days (IQR 4.0–8.75 days, range 1–11 days). Two out of 30 participants in the standard oxygen arm (6.7%) were managed entirely in an intensive care unit, with no preceding/subsequent level 2 critical care. The median length of critical care stay in the control arm was 15 days (IQR 11.0–19.0 days, range 7–23 days).

Mean and median lengths of hospital stay

Overall, 44 out of 66 participants were discharged after their initial admission (66.7%); 17 participants died during their index presentation (25.8%) and five participants remained in hospital for the duration of the 30-day follow-up period (7.8%). The overall median length of initial stay was 8 days (IQR 5.8–12.0 days, range 1–28 days). Length of stay was similar for both trial arms, with a median of 10.0 days (IQR 6.5–12.0 days) in the CPAP arm and 7.0 days (IQR 5.0–9.8 days) in the standard oxygen therapy arm.

Change in visual analogue scale dyspnoea score from initial presentation to immediately before emergency department arrival

Breathlessness, evaluated by patient-reported and clinician-assessed VAS score, was measured at enrolment and after arrival at hospital. There was an improvement in both patient-reported and clinician-assessed breathlessness over the prehospital interval. The overall median VAS score for initial breathlessness was 9 out of 10 (IQR 8–10) for both patient-reported (n = 37) and clinician-assessed measurements (n = 76). The VAS score for breathlessness on arrival at hospital improved to a median of 5.5 (n = 48, IQR 4–8) for patient-reported measurements and 6.0 (n = 76, IQR 5–8) for clinician-assessed values. Initial and hospital-arrival VAS breathlessness scores were similar across trial arms for both patient-reported and clinician-assessed values (see Table 12).

Mean EuroQol-5 Dimensions, five-level version, score

Baseline quality of life, measured using the EQ-5D-5L, was relatively poor overall, with a median score of 0.63 (n = 51, IQR 0.43–0.83, range –0.027 to 1). Baseline EQ-5D-5L scores were similar across trial arms, with median scores of 0.59 (n = 26, IQR 0.43–0.90) and 0.63 (n = 25, IQR 0.44–0.74) in the intervention and control arms, respectively. Follow-up EQ-5D-5L data were available for 40 participants, demonstrating improved quality of life with an increased overall median score of 0.76 (IQR 0.48–0.92). Follow-up scores were higher in the CPAP arm [median score 0.82 (n = 22, IQR 0.58–0.95) than in the standard oxygen therapy arm [median score 0.73 (n = 18, IQR 0.43–0.89)].

Key elements of health-care resource use up to 30 days

Of participants who were discharged after their initial admission, 11 were re-admitted to hospital during the 30-day follow-up period (n = 44, 25.0%). A high proportion of respondents to the 30-day health care resource use questionnaire consulted other health-care services (28/41, 68.3%), with the commonest providers including general practitioners (15/41, 53.6%), EDs (7/41, 25.0%) and physiotherapists (5/41, 17.9%). Post-discharge health-care resource use is summarised in Table 11.

Safety

In total, 47 AEs (in 40 participants) were reported, of which 39 (in 34 participants) were classified as SAEs. The proportion of participants experiencing AEs and SAEs across trial arms were, respectively, 54.8% and 45.2% for the CPAP arm, and 48.6% and 42.9% for the standard oxygen therapy arm. The non-SAEs comprised claustrophobia or distress associated with CPAP mask use (n = 5), a non-rebreather mask in the control arm not inflating (n = 1, mask subsequently confirmed as functional by supplier) and non-specific mild worsening of condition (n = 2). The majority of SAEs were deaths (20/39, 51.2%) or re-admission to hospital within 30 days (13/39, 33.3%). There were no unexpected related SAEs. Two patients (one intervention arm participant not receiving CPAP and one control arm participant) were categorised as having experienced related expected SAEs following diagnosis with pneumothoraces requiring intercostal drainage after hospital admission. The remaining SAEs were all unrelated and included an inpatient fall resulting in a fractured neck of femur, cardiac arrest prior to administration of prehospital CPAP, social care re-admission and a stroke. AEs and SAEs are summarised in Table 12 and detailed individually in Appendix 5.

Model parameters

The decision-analytic model assigned the standard oxygen therapy cohort with a baseline probability of death and intubation. The risks of death and intubation for prehospital CPAP were estimated by applying the ORs from the ACUTE trial and updated HTA network meta-analysis3 to the baseline risks of mortality and intubation. Costs were based on the prehospital treatment received, the probability of intubation and life expectancy. QALYs accrued according to the probability of mortality, life expectancy and utility values. The model parameters are described in Baseline risks, Effectiveness and Lifetime quality-adjusted life-years. and summarised in Table 13. Costs are detailed in Table 14.

Baseline risks

The baseline risk of mortality was modelled using the 30-day mortality data from the control arm of the ACUTE pilot trial, which reported nine deaths [n = 35 (25.7%), complete-case, modified intention-to-treat analysis set]. The uncertainty in the baseline risk was represented in the model as a beta distribution, with an alpha of 9 and a beta of 26. A key secondary outcome measure of interest was the intubation risk, which determines whether or not critical care admission is required. This parameter was also modelled using the data from the control arm of the ACUTE trial, which reported one intubation [n = 29 (3.4%), complete-case, modified intention-to-treat analysis set]. Uncertainty in the intubation risk was represented as a beta distribution, with an alpha of 1 and a beta of 28.

Effectiveness

The base-case cost-effectiveness analysis used relative effectiveness results from the ACUTE pilot trial for mortality and intubation. The ORs for effectiveness of CPAP for reducing mortality and intubation were 1.2 (95% CI 0.4 to 3.2) and 1.8 (95% CI 0.2 to 40.1), respectively. Both ORs were calculated using the intention-to-treat principle, but the OR for intubation only used patients for whom in-hospital intubation data were available.

A scenario analysis was also performed using different effectiveness parameters from a preceding HTA-funded evidence synthesis. 3 The network meta-analysis was revised with results from the ACUTE trial, using identical methods to those previously reported. The ORs with 95% CrIs for reducing mortality and intubation were 0.5 (95% CrI 0.2 to 1.4) and 0.4 (95% CrI 0.1 to 0.9), respectively.

Lifetime quality-adjusted life-years

Lifetime QALYs were estimated by multiplying the life-years with representative quality of life. The same estimates for life expectancy and quality of life were used as in the previous economic model, both derived from the 3CPO trial. 22 Discounted life expectancy was estimated at 2.67 years, parameterised as a normal distribution with a mean of 2.67 years and standard deviation of 0.16 years. The mean utility value was 0.6, represented as a beta distribution with an alpha of 640 and a beta of 425.

Intubation costs

The cost of intubation was estimated in the previous HTA economic model3 by multiplying intensive care unit costs by the average length of stay for intubation (assumed to be 5 days). In the previous model,3 the mean intubation costs were estimated as £3500. These costs were inflated using the ratio of Hospital and Community Health Service (HCHS) indices for 2016/17 and 2014/15, which were 302.3 and 293.1, respectively. This resulted in a mean annual cost of £3600. In the model, this cost was parameterised as a gamma distribution, with an alpha of 90 and a beta of 40, after consultation with ACUTE trial clinical experts.

Hospitalisation costs

The cost of ED care was assumed as £0 in both arms. Length of hospitalisation is dependent on whether or not a patient needs intubation, as length of stay is longer for patients who undergo intubation. The same estimate from the previous economic model, informed by Hubble and colleagues,86 was used, which reported that the mean hospital length of stay for patients without intubation is 5.84 days. The non-intubated participants in the ACUTE trial can be classified into those that received NIV in hospital (approximately 42.5% between both arms) and those that did not. ACUTE trial clinical advisors suggested that non-intubated patients with no NIV/critical care matched closely with patients with code DZ27S (respiratory failure without interventions, with complication and comorbidity score 11+) and those with no intubation but who did get NIV in hospital matched closely to the patients with code DZ27P (respiratory failure with single intervention, with complication and comorbidity score 11+) reported in NHS reference costs for 2017/18. 84 Thus, the mean inpatient admission cost for hospitalisations was calculated as the weighted average of the costs of patients with DZ27S and DZ27P, from the NHS reference costs for 2016/17. 84 The hospitalisation cost used in the model, with a mean cost of £3200, was represented as gamma distribution, with an alpha of 80 and a beta of 40.

Lifetime costs

Lifetime costs of survivors were estimated using the annual costs and the discounted life expectancy of patients captured from the 3CPO trial. In the previous HTA model,3 the mean annual costs were estimated as £5300, based on the 3CPO study published in 2009. These costs were inflated using the ratio of HCHS indices for 2016/17 and 2008/9, which were 302.3 and 267, respectively. This resulted in a mean annual cost of £6000. In the model, this annual cost was parameterised as a gamma distribution, with an alpha of 60 and a beta of 100, after discussions with ACUTE trial clinical experts. It was assumed that the lifetime costs were the same for all survivors, irrespective of whether they were in the standard care or prehospital CPAP arm.

Base-case cost-effectiveness results: ACUTE pilot trial effectiveness estimates

The base-case analysis used effectiveness estimates from the ACUTE pilot trial, representing NHS implementation of prehospital CPAP. Exploratory analyses indicated that 10,000 PSA runs were sufficient to sample fully from parameter probability distributions and achieve stable estimates of incremental cost-effectiveness. This analysis indicated that the prehospital CPAP strategy was cheaper and less effective than standard care. The ICER was therefore interpreted as the incremental costs and QALYs of standard care compared with CPAP (as the ICER is calculated by comparing with the next most effective alternative).

Incremental expected costs and QALYs from the PSA are shown in Figure 8 for standard oxygen therapy compared with the ‘baseline’ strategy of prehospital CPAP. Each PSA simulation, representing a realisation of the joint distribution of possible model inputs, is depicted by a single point on the cost-effectiveness plane. It is apparent that there is a large degree of uncertainty in incremental costs and effects, reflected in the dispersal of PSA simulations, falling in both the north-east and south-west quadrants of the cost-effectiveness plane. On average, the standard oxygen therapy strategy was more effective (mean incremental QALYs of 0.062), but also more expensive (mean incremental costs of £351) than prehospital CPAP. The mean ICER, estimated as standard care compared with CPAP, was £5685 per QALY. Given the typical NICE threshold of £20,000 per QALY, the base-case analysis indicates that standard care is cost-effective because it gains QALYs with an acceptable ICER, compared with CPAP.

FIGURE 8.

Cost-effectiveness plane showing incremental costs and QALYs for standard oxygen therapy compared with prehospital CPAP for the base-case analysis using ACUTE trial effectiveness data. Individual points depict a single PSA simulation (n = 10,000 PSA simulations). Solid line represents willingness-to-pay threshold of £20,000. Triangle corresponds to mean ICER.

The mean expected costs of standard oxygen therapy and prehospital CPAP were £15,201 and £14,850, respectively. The corresponding mean expected QALYs were 1.190 and 1.128. The NMB of standard oxygen therapy and prehospital CPAP were, therefore, £8598 and £7715, with an incremental NMB of £883 for standard oxygen therapy compared with prehospital CPAP, assuming a threshold value of £20,000 per QALY. Table 15 summarises the mean expected costs and QALYs, ICERs and NMB for the base-case analysis.

The base-case cost-effectiveness acceptability curve is shown in Figure 9, presenting the proportion of model runs for which each strategy is cost-effective over a range of potential willingness-to-pay thresholds. At thresholds of < £5000 per QALY, prehospital CPAP was the most cost-effective strategy in the majority of model runs, although the probability of cost-effectiveness remained close to 50%. However, at thresholds of > £5000 per QALY, standard care has more probability of being cost-effective. At the £20,000-per-QALY threshold, standard care was most likely to be cost-effective (67%).

FIGURE 9.

Cost-effectiveness acceptability curve for the base-case analysis using ACUTE trial effectiveness data. MAICER, maximum acceptable incremental cost-effectiveness ratio.

Scenario analysis cost-effectiveness results: updated network meta-analysis effectiveness estimates

A scenario analysis used effectiveness estimates from a previous HTA network meta-analysis,3 which synthesised previously published experimental data, updated with results from the ACUTE trial. The pooled effect estimate reflects the efficacy of CPAP that might be achievable in more developed EMS systems. Exploratory analyses indicated that 10,000 PSA runs were sufficient to sample fully from parameter probability distributions and to achieve stable estimates of incremental cost-effectiveness. This analysis indicated that the prehospital CPAP strategy was more expensive and more effective than standard care. The ICER was therefore interpreted as the incremental costs and QALYs of prehospital CPAP compared with standard care (because the ICER is calculated by comparing with the next most effective alternative).

Incremental expected costs and QALYs from the PSA are shown in Figure 10 for prehospital CPAP compared with the ‘baseline’ strategy of standard oxygen therapy. The model was re-run 10,000 times, each time with different values for the ORs, costs and utilities sampled from the probability distributions, indicating less uncertainty than the base case, with incremental expected costs and effects clustering in the north-east quadrant of the cost-effectiveness plane. On average, the prehospital CPAP strategy was more effective (mean incremental QALYs of 0.157) and also more expensive (mean incremental costs of £1522) than standard care. The mean ICER, estimated as prehospital CPAP compared with standard care, was £9712 per QALY. Given the typical NICE threshold of £20,000 per QALY, in this analysis it would be concluded that prehospital CPAP is cost-effective because it gains QALYs with an acceptable ICER, compared with standard care.

FIGURE 10.

Cost-effectiveness plane showing incremental costs and QALYs for prehospital CPAP compared with standard oxygen therapy in the updated network meta-analysis scenario analysis (n = 10,000 PSA simulations).

The mean expected costs of standard oxygen therapy and prehospital CPAP were £15,201 and £16,722, respectively. The corresponding mean expected QALYs were 1.19 and 1.35, respectively. The NMB of standard oxygen therapy and prehospital CPAP were £8598 and £10,209, respectively, with an incremental NMB of £1612, assuming a threshold value of £20,000 per QALY. Table 15 summarises the mean expected costs and QALYs, ICERs and NMB for this scenario analysis.

The updated meta-analysis scenario analysis cost-effectiveness acceptability curve is shown in Figure 11. The percentage of model runs in which prehospital CPAP was the most cost-effective strategy did not exceed 50% at thresholds of < £10,000 per QALY. At the £20,000-per-QALY threshold, prehospital CPAP was highly likely to be the most cost-effective strategy (94%).

FIGURE 11.

Cost-effectiveness acceptability curve for the updated meta-analysis scenario analysis. MAICER, maximum acceptable incremental cost-effectiveness ratio.

Value-of-information analyses

Expected value of perfect information

Reflecting the uncertainty in effectiveness of prehospital CPAP, together with the large, potential, opportunity losses from making the incorrect adoption decision, base-case individual EVPI was considerable at the NICE willingness-to-pay threshold: £300 at λ = £20,000. Given the relatively large annual population with ARF eligible for prehospital CPAP treatment (11,000 across England and Wales), and the long time period over which the technology is likely to be applicable (5 years), base-case population EVPI was also correspondingly large in the base-case analysis: £16.5M at λ = £20,000 per QALY. This indicated that it would be worth spending up to £16.5M on research investigating the effectiveness of prehospital CPAP for ARF. Population EVPI was substantial at thresholds of < £8000, but decreased to a minimum at a threshold of £10,000 per QALY, for which there was little uncertainty about whether to adopt or reject CPAP based on existing evidence. Population EVPI continued to increase at higher cost-effectiveness thresholds, representing rising uncertainty. The base-case population EVPI is presented in Figure 12.

FIGURE 12.

Base-case population EVPI. MAICER, maximum acceptable incremental cost-effectiveness ratio.

For the updated network meta-analysis scenario analysis, individual EVPI was also appreciable at λ = £20,000: £67.60, corresponding to a population EVPI of £3.72M. The population EVPI for this scenario analysis is presented in Figure 13.

FIGURE 13.

Updated network meta-analysis scenario population EVPI. MAICER, maximum acceptable incremental cost-effectiveness ratio.

Expected value of partial perfect information

The population EVPI places an upper limit on the total value of additional research relating to a specific decision problem, but does not indicate where future research may be beneficial. The population EVPPI indicates the value of reducing the uncertainty surrounding particular input parameters in the decision model. Ten parameters, reflecting targets for potential future research designs, were considered in the EVPPI analyses: baseline mortality and risks; relative effectiveness for mortality and intubation; costs of prehospital CPAP, hospitalisation, intubation and long-term survival; life expectancy; and lifetime quality of life.

The EVPPIs associated with each of these parameters in the base case are illustrated in Figure 14. At the threshold of £20,000 per QALY, the individual EVPPI associated with the effectiveness of CPAP in reducing mortality was £299.60, and individual EVPPIs for all the other parameters were zero. The population EVPPI for CPAP effectiveness on mortality in the base case was £16.5M.

FIGURE 14.

Base-case population EVPPIs at a £20,000 per QALY cost-effectiveness threshold.

The EVPPIs for model parameters in the updated meta-analysis scenario analysis are illustrated in Figure 15. At the threshold of £20,000 per QALY, the individual EVPPI associated with the effectiveness of CPAP in reducing mortality was £67.60, and EVPPIs for all the other parameters were zero. The population EVPPI for CPAP effectiveness on mortality in this scenario analysis was £3.72M.

FIGURE 15.

Updated network meta-analysis scenario EVPPIs at a £20,000 per QALY cost-effectiveness threshold.

Sample characteristics

A valid prehospital primary diagnosis was available for 76 out of 77 patients. For one case, the primary clinical impression was recorded as ‘other’, but lacked interpretable information to assign an underlying aetiology for ARF. COPD (25/76, 32.9%) and LRTI (25/76, 32.9%) were the most commonly suspected primary prehospital diagnoses. In six out of 76 cases (7.9%), a non-respiratory primary diagnosis was recorded: ruptured abdominal aortic aneurysm (n = 1), liver failure (n = 1), sepsis (not specified further, n = 2) and urinary tract infection (n = 2). A secondary diagnosis was recorded for 36 patients (N = 77, 46.8%), with a single contributory condition suspected in 29 patients (N = 77, 37.7%) and two supplementary diagnoses made for seven patients (N = 77, 9.1%). LRTI (9/77, 11.7%) and heart failure (10/77, 13.0%) were the most common concomitantly diagnosed conditions.

A final hospital primary diagnosis was available for 65 patients. Consent was declined for data collection in nine cases, clinical records were unavailable in two cases and in one case there was no clear underlying diagnosis apparent in the notes. The most common final diagnoses were COPD (21/65, 32.3%) and LRTI (n = 28/65, 43.1%). For four cases, a non-respiratory final diagnosis was given: myocardial infarction, ruptured abdominal, liver failure and sepsis (not specified further). Secondary conditions accounting for ARF were diagnosed in 27 patients (N = 65, 41.5%), with one additional condition recorded for 23 cases, two contributory diseases given for three cases and three further supporting diagnoses for one case. The most common secondary diagnoses were COPD (7/65, 10.8%) and heart failure (8/65, 12.3%). Notably, two patients were diagnosed with a pneumothorax in hospital (one primary diagnosis and one secondary diagnosis, both requiring intercostal drains). Prehospital and final hospital diagnoses are summarised in Table 18.

Agreement

There was limited reproducibility between the primary prehospital and hospital diagnoses, with raw agreement of 58.5% (38/65). However, if both primary and secondary diagnoses were considered together, counting any match and ignoring the precedence placed on each condition, there was higher raw agreement of 76.9% on at least one causative disease for ARF (50/65). Chance-corrected agreement between prehospital and hospital primary diagnosis was moderate, as demonstrated by a Gwet’s AC1 coefficient of 0.56 (95% CI 0.43 to 0.69). When both primary and secondary diagnoses were assessed together, there was substantial chance-corrected agreement on at least one condition, with a Gwet’s AC1 coefficient of 0.75 (95% CI 0.64 to 0.87). Agreement between prehospital and hospital diagnoses is summarised in Table 19 and Figure 17.

FIGURE 17.

A mosaic plot of prehospital and final primary diagnoses. AAA, abdominal aortic aneurysm; UTI, urinary tract infection.

Diagnostic accuracy

The performance of ambulance service clinicians’ assessment was then investigated by calculating diagnostic accuracy metrics for the most prevalent conditions (COPD, LRTI and heart failure). Other conditions were not evaluated because of small sample sizes, with consequent imprecision and intractability. Although each condition was identified correctly more often than not, all three were commonly missed as the primary diagnosis: the sensitivities for COPD, LRTI and heart failure were 71%, 54% and 67%, respectively. The specificity (‘rule-out’) was higher (COPD, 84.1%; LRTI, 86.8%; and heart failure, 86.7%). When both primary and secondary diagnoses were assessed together, diagnostic accuracy improved. Considering the index test and reference standard to be positive if the condition was recorded in either the primary or secondary diagnosis gave sensitivities of 95.2% for COPD, 69.2% for LRTI and 85.7% for heart failure, meaning that all three conditions were typically identified, even if not as the primary diagnosis. Specificities in this contingency were COPD, 84.1%; LRTI, 92.3%; and heart failure, 96.6%. Diagnostic accuracy metrics are summarised in Table 20.

Ambulance service clinician survey

Ambulance service clinicians were invited to complete a survey after each new participant enrolment. Responses were received in 40 cases (n = 77, 52%). Personal identifiable data were not collected, in line with ethics permission; therefore, it was not possible to confirm whether or not repeat surveys were completed by the 41 individual recruiters. Quantitative responses to survey questions are summarised using cumulative bar charts in Figure 18.

FIGURE 18.

Cumulative bar charts summarising responses to the ambulance service clinician survey. a, Question asked only if patient was randomised to CPAP arm; and b, question asked only if patient was randomised to standard care arm.

Clinicians were confident in both the diagnosis of ARF (98% agreed or strongly agreed) and determining whether or not potential patients were eligible to be recruited to the ACUTE trial (98% agreed or strongly agreed). Respondents from the standard care arm were less confident in managing ARF (36% strongly agreed) than those in the CPAP arm (73% strongly agreed).

Standard oxygen therapy arm clinicians presented mixed views on CPAP treatment, with only 43% strongly agreeing that they would have liked to have had CPAP available. One clinician indicated a lack of evidence, stating:

I understand for control purposes that comparison must be made by using standard therapy.

Conversely, one clinician felt that CPAP should have been available, stating:

I struggle with talking to the PT [patient] about CPAP and then finding only 0₂ mask in the box [sic].

Clinicians who were assigned to the CPAP arm were overwhelmingly positive with regard to their experiences with the intervention, with a large majority (> 75%) agreeing or strongly agreeing that training allowed appropriate use of CPAP; CPAP was easy to use; the CPAP mask correctly fitted; it was straightforward to titrate CPAP; CPAP did not delay transport; and there were no problems with delivering prehospital CPAP. Although respondents were positive about training and the equipment, they did suggest some practical improvements that could be made in a future trial:

A bit more training on the practical training aspect could have been added in to the training day, e.g. how to fit one on properly. Perhaps you could have demonstrated on the stage during the training day.

A diagram showing how it goes together on the patient may be helpful for staff who have not used it for a while and may have forgotten.

Difficulty with handover at hospital, nursing staff unaware of trial and whether patient should remain on CPAP or have it removed.

Clinicians were very comfortable with the amount of prehospital data collection, with 95% considering it ‘about right’. There was only one specific data collection concern regarding the patient-reported VAS breathlessness score, with a respondent stating:

Scale of 1–10 for how breathless a patient feels doesn’t seem appropriate as the vast majority are never going to be able to give a rating.

Intervention arm respondents were generally positive about how their patients perceived CPAP, with > 85% agreeing or strongly agreeing that CPAP was acceptable to patients and that they were comfortable administering this treatment to patients. Clinicians suggested that those patients who found it less acceptable did not tolerate the mask or the increased airway pressure, whereas one respondent who strongly agreed with both questions offered the following practical advice:

Letting the patient feel the pressure delivery on their face first is probably vital to a successful application, instead of applying the mask and then turning it on.

There were no significant issues raised in the closing general open question of the survey. Of note, there were no comments regarding allocation concealment.

Focus groups

Nine ACUTE trial-trained ambulance service clinicians took part in three focus groups, comprising six staff who had recruited to the trial, one who had not and two who had withdrawn from trial participation. Seven staff had attended the face-to-face ACUTE trial training event, whereas two had trained using web-based training videos.

Participants described a range of facilitators that helped participation in the ACUTE trial. Specific themes included the following.

Pilot trial

The ACUTE pilot trial enrolled 77 participants over 12 months, below the recruitment target of 120 participants. CPAP was fully delivered as planned in 74% (31/42) of intervention arm participants, compared with the feasibility target of 75%. There were no major protocol violations/non-compliances. Full data were available for key outcomes, including all feasibility end points and vital status at 30 days, compared with the feasibility targets of ≥ 90% retention at 30 days and ≥ 90% data completeness. However, data were missing on other hospital variables and follow-up outcomes owing to declined consent, lack of research approvals to access data, non-interpretable clinical records and non-completion of questionnaires (ranging from 0% to 29%).

Mortality was higher than expected, with 28.6% (12/42) of participants in the CPAP arm and 25.7% (9/35) of participants in the standard oxygen arm dying by 30 days. Of the deceased patients, 68% (13/19 with available data, n = 21) either did not have a respiratory condition or had explicit or implicit ceiling-of-treatment decisions, which excluded hospital NIV or critical care. The risk of intubation was low (3/62, 4.8%): two participants in the CPAP arm (n = 33, 6.1%) and one participant in the standard oxygen therapy arm (n = 29, 3.4%) were intubated. A small proportion of participants were admitted to critical care (6/65, 9.2%), with risks of 11.4% and 6.7% in participants treated with CPAP (4/35) and standard oxygen therapy (2/30), respectively. Hospital length of stay was similar for both trial arms, with a median of 10.0 days (IQR 5.8–12.0 days) in the CPAP arm and 7.0 days (IQR 5.0–9.8 days) in the standard oxygen therapy arm (n = 44). Breathlessness, evaluated by ambulance service clinician-assessed VAS score, was similar across trial arms, and improved over the prehospital interval from an initial median of 9 out of 10 (n = 76, IQR 8–10) to 6 out of 10 (n = 76, IQR 5–8) on arrival at hospital. Median follow-up EQ-5D-5L scores at 30 days were 0.82 (n = 22, IQR 0.58–0.95) for participants allocated to CPAP and 0.73 (n = 18, IQR 0.43–0.89) for those allocated to standard oxygen therapy.

Adverse events related to CPAP comprised mild claustrophobia or distress associated with CPAP mask use. Two participants were diagnosed with pneumothorax in the ED, which were reported as expected related SAEs (one participant in the intervention arm not receiving CPAP and one participant in the control arm; both required intercostal drainage). There were no other expected or unexpected related SAEs. During week 10 of recruitment, some intervention arm equipment boxes began to ‘rattle’, due to packaging of CPAP face masks under tension and deflation of the CPAP mask air cushion seal. After repackaging and redistribution, no further concerns were noted during equipment box audits. No other feasibility issues were evident.

Economic evaluation

The base-case analysis, using CPAP effectiveness estimates from the ACUTE pilot trial representative for the NHS setting, indicated that the standard oxygen therapy strategy was more effective (mean incremental QALYs of 0.062), but also more expensive (mean incremental costs of £351), than prehospital CPAP. The mean ICER, estimated as standard care compared with CPAP, was £5685 per QALY. At the £20,000-per-QALY threshold, standard care was most likely to be cost-effective (67%).

A scenario analysis, using effectiveness estimates from an updated meta-analysis typical of non-NHS settings with more developed EMS systems,3 suggested that prehospital CPAP was more effective (mean incremental QALYs of 0.157), but also more expensive (mean incremental costs of £1522), than standard care. The mean ICER, estimated as prehospital CPAP compared with standard care, was £9712 per QALY. At the £20,000-per-QALY threshold, prehospital CPAP was highly likely to be the most cost-effective strategy (94%).

Values of information analyses demonstrated that there was considerable uncertainty about whether or not to adopt prehospital CPAP. In the base-case analysis, the population EVPI indicated that it would be worth spending up to £16.5M on research investigating the effectiveness of prehospital CPAP for ARF. This compared with a population EVPI of £3.72M in the updated meta-analysis scenario analysis. EVPPI analyses indicated that effectiveness of prehospital CPAP on mortality was the only important variable for future research, with population EVPPIs of £16.5M and £3.72M, respectively, in the base case and updated meta-analysis scenario analysis.

Acute respiratory failure incidence study

Between 1 August 2017 and 31 July 2018, 1017 patients were identified from the WMAS with ARF, and were eligible for the ACUTE pilot trial, giving an overall incidence rate of 17.4 per 100,000 persons per year (95% CI 16.3 to 18.5 per 100,000 persons per year). A mild seasonal variation was apparent, with an increased incidence rate during autumn and winter months.

Acute respiratory failure diagnostic accuracy study

Chronic obstructive pulmonary disease (25/76, 32.9%) and LRTI (25/76, 32.9%) were the most frequently suspected primary prehospital diagnoses for ARF, with secondary contributory conditions recorded in 36 out of 77 participants (46.8%). The most common final hospital diagnoses were also COPD (21/65, 32.3%) and LRTI (28/65, 43.1%). In seven cases, a final diagnosis was made for which CPAP would not be expected to be effective or could be harmful, including myocardial infarction, ruptured abdominal aortic aneurysm, liver failure, sepsis and pneumothorax (7/65, 10.8%). Secondary conditions accounting for ARF were diagnosed in 27 participants (n = 65, 41.5%). There was moderate agreement between the primary prehospital and hospital diagnoses, with raw agreement of 58.5% (38/65) and a Gwet’s AC1 coefficient of 0.56 (95% CI 0.43 to 0.69).

Clinicians’ views: mixed-methods study

The post-recruitment survey demonstrated that ambulance service clinicians felt confident in the diagnosis of ARF and in determining trial eligibility. Respondents assigned to the intervention arm were positive regarding clinician and patient experiences with CPAP. The amount of prehospital data collection was considered to be ‘about right’. Focus group participants identified a number of facilitators that aided participation in the ACUTE trial, including uncomplicated equipment boxes, successful multimedia promotion of the trial, ease of use of trial documentation, simplicity of consent processes and the importance of the research question. Conversely, lack of awareness of the ACUTE trial in receiving hospitals, problems in measuring patient-reported breathlessness, difficulties in completing web-based trial training and a desire to provide CPAP treatment were highlighted as important challenges. No issues were raised in the post-recruitment survey or focus groups regarding allocation concealment.

Allocation concealment substudy

Of the ambulance service clinicians participating in the allocation concealment substudy, 58.6% were unable to distinguish a difference between control and intervention arm boxes (163/278). The participants who felt that they were able to tell which box contained CPAP equipment (115/278, 41.4%) indicated a median certainty of 5 (moderate certainty, IQR 4–7), with 70.4% choosing box contents correctly (81/115, 95% CI 61.1% to 78.4%).

Evidence of feasibility

Feasibility studies are stand-alone investigations completed before the start of a definitive RCT, investigating whether or not a large-scale trial can be done, whether or not it should be done and, if so, how it should be conducted. 48 Pilot trials, a subset of feasibility studies, additionally test the proposed main trial procedures in miniature to determine whether or not the components of the main trial work together. 51,52 The ACUTE pilot trial recruited only 21% of the potentially eligible patients (77/364) and achieved only 64% of its recruitment target (77/120); therefore, feasibility was not demonstrated. Nevertheless, a number of issues were successfully addressed. Patients were appropriately recruited using a deferred consent model, with no reported issues with using verbal consent for recruitment of patients with capacity. There were no major protocol violations/non-compliances and only one minor non-compliance. Adherence to the allocation schedule was complete: commencement of CPAP was attempted in all intervention arm participants and control arm participants received appropriate standard oxygen therapy. Full data were available for key outcomes, including all feasibility end points and vital status at 30 days. Furthermore, ambulance service clinicians were able to recognise ARF, enrol patients and use the CPAP equipment.

The recruitment rate could be improved by addressing a number of remediable factors. Ambulance service clinician participation was voluntary and the majority of potentially eligible cases presented to non-trial-trained staff (203/364). Therefore, it is likely that recruitment could be improved if, consistent with some previous prehospital studies,87,88 a full trial mandated participation of all ambulance service personnel. Focus groups revealed a number of solvable issues, including improving access to hub research cabinets and providing designated storage space on ambulances for trial equipment boxes. However, determining the impact of these interventions would require further piloting.

Several other issues identified in the pilot trial would need to be addressed before considering a full trial. First, a relatively large number of patients (9/77, 12%) declined consent for further participation and data collection when approached in hospital for confirmation of previously provided prehospital verbal consent. Although the primary clinical outcome of 30-day mortality was available, important secondary end points were, consequently, missing. These patients were less unwell than incapacitated patients included following consultee consent, and selection bias would be a risk if there were an association between trial arm and the probability of providing consent. No issues with the provision of initial verbal consent were reported by patients or participating clinicians, but informal feedback to research paramedics later during the consent process suggested that withdrawal occurred as patients did not wish to complete the 30-day follow-up questionnaires. To mitigate this problem, any future trial could offer tiered consent, allowing participants to opt out of 30-day follow-up questionnaires, but to agree to collection of hospital data and secondary clinical outcomes from their records.

Second, although the main feasibility and clinical outcomes had complete follow-up, some hospital data were missing because of failure to gain research approvals in two hospitals. In one case, this occurred after unanticipated patient conveyance to an out-of-area hospital. Challenges arising from transport across regional and ambulance service boundaries have been noted in previous prehospital trials and could be magnified in a scaled-up definitive trial. 89 In the second case, it was not possible to engage the research and development department because of an ongoing merger between separate hospital trusts. The secondary care landscape is undergoing considerable evolution and, as pressures on acute care services continue, similar problems might be expected in the future. 33 However, the number of missing data was very small and such challenges would not present a significant barrier to a definitive trial.

Third, a novel prehospital method of randomisation using identically sealed equipment boxes was piloted, which could have been unreliable. Correctly implemented, randomisation results in balanced comparison groups that at baseline differ only by chance in potentially confounding variables. 90 A valid randomisation process requires two essential and independent components: generation of an unpredictable random allocation sequence and concealed implementation, with irreversible assignment to trial arms without foreknowledge of the allocation when enrolling patients in the study. 38 If the expected trial arm is known in advance, then the decision to accept or reject potential participants may be influenced by personal prejudices, resulting in patients with better prognosis being assigned to one arm rather than the other. Selection bias in effect estimates will subsequently occur secondary to imbalance in baseline prognostic variables. 91

The importance of secure allocation concealment in the ACUTE trial is highlighted by the focus group findings that clinicians and patients were uncomfortable with randomisation to standard care, for example:

. . . I hope to God this has got a CPAP in it . . .

I opened the box and it was the 100%, and the lady herself went [big sigh] and I felt dreadful then.

Systematic reviews have consistently demonstrated that trials using inadequate or unclear allocation concealment methods report greater heterogeneity in results, providing evidence of bias that may result in either exaggerated effect estimates or failure to identify a true treatment effect.

Prehospital trials pose unique barriers to randomisation. 92,93 The environment is uncontrolled, with recruitment potentially occurring away from ambulances at the scene of the incident. The location of enrolment may be out of radio, telephone or internet signal, and emergency, time-critical conditions are often studied. Individual randomisation options are, therefore, limited, as central randomisation (e.g. web- or telephone-based randomisation) is often not practical, and sequentially numbered opaque sealed envelopes have well-known limitations. 94 Scratch cards have recently been proposed as a novel randomisation method for prehospital trials, although there is limited experience with this method. 92 Cluster randomisation offers one potential way to avoid these problems, with recruiting centres, ambulances or clinicians (rather than patients) being randomised to delivering either the intervention or control to all their trial participants. 95,96 This inevitably leads to advance knowledge of the treatment a future participant would receive and is therefore prone to post-randomisation recruitment bias, whereby different types of participants are recruited in the trial arms. 95,96 Therefore, undertaking a valid randomised trial in a prehospital setting requires a different approach.

Sequential, identical medicine containers are recommended in the CONSORT statement for allocation concealment. 37 Similarly, identical equipment boxes seem to offer a promising method for randomisation in medical device trials. Ensuring that medications are indistinguishable across arms is likely to be straightforward, as they can be specifically manufactured to be identical. 37,91 However, medical device trials pose a greater challenge, as it will often not be possible, or desired, in pragmatic trials to use a ‘mock’ device in the control arm. The difficulties of ensuring that all containers are identical is highlighted by the unexpected development of ‘rattling’ in the intervention arm boxes 10 weeks into the trial and the weight discrepancy detected in an equipment box used in the allocation concealment substudy.

Although a loss of allocation concealment cannot be entirely ruled out, there is no evidence that any subversion of allocation actually occurred. The problem of boxes ‘rattling’ was quickly detected and immediately rectified. No further issues were subsequently observed during careful weekly audits. Post-recruitment surveys and clinician focus groups did not suggest any issues with randomisation. The number of patients recruited per arm, and baseline characteristics, were also reassuringly similar. Furthermore, the allocation concealment substudy results were inconclusive, with only 29% of participants indicating that they could tell a difference between boxes and correctly identifying the treatment arm (81/278). Regardless of any problems with box randomisation, the other feasibility findings of the pilot trial will remain unaffected.

Given the finding that randomisation using equipment boxes in medical device trials may be unreliable, it would be important that any future definitive trial implementing this technique uses rigorous quality control processes to avoid potential selection bias. Scratch card randomisation is an alternative method that could be used in a definitive ACUTE trial. 92,97 This method has its own limitations, most obviously damaged cards, which may arise through wear and tear, but could also be perceived as an attempt to decipher future allocations. A further problem would arise if several potentially eligible cases presented on a given shift, as it would be difficult to verify the order in which allocations were given (and, in particular, whether these were manipulated dependent on the revealed allocation). Nevertheless, the relatively low incidence of ARF means that this is unlikely to present a limitation in this setting.

Cost-effectiveness of prehospital continuous positive airway pressure

The decision-analytic model shows that the key determinant of cost-effectiveness is whether or not prehospital CPAP is effective in reducing mortality. This contrasts with the preceding HTA economic model,3 which suggested that the incidence of ARF was very important, secondary to its influence on prehospital CPAP costs. 3 The O-Two CPAP device used in the ACUTE trial is much cheaper and requires less training than the system previously examined, meaning that the costs of providing prehospital CPAP, and thus the incidence of ARF, are no longer critical in determining cost-effectiveness. 44

The base-case analysis, using ACUTE pilot trial effectiveness data, suggested that prehospital CPAP was cheaper than standard care. This arises from increased short-term mortality, with fewer patients incurring critical care or lifetime health costs. However, this also results in fewer lifetime QALYs, and, at the conventional £20,000 threshold, there is a 67% probability that standard care is the most cost-effective option. The ACUTE pilot trial should be more representative of NHS ambulance services, but the low sample size gives very imprecise effectiveness estimates and leaves considerable uncertainty around cost-effectiveness, reflected in the large population EVPPI for the mortality effectiveness parameter.

In addition to ‘second-order’ sampling uncertainty, there is also significant uncertainty around what is the most valid and applicable effectiveness estimate for prehospital CPAP. The scenario analysis, using updated meta-analysis effectiveness data, gives the opposite conclusion to the base case and suggests that CPAP is highly likely to be cost-effective at a threshold of £20,000, although population EVPI and EVPPI for CPAP effectiveness still remain high. Overall, the economic evaluation indicates that cost-effectiveness is principally dependent on the clinical effectiveness of CPAP, and expected value-of-information analyses are favourable, supporting the commissioning of a large pragmatic effectiveness trial, providing feasibility and plausibility conditions are met.

Other insights from the ACUTE pilot trial

The ACUTE pilot trial failed to achieve its recruitment target, but cost-effectiveness analysis and value-of-information analysis suggested that a large pragmatic trial to determine the effectiveness of CPAP could still be worthwhile. This could be delivered if the remedial actions identified above were successful or if the trial recruited across a large number of sites. Alternatively, given that the mortality rate in the ACUTE pilot trial was higher than expected, a smaller sample size than previously anticipated could detect a potentially important absolute difference in mortality.

External pilot trials are not designed or powered to generate estimates of clinical effect that should be used for decision-making. 48,51,52 We have therefore drawn no conclusions from comparisons of outcomes between CPAP and standard care. However, a number of findings from the ACUTE pilot trial can be used to inform a judgement regarding whether or not it would be plausible for a large trial to detect an effect from CPAP on mortality and, therefore, whether or not a trial might be worthwhile.

First, the diagnostic accuracy study indicated that prehospital assessment of ARF is difficult and identifying patients with the potential to benefit from CPAP may be challenging. Respiratory distress with low oxygen saturations is common to many conditions, with symptoms and clinical signs shared between a range of differential diagnoses, including non-respiratory conditions. 2,7 It is therefore unsurprising that agreement between the primary prehospital clinical impression and primary final hospital diagnosis was limited. COPD, LRTI and heart failure were the most commonly identified conditions, and it is notable that, in many cases, these diseases were given concurrently as supplementary diagnoses. When both primary and secondary diagnoses were assessed together, counting any match and ignoring the precedence placed on each condition, raw agreement on at least one causative disease for ARF improved substantially from 58% to 77%. However, difficulties in prehospital diagnosis inevitably reduce our ability to target patients for CPAP appropriately.

Given that the most important treatment for ARF is provision of oxygen and that other treatment modalities currently available to NHS ambulance service clinicians (e.g. nebulisers) have few side effects, it could argued that an exact prehospital diagnosis is unnecessary prior to definitive hospital care. 2,7,11,14 The most important factor, therefore, is to recognise conditions when CPAP would not be beneficial and identify any contraindications. In the ACUTE pilot trial, a small, but significant, minority of cases (4/65, 6%) were ultimately diagnosed with conditions for which CPAP could not conceivably be beneficial. A further two participants were ultimately found to be suffering from asthma, for which the evidence base for definitive care with CPAP is uncertain. 98 It is also concerning that two participants were diagnosed in the ED with clinically significant pneumothorax, which required intercostal drainage. Although neither participant received prehospital CPAP, the potential for iatrogenic harm if these cases had been allocated to the intervention arm is conspicuous.

Second, delivery of CPAP was relatively limited, with only 74% of intervention arm participants continuing treatment to hospital as planned, just below the lower limit of the feasibility criterion of > 75% usage. Almost one-fifth of participants either refused to commence CPAP or did not tolerate treatment (8/42, 19%). Although CPAP may be efficacious, the potential to demonstrate effectiveness would be restricted by lack of treatment compliance. Feedback from the post-recruitment survey suggested that unhurried and gentle application of the mask was ‘vital to successful application’. However, such an approach may not be practical in the emergent and confused prehospital setting.

Third, a key rationale for the implementation of prehospital CPAP is that earlier instigation of treatment will improve outcomes over and above the availability of hospital NIV. 24 In the pilot trial, relatively short on-scene times and conveyance times were recorded (median of 40 minutes and 13 minutes, respectively). It could be argued that, in urban or semirural settings, the potential time saved from prehospital administration of CPAP is likely to be too small to produce meaningful benefit.

Fourth, only a small proportion of participants (28%) received CPAP or NIV after arrival at hospital. Clinical improvement during the EMS interval, secondary to prehospital CPAP, and lack of familiarity with the O-Two device among ED staff are possible reasons underpinning this observation in the intervention arm. However, although the majority of hospitals had access to ED NIV, no standard care arm participants received CPAP and only a very small number were treated with BIPAP (15%). This could suggest that enrolled participants were satisfactorily treated by current prehospital practice, or would not be expected to benefit from NIV.

Finally, illness severity was much higher than anticipated. The increased overall 30-day mortality risk of 27.3% might initially appear to offer a definitive trial a greater opportunity to detect a clinically relevant survival benefit. However, despite designing eligibility criteria to exclude those for whom critical care interventions may be inappropriate, many enrolled participants had apparent treatment limitation decisions for ward-level hospital care only [11/17 (65%) with explicit or implicit ceiling-of-treatment decisions that excluded hospital NIV or critical care for respiratory illness]. The observed risk of intubation and admission to critical care was consequently low, at 4.8% (3/62) and 9.2% (6/65), respectively. The fact that NIV and higher levels of treatment were considered inappropriate for many enrolled cases indicates a study population with a high prevalence of end-stage cardiorespiratory disease, multiple severe comorbidities or very poor pre-morbid performance status, for which more aggressive treatment might be futile and overly burdensome. 99,100 Given the problems in establishing this information at the scene, prehospital CPAP would be unlikely to provide benefit in such a contingency.

In summary, although the higher than expected mortality rate might suggest increased potential to detect an absolute difference in mortality, the challenges of providing prehospital CPAP and the characteristics of the patients who would benefit from CPAP, suggest limited potential to improve survival. It therefore appears unlikely that a trial powered to detect a plausible effect size could be designed.

Pilot trial

The purpose of a feasibility study is to determine if a large-scale trial can be performed. Therefore, challenges that might be interpreted as weaknesses when appraising a definitive RCT actually represent important learning points in the feasibility setting. 48,51,52 In addition to the previously discussed issues, a number of other factors were noted during the pilot study that would help inform the design of any definitive trial. Modification of eligibility criteria to specify the presence of a primary cardiorespiratory diagnosis as an inclusion criterion, and exclusion of patients with home CPAP machines or reduced level of consciousness, would ensure that a more appropriate trial population is selected. Recording level of consciousness using the Alert, Voice, Pain, Unresponsive (AVPU) scale would also help improve data completeness for this domain. Designing ‘carbon-copy’ consent forms in triplicate was also identified as a potential method to reduce the research burden on patients and recruiting clinicians.

The ACUTE pilot trial followed best practice research conduct and reporting recommendations. 37,38 However, a weakness was that hospital data collection was reliant, to some extent, on retrospective case note review. This approach improved the efficiency of data collection, but is associated with well-recognised limitations that could have resulted in inaccurate measurement of trial end points. 101,102 If information was not present, it was not possible to be certain whether it was not recorded or whether the variable value was genuinely null. Subjective variables could also potentially be misinterpreted. However, data collection rules, such as using the first recorded value, should have minimised any information bias, and feasibility and clinical outcomes represented routinely collected objective end points, robust to misclassification.

Health economics

The economic evaluation updated a previously published decision-analytic model and followed NICE base-case recommendations. 3,103 Taking the perspective of the NHS in England and Wales, valued outcomes, such as QALYs, used a lifetime horizon and included PSA. 54,55,64 Other strengths included detailed costing at the level of the ambulance service, and use of relevant existing data sources to estimate key population, cost and outcome parameters. Decision uncertainty was explored in scenario analyses using different effectiveness estimates and the potential benefit of future research was evaluated in expected value-of-information analyses. Using ACUTE trial data directly relevant to NHS practice for key ARF and effectiveness parameters in the base case helped overcome the main limitations of the preceding HTA economic analysis,3 which was reliant on potentially non-generalisable and biased estimates.

However, there are limitations in the model design and parameterisation, which could challenge the internal validity of results. The model assumed that the proportion of patients who would receive NIV in hospital was similar in both arms, irrespective of whether or not a patient received prehospital CPAP. This appears plausible based on the limited pilot data, but it is conceivable that there could be an association between the effectiveness of treatment during the EMS interval and ED management. It was also assumed that the lifetime QALYs were same for all survivors, irrespective of whether they were in the standard care or prehospital CPAP arm. Although unproven, this appears to be reasonable, as CPAP would be expected to help with acute presentations and short-term outcomes only, rather than modify underlying chronic diseases. No information was available on the covariance between individual model parameters. Treating these variables as independent in the PSA is a further possible limitation of the model structure.

Within the modelled population, there will be a considerable diversity of patients with differing characteristics, underlying diagnoses and prognoses. Applying a cohort methodology, with consequent use of mean values, impeded an examination of uncertainty due to heterogeneity. However, competing management strategies are service-level interventions and, hence, would be applied to the entire population presenting with ARF and ostensibly eligible for CPAP. Exploration of heterogeneity, for example the cost-effectiveness in different underlying diseases, is therefore less relevant. Finally, there was a limited evidence base available to parameterise lifetime QALYs and costs of care, with data provided by the 3CPO trial. 22 This trial enrolled patients with pulmonary oedema receiving ED NIV, rather than the undifferentiated EMS ARF cases relevant for prehospital CPAP. However, baseline characteristics of participants in the 3CPO trial22 appear similar to those included in the ACUTE trial.

Finally, an EVSI analysis was not undertaken. 104 EVSI extends the value-of-information methodological framework to establish the expected value of conducting studies with different designs and sample sizes. The expected benefits of a given study sample (the population EVSI) can be compared with the expected costs of collecting these data, with the difference denoting the expected net benefit of sampling (ENBS), measuring the societal reward from conducting additional research. ENBS values > 0 demonstrate that the marginal benefits of gathering further evidence exceed the marginal costs, with higher ENBS values representing more efficient study designs. 104 However, although the population EVPPI result suggests that a pragmatic trial would represent value for money, as expounded above, it does not seem credible that a clinically significant effect size could be demonstrated, which suggests that the computationally expensive EVSI analysis is redundant.

Ancillary substudies

The ARF diagnostic accuracy and agreement study has a number of strengths, including the prospective prehospital data collection and defined nominal categorisation for ARF. The index tests and reference standard were also independently applied, with no possibility of incorporation, partial or differential verification biases. 75,105–107 However, there was the potential for reference standard misclassification, as the final diagnosis was recorded from the hospital record or discharge letter, rather than determined through formal expert case review. 108 Although comparing favourably with other published reproducibility small studies,109,110 the sample size was relatively low, resulting in imprecise results consistent with either fair or substantial agreement. This sample size constraint also prevented modelling of any clustering effects arising from ambulance service clinicians and hospitals assessing multiple patients. Furthermore, some reference standard data were missing. Although this represented a relatively small number of patients, with similar characteristics to included cases, selection bias is possible if excluded patients differed systematically from the trial population. Finally, the relatively liberal Landis and Koch79 scale was used for benchmarking agreement coefficients. Although well established and widely used, this may overstate agreement compared with other benchmarks (e.g. Fleiss’s111 or McHugh’s112 proposed scales).

The remaining ACUTE trial substudies also had strengths and weaknesses. The allocation concealment substudy tested a large sample of ambulance services clinicians using random pairs of ACUTE trial equipment boxes. However, some informative details about allocation concealment were not available, and the exact weights of all trial equipment boxes during, and at the end of, the trial were not recorded. The ARF incidence study was reliant on electronic filtering of the WMAS database followed by patient record review. Although a census sample of cases was studied, and inter-rater agreement examined, the findings are similarly limited by problems arising from retrospective clinical note review,101,102 as discussed in Pilot trial. Determining whether or not a patient improved rapidly with supplemental oxygen, or met the definition of ARF with low peripheral oxygen saturations despite treatment, was particularly difficult to deduce using this approach. It is therefore possible that the reported incidence of ARF is underestimated. Finally, the response rate of the post-recruitment survey was low and omission of a personnel identifier prevented e-mail reminders being sent to non-completers and the identification of repeat recruiters.