Electronic self-reporting of adverse events for patients undergoing cancer treatment: the eRAPID research programme including two RCTs

Accessibility to patient population

In January to February 2020, 96% of households in the UK had access to the internet (as reported by the Office for National Statistics). 50 In our research (NIHR Research for Patient Benefit grant PB-PG-0107-12239), patients aged > 65 years and those of lower socioeconomic status declined participation in web-based studies because of lack of internet access.

Alignment with NHS policy

The NHS White Paper (2010)51 recommends wider use of PROMs and patient experience data to put patients at the heart of the service. Our approach is in line with the NHS agenda, emphasising wider, effective use of PROMs and improved use of technology, telehealth, remote monitoring and self-care. The DHSC has recently reported results from the Whole System Demonstrators programme,52 showing that telecare/telehealth helps people manage their health. A project is underway in elective surgery collecting PROMs to evaluate NHS organisations’ performance and quality of care. 26 The DHSC recommended the introduction of an innovative quality-of-life metric to track and respond to the long-term impact of cancer. 53 The Independent Cancer Taskforce, in its strategy for England 2015–2053 and NHS Long Term Plan,54 recommended the introduction and implementation of QoL data collection for cancer survivors in England. Pilot projects are underway looking at electronic methods of patient self-reported questionnaires.

Changes to the original plans

Instead of performing a sequential pilot study followed by a RCT, on completion of WPs 2 and 3 we decided to carry out a usability study to test all active components of the intervention together. Afterwards, with permission from NIHR, we carried out an internal pilot in the RCT, allowing more time-efficient progress to the large-scale RCT. After the internal pilot we decided to include metastatic colorectal cancer patients as their treatment and pathway was similar to the adjuvant group. In 2017, we also included metastatic breast cancer patients in an attempt to improve recruitment and complete the RCT on time.

Changes to the original plans

In 2015, after completing WP 3, we established that the radiotherapy techniques/doses and side effects differ substantially between prostate cancer patients receiving radiotherapy only and those in the chemotherapy arms. We decided to stratify the pilot study and increase the sample size to recruit 42 intervention-arm and 42 control-arm patients in the prostate radiotherapy-only arm, and 42 intervention-arm and 42 control-arm patients in the chemoradiotherapy arm. Two websites were introduced to more closely match local practices in terms of patient information, advice and available services.

Changes to the original plans

Original plans included development, pathway mapping and usability work (WPs 1–3), followed by a small randomised pilot study in the Bristol site (WP 4). In 2015, following initial work on WPs 1–3, we proposed to undertake the development and mapping work prior to performing extensive usability and feasibility testing in a larger, two-centre, non-randomised pilot study in the Bristol and Birmingham sites. A larger pilot study was considered beneficial for three reasons. First, AEs (e.g. hospital re-admission, reoperation, sepsis, pneumonia) occur in relatively fewer patients undergoing major cancer-related upper gastrointestinal surgery than in patients undergoing systemic therapy, during which toxicity is common. Second, the use of electronic methods for patients to self-report AEs following surgery in the Bristol site was novel, whereas at the Leeds site clinicians are experienced in using the online platform to collect PROMs. Therefore, we needed to fully examine how HCPs interact with the eRAPID interface and the acceptability of the eRAPID IT system and its integration into the clinical pathway before proceeding to a randomised pilot study. In 2018, a second site, Birmingham, was added to the pilot study to explore the acceptance of the intervention in a different surgical oncology setting (where electronic AE reporting and monitoring is also novel) and to reflect a wider range of services. Third, the problems with IT integration with EPRs that we experienced in the Bristol site meant that a more thorough evaluation over a longer time period would be beneficial. Consequently, we undertook a larger, two-centre pilot study to allow comprehensive exploration of the feasibility and acceptability of the eRAPID surgery system at a clinical site at which the system had not been developed. The version of eRAPID tested at the Birmingham site was not integrated within the hospital EPR system. Therefore, data from the Birmingham site enabled full exploration of integration and ease of use of the online system where EPR system integration was not viable.

Patient reminders

Following the clinical usability testing, patients requested to be sent reminders when they are due to complete the online symptom reports. Therefore, a patient reminder system was developed that would automatically send out e-mails or short message service (SMS) text messages to patients on a pre-determined schedule. In the reminder system, a patient was automatically set to have no contact if they were flagged as having left the trial or deceased.

Integration with electronic patient records

The eRAPID website at the Christie Hospital was developed with the same technology as at Leeds Teaching Hospitals NHS Trust (WordPress) and hosted at the University of Leeds; however, research staff at the Christie Hospital were given access to manage the content pages containing patient advice.

The version of QTool was the same as the version used at Leeds Teaching Hospitals NHS Trust (hosted at University of Leeds), except that a separate organisation was created for the Christie Hospital, and staff there were given access to manage their eRAPID questionnaires.

A bespoke version of QStore was developed for the Christie Hospital because it was required to be integrated with the local bespoke EPR system, CWP, which is owned by and managed by the Christie Hospital IT team (Christie NHS Foundation Trust). The integration was tailored specifically to work with the CWP system, which involved the Christie Hospital IT team creating a small web service that QStore could call to obtain patient demographic information directly from CWP and which was subsequently used for patient identification in QStore. QStore was hosted locally at the Christie Hospital on their development server and could be accessed and maintained only by the Christie Hospital IT team.

For managing patient reminders, research staff at the Christie Hospital were lent a University of Leeds laptop that was installed with our reminder system front end, which linked via a virtual private network to the University of Leeds server used to send out patient reminders.

Challenges

Initially, good progress was made with developing the test version of the system and early usability testing with staff. However, during 2015 and the beginning of 2016, changes in IT staff in the Manchester site caused delays in progress with making QStore live.

Once live, the system largely functioned well and worked reliably (bar some instances for downtime when the Christie Hospital servers were taken down for maintenance purposes).

One issue that could not be resolved was the identification of patients reporting symptom notifications in QStore. This became problematic because the real-time patient identification relied on a web service developed by the Christie Hospital IT team on a test database, but was not implemented on the live CWP server. This meant that the identification of patients with severe symptom notifications had to be carried out by manual look-up rather than by having the patient’s name display in QStore.

Integration with electronic patient records

The eRAPID website at the Bristol site was developed with the same technology that was used at Leeds Teaching Hospitals NHS Trust (WordPress) and hosted at the University of Leeds; however, research staff at Bristol Royal Infirmary were given access to manage the content pages containing patient advice.

The version of QTool was the same as the version used at Leeds Teaching Hospitals NHS Trust (hosted at University of Leeds) except that a separate organisation was created for the Bristol site, and staff there were given access to manage their eRAPID questionnaires.

A bespoke version of QStore was developed for the Bristol site because it needed to be integrated with Medway (a commercial EPR system); however, because Medway was supplied and managed by a third-party software company, we had very limited ways in which we could integrate QStore into Medway. The agreed web integration was via a secure, encrypted URL that required passing a unique hospital code along with patient name, NHS number and date of birth. The encryption used a unique 128-bit Advanced Encryption Standard (AES) key. QStore could not be hosted locally at Bristol Royal Infirmary; therefore, a version was hosted on a dedicated University of Leeds virtual server and managed by the University of Leeds IT team. Administration rights to directly log in to access and manage parts of QStore (such as alert reports) were given to research staff at the Bristol site.

For managing patient reminders at the Bristol site, a bespoke, stand-alone reminder app was installed on a local computer connected to the Bristol Royal Infirmary network, so staff there could set up reminders for their patients.

Challenges

At the start of the eRAPID research programme, a new release of Medway was being installed and QStore IT integration could not begin until this was complete. Significant time and input from the local eRAPID co-applicant/surgical lead was required to establish working relationships with key staff. Between April 2016 and August 2017, there were four instances of loss of integration with Medway, totalling 145 days, resulting in clinicians being unable to access eRAPID results through the EPR system. A separate ‘Administration and Report’ section in QStore was developed, allowing staff to log in directly to QStore via a secure login to administrate and see participant results and notification reports during periods of downtime. From May 2016 to March 2018, there were five instances of reminder system downtime, totalling 39 days.

Design

The eRAPID study was a prospective, single-centre, randomised (1 : 1), two-arm, parallel-arm study with an internal pilot phase. Full inclusion and exclusion criteria and trial procedures are described in the published protocol. 75 The sample included patient participants receiving systemic treatment (i.e. chemotherapy with or without targeted therapies) for breast, colorectal or gynaecological cancer at Leeds Cancer Centre. HCPs (i.e. senior oncologists, trainees and senior nurses) involved in the care of those patients were invited to participate. Consenting patients were randomised to either the intervention arm (i.e. eRAPID added to usual care) or the control arm (i..e. usual care). Randomisation was stratified by cancer site (breast, colorectal and/or gynaecological), sex (breast and colorectal cancer patients only) and previous chemotherapy. The eRAPID patients were asked to complete symptom questionnaires weekly, and also when experiencing symptoms, for 18 weeks. The HCPs were trained on how to access eRAPID reports through EPRs and they saw participants in both arms.

Outcome measures

The primary outcome measure was Functional Assessment in Cancer Therapy Scale – General (FACT-G) Physical Well-Being (PWB) score at 18 weeks. 80 The FACT-G PWB score was chosen as the primary outcome as it performed well in previous trials and covered the most common treatment-related symptoms and their impact on patients’ lives. We anticipated a cumulative effect of the intervention; therefore, we selected 18 weeks (i.e. end of chemotherapy) as the primary end point, and 6 and 12 weeks as the secondary end points.

The main secondary outcome was cost-effectiveness assessed using clinical process measures (i.e. acute admissions, calls to the emergency hotline and other hospital calls), use of health-care services [including general practitioner (GP) contacts, medications and personal expenses], EuroQol-5 Dimensions, five-level version (EQ-5D-5L), utility scores81 and utility scores derived from the European Organization for the Research and Treatment of Cancer Quality of Life Questionnaire (EORTC QLQ-C30)82 (see Health economics analysis of eRAPID in systemic treatment). We evaluated adherence to the eRAPID intervention by patients and clinicians. Other secondary outcome measures included questionnaires on aspects of patient self-efficacy and activation [e.g. the Self-Efficacy for Managing Chronic Disease 6-Item Scale (SES),83 Cancer Behaviour Inventory–Brief (CBI-B)84 and Patient Activation Measure85 (PAM)].

To separate the possible effects of the online intervention questions from the outcome measures, patients were given the outcome questionnaires on paper to complete at home and return by post at baseline and at 6, 12 and 18 weeks post randomisation. Clinical process measures (e.g. acute admissions and calls to hospital staff) were downloaded from the electronic hospital records. A subset of participants (where feasible within the funding time frame) were also assessed at 12 months for further cost-effectiveness analysis to examine any potential longer-term impact of the intervention on QoL and clinical processes.

Statistical analysis

The sample size for the full trial was based on a patient-reported outcome measured by the FACT-G PWB subscale at 18 weeks. At 80% power and 5% significance, a sample of 176 patients per arm was necessary to detect a 2-point change in this subscale score. A 2-point change in score corresponds to a small to moderate effect size (0.3). Allowing for 30% attrition, a minimum of 252 patients per arm was required (i.e. 176/0.7), equalling 504 patients in total across the two arms.

Multivariable, multilevel, mixed-effects, repeated measures models were used to compare the differences in patient outcome scores between the treatment arms over time [using PROC MIXED in SAS^® software version 9 (SAS Institute Inc., Cary, NC, USA)]. The model was adjusted for stratification factors (i.e. cancer site, sex and previous chemotherapy), time (i.e. follow-up week), treatment arm, treatment arm by time interaction, baseline PWB score and age at study entry as fixed effects. Participant and participant by time interaction were modelled as random effects (by including the ‘repeated’ statement in the analysis model).

Multiple correlation structures appropriate to the repeated measures setting were fitted, including compound symmetry, auto regressive and unstructured. The model using unstructured correlation had the lowest Akaike information criterion score (measure of model fit) and the fewest assumptions on the structure of the data. For these reasons, the unstructured correlation structure was chosen for the final model. The normality of the residuals was checked and shown to be approximately normal.

Missing data patterns were explored. Baseline covariates were found to be associated with missingness and the missing data mechanism was suggested to be missing at random. Therefore, multiple imputation by predictive mean matching was performed. The main primary analysis was performed on the imputed data, followed by sensitivity analysis using all observed data. An ad hoc exploratory subgroup analysis of patients with metastatic cancer and patients with non-metastatic cancer was performed on the imputed data. A negative binomial model was used for the analysis of the process of care measures. No adjustment for multiple testing was undertaken. Post hoc analysis was conducted to explore associations between patient adherence to online symptom reporting and clinician use of symptom data in clinical consultations. A median split was used to categorise patients as ‘adherent’ or ‘non-adherent’ and a clinician use composite score was calculated per patient from clinician feedback checklists.

Patient and staff qualitative feedback

Qualitative feedback on the eRAPID IT system was obtained from a subset of patients and clinical staff through end-of-study interviews. Additional written feedback was also received through free-text comments in (1) patient end-of-study feedback forms and (2) clinician feedback forms (completed by clinicians when reviewing online symptom reports during consultations).

Interviews were transcribed and analysed using thematic analysis. 86 Free-text comments were collated and reviewed for key content and summarised under overarching key themes. A report of the qualitative data is provided in Appendix 1, eRAPID systemic RCT qualitative findings.

Internal pilot phase

Between 22 January and 23 September 2015, 134 patients were identified, of whom 25 were excluded after screening as ineligible and 87 out of 109 eligible patients consented, giving a consent rate of 80%. Only 13 participants (15%) withdrew. No significant IT problems were encountered. The research team improved the methods to gather data from the electronic records on clinical process measures (e.g. admissions and changes to treatment). The overall recruitment (consent rate of > 60% of eligible patients, recruitment rate of > 10 patients per month) and attrition (attrition rate of < 30%) targets were met and the TSC recommended progression to the main trial.

Full trial

Between 22 January 2015 and 11 June 2018, 1484 patients were assessed for eligibility; 702 were excluded pre approach and a further 92 were excluded after being approached for full eligibility assessment. From 690 fully eligible patients, 182 (26.4%) declined participation and 508 (73.6%) consented and were randomly assigned to the eRAPID intervention (n = 256) or usual care (n = 252). A total of 55 HCPs participated (16 oncologists, 25 trainees, two trust oncologists, 10 senior nurses and two pharmacists). Of these HCPs, 19 were regular team members and saw > 10 patients in the eRAPID arm; the remaining 36 were either trainee oncologists who rotated between different teams or temporary staff.

Primary outcome

For the main primary analysis on the imputed data, there were no clinical or statistical significant effects of the eRAPID intervention at 18 weeks [adjusted difference in mean FACT-G PWB score between treatment arms 0.20, 95% confidence interval (CI) –0.81 to 1.20; p = 0.6992]. There was a statistically significant effect of the eRAPID intervention at 6 weeks (adjusted difference in mean FACT-G PWB score 1.08, 95% CI 0.12 to 2.05; p = 0.0280) and 12 weeks (adjusted difference in mean FACT-G PWB score 1.01, 95% CI 0.05 to 1.98; p = 0.0395). Neither point estimate is clinically relevant (change > 2 points); however, 2 is contained in the upper end of the interval for 6 weeks, so clinical relevance cannot be ruled out.

The descriptive responder analysis of FACT-G PWB ‘change scores’ at an individual level suggests that patients receiving the eRAPID intervention experience less clinically relevant deterioration at 6 and 12 weeks and more symptom stability (symptom control).

A sensitivity analysis on the unimputed observed data yielded similar estimated differences at the follow-up time points. There was no statistically significant effect of the eRAPID intervention at 18 weeks (adjusted mean difference in mean FACT-G PWB score –0.05, 95% CI –1.08 to 0.98; p = 0.920). There was a borderline statistically significant difference at 12 weeks (adjusted mean difference in mean FACT-G PWB score 0.9, 95% CI –0.02 to 1.89; p = 0.055), but no statistically significant difference at 6 weeks.

One limitation of using imputed data sets is the inability to robustly combine type-III global effects. Because both analyses present similar results, the global effects from the unimputed data give a practical estimate for the overall effect of treatment over time. The overall effect of eRAPID intervention, compared with usual care over time, was not significant (treatment × time interaction p = 0.1233).

As expected in the model, the baseline FACT-G PWB score was highly statistically significant (p < 0.0001), as was the effect of cancer site. Both participants with gynaecological cancer (p = 0.0026) and participants with colorectal cancer (p = 0.0006) had better FACT-G PWB scores than participants with breast cancer, after adjusting for baseline score.

A pre-planned exploratory subgroup analysis was conducted on the imputed data to determine whether or not there was any evidence of heterogeneity of the intervention effect for patients with metastatic cancer and patients with non-metastatic cancer. The primary analysis was repeated separately on the two new data sets. There was no evidence of an eRAPID effect in the metastatic cancer subgroup.

In the non-metastatic cancer arm, no effect was seen at 18 weeks (adjusted difference in mean FACT-G PWB score 0.04, 95% CI –1.08 to 1.17; p = 0.9430). However, a small, statistically significant, positive effect for the eRAPID intervention was observed at 6 weeks (adjusted difference in mean FACT-G PWB score 1.45, 95% CI 0.32 to 2.58; p = 0.0112) and 12 weeks (adjusted difference in mean FACT-G PWB score 1.13, 95% CI 0.07 to 2.19; p = 0.0362). At both time points, the CIs do not rule out a clinically significant effect.

Process of care measures

There was no effect for the eRAPID intervention on admissions [estimate 0.13, standard error (SE) 0.18, Wald χ² = 0.71, incidence rate ratio (IRR) 1.14, 95% CI 0.84 to 1.53; p = 0.4003] or calls/visits to acute oncology (estimate 0.05, SE 0.12, Wald χ² = 0.20, IRR 1.05, 95% CI 0.84 to 1.31; p = 0.6516). No between-arm differences were found for the measures of chemotherapy delivery: dose reductions, delays, number of delivered cycles and chemotherapy discontinuation.

Adherence to the intervention

A total of 3314 online symptom reports were completed during the study. The mean number of reports per patient was 12.7 [standard deviation (SD) 12.6, median 14.0, range 0–117)]. The median compliance per patient (defined as completed/expected reports adjusted for withdrawals) was 72.2%. Compliance per week gradually decrease between week 1 (71.9%) and week 17 (56.7%; lowest compliance), and the average weekly compliance was 64.7%.

A total of 18,867 individual online symptoms were reported (as part of 3314 logins); of those, 323 (1.7%) were severe and 4342 (23.0%) were moderately severe. Emergency alerts were activated in 29 cases (of 3314 online completions; 0.9%) and serious symptoms not requiring immediate medical attention were reported on 163 occasions (4.9%). The vast majority of self-reported symptoms triggered self-management advice (2714/3314; 81.9%).

This was a pragmatic trial including all staff assessing patients during chemotherapy (i.e. 15 oncologists, 25 trainees, two staff doctors, 10 senior nurses and two pharmacists). Clinicians saw a variable number of patients (mean 14.3, SD 23.4, median 3, range 1–123). Clinician engagement was evaluated by self-reported use of the online data using a checklist completed after each consultation with a patient allocated to the eRAPID intervention. Overall, 787 forms were completed. Broadly, in about 70% of cases clinicians reported looking at the online reports, in 56% of cases they used them at least ‘somewhat’ and in 62% of cases found them at least ‘somewhat useful’. On 35–45% of occasions, the online reports were not considered at all, suggesting somewhat limited buy-in from clinicians and suboptimal intensity of the eRAPID intervention.

In an exploratory logistic regression analysis, we found that greater patient age, higher baseline FACT-G PWB score and clinician use of the data during consultation were predictors of higher levels of patient adherence (defined as ≥ 72.2% of expected completions). Furthermore, patients with high levels of adherence had significantly better FACT-G PWB scores over time (eRAPID intervention: adjusted mean FACT-G PWB score 21.7, 95% CI 21.0 to 22.5; usual care: adjusted mean FACT-G PWB score 20.2, 95% CI 19.4 to 21.0; p < 0.000) than those with lower adherence. The significant differences were seen at 12 and 18 weeks.

In a post hoc exploratory logistic regression analysis, we found that greater patient age, higher baseline FACT-G PWB score and clinician use of the data during the consultation were associated with higher levels of patient adherence (defined as ≥ 72.2% of expected completions). In a post hoc, mixed-effects model, patients allocated to the eRAPID intervention with high levels of adherence had significantly better FACT-G PWB scores over time (high level of adherence: adjusted mean FACT-G PWB score 21.7, 95% CI 21.0 to 22.5; low level of adherence: adjusted mean FACT-G PWB score 20.2, 95% CI 19.4 to 21.0; p < 0.0001) than those with lower adherence. The significant differences were seen at 12 and 18 weeks.

Patient and staff qualitative feedback

A total of 45 patients and 19 staff were interviewed, and 185 patients provided usable feedback in end-of-study questionnaires. A total of 123 staff feedback forms contained free-text comments. The qualitative findings provide vital insight into use of the eRAPID IT system and aid interpretation of the quantitative results. A summary of the results are outlined in Table 2 and a report is provided in Appendix 1, eRAPID systemic RCT qualitative findings.

From the patient feedback, it was apparent that many patients found the system easy to use and felt that it enhanced connections with the hospital – ‘. . . it’s like keeping in touch . . . without making an appointment to see anyone’ (study ID 44) – and helped guide decision-making: ‘. . . gave me and my family more confidence to manage side effects especially early on in the treatment. Gave me “permission” to contact the hospital if I was worried by side effects’ (study ID 402). Patients also mentioned that the system served to heighten personal involvement with health care: ‘felt good to record my symptoms every week – felt like I was taking an active role in my treatment’ (study ID 123).

Patients valued explicit use of their AE symptom reports by staff: ‘at clinic visits I had sometimes forgotten about some of the symptoms I had experienced over the 3-week period since my last visit, however the consultants using the system were able to raise issues raised – prescribing additional medication if necessary’ (study ID 216). Patients were disappointed when staff failed to use the data and there was evidence that this could affect intervention adherence: ‘no feedback from anyone – was expecting at least someone discussing usage of system but didn’t happen at all after using it for three times – so stopped using it’ (study ID 82). In staff feedback, many acknowledged the benefit of the eRAPID IT system and felt that it had been useful in practice, although others expressed reservations about the relevance of AE reports and the potential increase in consultation length.

Secondary patient-reported outcomes

Strength and limitations of the trial

This was a large, pragmatic trial of a complex online intervention with a classic parallel-arm design that achieved good recruitment and retention rate. We recruited > 50% of the total chemotherapy population, despite the strict requirement for patient internet access. This confirms the generalisability of our findings.

However, it was a single-centre trial because of the complexities of setting up the secure online system integrated with EPRs. We engaged a large number of clinicians with different skills (e.g. oncologists and nurses) across several clinical teams, which is of course a strength. However, in practice, each clinician saw a relatively small number of patients allocated to the intervention and needed ongoing reminders to engage in the trial. These difficulties appear to have affected patient adherence and no doubt diluted the intervention effects.

Measurement of outcomes, resource use and costs

Quality-adjusted life-years were calculated based on patient health state utility values obtained from the EQ-5D-5L questionnaire at baseline, at 6, 12 and 18 weeks and at 12 months, which were mapped to EuroQol-5 Dimensions, three-level version (EQ-5D-3L), utility values using the van Hout mapping algorithm. 81,88–90

All health-care resource use over the trial period was collected using hospital records and patient-completed questionnaires at 6, 12 and 18 weeks and 12 months, and was converted to costs using appropriate UK unit costs. 91–93 Total costs for each patient were calculated as the sum of costs assigned for hospital services, community health and social services, medications and the intervention cost.

Adjusting for baseline imbalance

Multiple regression analysis was used to estimate differential mean QALYs and predict adjusted QALYs, controlling for utility at baseline. 94,95

Missing data

Where there were missing quality-of-life or cost follow-up data, multiple imputation methods were used to generate estimates of missing values based on the distribution of observed data, as per recommended best practices for economic evaluation alongside clinical trials. 96–98

Cost-effectiveness analysis

The primary analysis consisted of a cost–utility analysis over the 18-week period of the main trial and included adjustment for baseline variables and imputation of missing data. The incremental cost per QALY gained by patients randomised to use the eRAPID IT system compared with patients randomised to usual care was calculated, producing an incremental cost-effectiveness ratio (ICER).

Sensitivity analyses were conducted to explore the impact of assumptions made in the primary analysis and alternative perspectives for analysis.

The level of sampling uncertainty around the ICER was explored using a non-parametric bootstrap to generate 10,000 estimates of incremental costs and benefits. This was used to illustrate the probability that use of the eRAPID IT system is cost-effective at a range of cost-effectiveness threshold values. 99,100 The secondary analysis was undertaken from a societal perspective.

Resource use and costs

Resource use over the trial and associated costs are presented in Appendix 1, eRAPID health economics report, tables 1–4. There was little difference in use of health-care resources and associated costs between arms, and multiple regression analysis indicated that the difference in total costs between arms was not statistically significant (£25.28, 95% CI –£1240.91 to £1167.69; p > 0.05).

Quality of life

Patient and carer EQ-5D-5L scores are presented in Appendix 1, eRAPID health economics report, table 5. EQ-5D-5L scores decreased over the trial period in both arms, but scores were higher at each time point in the eRAPID arm than in the usual-care arm. Multiple regression analysis indicated that there was no significant difference in total QALYs gained between arms (0.003, 95% CI –0.005 to 0.011; p < 0.05).

Missing data

The EQ-5D-5L scores were complete at all follow-up points for 349 (69%) patients. Resource use questionnaires were complete at all follow-up points for 205 (40%) patients.

Cost-effectiveness results

Cost-effectiveness results are presented in Table 3. The eRAPID arm had both the highest QALY gain and the lowest costs over the trial period. This indicates that the eRAPID IT system dominates usual care and may be preferred as the more cost-effective option.

Bootstrapped estimates of the incremental costs and effects indicated that use of the eRAPID IT system may be a more cost-effective use of resources than usual care. However, mean differences in costs and QALYs were small and not statistically significant and, consequently, at the NICE-recommended cost-effectiveness threshold of £20,000 per QALY gained, the eRAPID IT system had a 55% probability of being cost-effective.

Results of sensitivity analyses are presented in Appendix 1, eRAPID health economics report, table 9. The results of the primary analysis were robust to all sensitivity analyses explored, with the eRAPID IT system dominating usual care in each case.

Results of the secondary analysis are presented in Appendix 1, eRAPID health economics report, table 9. This analysis shows that the eRAPID IT system has lower costs and higher QALY gains, indicating that its use may be cost-effective even when societal perspective costs are included.

Based on data collected for the subsample with 12-month follow-up data (see Appendix 1, eRAPID health economics report, table 11), the probability that eRAPID is cost-effective compared with usual care is 36%.

Principal findings

The primary within-trial cost-effectiveness analysis indicated that use of the eRAPID IT system may be more cost-effective than usual care for the management of AEs in patients receiving systemic treatment for breast, colorectal or gynaecological cancer. Higher QALY gains and lower costs were observed in the eRAPID arm than in the usual-care arm. However, these mean differences were small and not statistically significant, and the cost-effectiveness acceptability curve showed that eRAPID had only a 55% probability of being cost-effective at the NICE-recommended cost-effectiveness threshold of £20,000 per QALY gained. Nevertheless, the results were robust to all scenarios explored in the sensitivity analyses.

Secondary analysis indicated that use of the eRAPID IT system may also be more cost-effective than usual care when costs are analysed from a societal perspective, although the difference in costs remained small.

Strengths and weaknesses of the economic analysis

A strength of this analysis lies in the design of the study. The randomised controlled design enabled the collection of good-quality data for use in this within-trial cost-effectiveness analysis. In addition, the use of linked hospital data has provided more robust data and meant that some of the biases common in self-reported data have been avoided.

However, the lack of a completely connected system for all health-care records meant that self-reported data was nevertheless relied on to record use of non-hospital health-care services. Consequently, some biases from self-report may have remained. In addition, health-care resource use questionnaires were not well completed, with the number of missing resource use questionnaires increasing over the trial period. This resulted in a large proportion of patients with missing community health-care use data.

The internal pilot phase of the trial provided a valuable opportunity to review the design of the trial and the feasibility of data collection mechanisms. Following the pilot phase, some changes were made to the data collection forms (hospital resource use was no longer collected in patient case report forms because this would be collected directly from hospital records). However, this produced some inconsistencies in the data collected for patients in the pilot phase compared with in the main trial, which could be viewed as a limitation.

The exploratory analysis considered the longer-term impact of using data from a subsample with 12-month post-randomisation data, but results should be treated with caution given the small sample size and differences in patient population compared with the main trial (particularly the increased proportion of patients with metastatic cancer).

Meaning of the study

The economic analysis has followed recommended best practices for economic evaluations conducted in UK settings and consequently uses QALYs based on responses to the EQ-5D-5L as the primary outcome. The use of QALYs, preferably calculated based on the EQ-5D-5L, is mandated by NICE to ensure consistency across evidence used to inform their recommendations and guidelines. However, QALYs focus on the measurement of patient health and other important aspects, such as patient experience and satisfaction, may not be captured. Furthermore, patient responses to the EQ-5D-5L were necessarily collected at fixed time points according to pre-defined data collection schedules. However, owing to the intermittent and cyclical nature of patient health care and the potential for AEs during chemotherapy cycles, the use of fixed data collection points may mean that fluctuations in patient health are missed. 101

The results of this analysis indicate that the eRAPID IT system may be more cost-effective than usual care in the management of patients receiving systemic treatment for breast, colorectal or gynaecological cancer. However, the differences in QALYs and costs were small in real terms and were not statistically significant. Therefore, other factors (such as patient and health-care provider acceptability of the system) are also likely to be important.

Given that the cost of developing the eRAPID IT system, which was covered in earlier research grants,102 represents a sunk cost, it was not included in the cost-effectiveness analysis presented here. Instead, the intervention cost was calculated based on the costs that would apply if the system was implemented more widely, including the maintenance cost of QTool and the cost of a training manual for each patient. However, the costs included are likely to overestimate the per-patient cost because the software maintenance cost was split only between the patients in the eRAPID intervention arm, but in practice would be split across a much larger patient group. In addition, patient feedback on the training manual indicated that this resource could be simplified and shortened significantly, which would further reduce the cost in producing it. Consequently, the costs associated with the intervention may be overestimated and the resulting cost-effectiveness of the eRAPID IT system underestimated. For example, during the 1-year period between April 2019 and March 2020 there were approximately 1761 new patients with breast, gynaecological or lower gastrointestinal cancers across hospitals in Leeds, compared with 508 in the intention-to-treat population of this study. However, the maintenance costs for the system may apply to each region that implements eRAPID, with each having a different number of eligible patients across which to split the costs. Therefore, we split only by the study population to present a conservative estimate of likely costs and cost-effectiveness.

Stakeholders interviews

A total of 48 patient interviews and nine relative interviews were carried out (across both sites) during radiotherapy (n = 14), at the end of treatment (n = 21) and re-interviewed a proportion of patients (n = 13) 6 weeks after treatment. A total of 26 key staff members (across both sites), including 10 clinical oncologists, eight nurse specialists, two clinic nurses, four radiographers and two service managers, were interviewed.

Care pathway maps were developed for patients receiving radiotherapy for prostate, cervical, anal, rectal, endometrial and vulval cancer at both sites (six in total). Analysis of the pathways determined the most appropriate clinical contacts. At the Leeds site, this was the oncology bleep holder (current treatment) and the medical team (post treatment). At the Manchester site, it was the treatment hotline (current treatment) and medical team (post treatment). The contact numbers were integrated into QTool advice.

The complexity of the symptom patterns and treatment pathways necessitated development of new design features in the online symptom report to accommodate overlap from AEs associated with different types of treatment and time dependencies across three disease sites. The resulting questionnaire had a more complex structure than that developed for systemic therapy. This higher degree of complexity was also reflected in the self-management advice. Two websites were developed, one tailored to each hospital, because there was variation in local practices.

Patient adverse event item development

A scoping review was conducted of existing, established, validated items from the EORTC QLQ-C30 and related modules identified for inclusion in the patient symptom report. This measurement system is appropriate for patients undergoing cancer surgery and is established and routinely used as a PROM in international trials to improve patient outcomes and care. Items of relevance to symptoms and complications experienced by patients after cancer-related major abdominal surgery (e.g. surgery for oesophageal, gastric and hepatopancreatobiliary upper gastrointestinal cancer) were selected. Face-to-face cognitive interviews with a purposively selected broad sample of upper gastrointestinal cancer surgery patients aged ≥ 18 years evaluated the suitability, acceptability and comprehension of the items selected. Participants’ thoughts during item completion were explored using verbal probes to identify problems experienced with understanding or responding to items. Sampling and data analyses proceeded iteratively until no further refinements were required.

Algorithm development

Clinical algorithms to determine when tailored feedback/advice would be triggered based on patient-reported symptoms were developed from the following:

1. Data from eRAPID patient-generated symptom report data from consecutive patients discharged from the Bristol site from 15 April 2016 to 15 August 2017 following cancer-related upper gastrointestinal surgery. Patients completed the symptom self-report at discharge (baseline), twice in the first week post discharge (days 2 and 7) and weekly thereafter for 8 weeks (although participants could complete the symptom report at additional time points if they wished, e.g. if experiencing new symptoms).

2. Data from weekly telephone interviews between participants and a study researcher/research nurse, focusing on patients’ reflections of the nature and severity of post-discharge symptoms and contact with and use of health-care services in the previous week.

3. Two stakeholder meetings with clinicians responsible for the clinical care of patients undergoing major cancer-related upper gastrointestinal surgery and a patient representative. The objective of the stakeholder meetings was to determine symptom severity thresholds to categorise which symptoms or symptom combinations would be considered clinically important/concerning at a given time point post discharge. Discussions focused on identifying bothersome or concerning (e.g. indicative of a potential AE) symptoms that may warrant advice from either a patient or a clinician perspective, and appropriate advice content. A patient participant attended stakeholder meetings to facilitate discussion of patients’ perspectives on the findings from earlier PPI work (described in Patient and public involvement specific to eRAPID surgery system development) around which symptoms patients find concerning and may warrant advice/reassurance.

4. End-of-study interviews with a purposively sampled 10% subset of participants, approximately 10 weeks post discharge. Interview topic guides were developed, which were informed by literature, analysis of nurse-led telephone consultations and study team discussion (including a professor of upper gastrointestinal surgery and cancer nurse specialist). Data were analysed in an iterative, cyclical manner as data collection proceeded to develop and refine the clinical algorithms.

Algorithms were further refined by comparing patient symptom report data and associated actions and advice triggered by the system with (1) data from the weekly telephone interviews; (2) advice and reassurance about symptoms and problems provided by a cancer nurse specialist and participants during audio-recorded routine (i.e. usual care) telephone consultations; and (3) post-discharge clinical events or outcomes of participants (e.g. re-intervention, re-admission to hospital or visit to a GP or primary health-care provider) identified from hospital EPRs, re-admission alerts and patient-reported health-care use.

Patient advice development

Three data sources informed the development of tailored patient self-management advice: (1) end-of-study interviews, (2) audio-recorded routine telephone consultations between a cancer nurse specialist and participants during the first week (at approximately days 2 and 7) post discharge and (3) a scoping review and thematic content analysis of NHS-recommended patient recovery information and patient information leaflets (conducted between September 2016 and January 2017). Relevant targeted sections of audio-recordings were transcribed verbatim. Data were analysed to identify themes relating to post-operative advice and reassurance, and options for appropriate phrasing and terminology for the self-management advice were identified from transcriptions and from NHS patient information and patient information leaflets. Draft advice was iteratively refined through discussion with the study team and clinicians involved in the stakeholder meetings.

Patient adverse event item development

A total of 30 relevant items were identified from 95 potential items included in seven validated EORTC questionnaires. Five additional items were included following consultation with clinicians, resulting in a total of 35 items included in the patient symptom report. Item comprehensiveness and acceptability were verified through interviews with 18 participants (men, n = 16; mean age, 66.3 years).

Algorithm development

Some 300 patients were screened, of whom 130 (43%) were eligible and invited to participate and of whom 61 (47%) consented. A total of 59 (97%) participants (men, n = 34; mean age, 61 years) accessed the eRAPID IT system a total of 459 times, and 444 completed symptom self-reports were obtained. Weekly follow-up interviews with these 59 participants (men, n = 34; mean age, 61 years) and end-of-study interviews with seven participants (men, n = 4; mean age, 58 years) led to refinements to patient symptom report branching logic, and the addition of nine subitems to distinguish between ‘typical’ and ‘atypical’ symptoms, and eight subitems to distinguish between ongoing and resolved symptoms.

One patient representative, six nurses, two dieticians and one surgeon from four hospital sites participated in two stakeholder meetings. Findings indicated that algorithms should account for date since hospital discharge to allow more accurate evaluation of whether or not symptoms were typical and atypical (i.e. indicative of an AE) depending on the individual patient’s stage of recovery. The importance of reassurance that some symptoms are expected during recovery was also identified.

A comparison between data sets (e.g. system-triggered actions/advice, clinician consultations and clinical events/outcomes) from 27 participants (men, n = 18; mean age, 63 years) reporting clinically significant symptoms resulted in several further refinements to the patient symptom report questionnaire (e.g. addition of items to evaluate wound problems, feeding tubes and contact with HCPs) and algorithms (e.g. refinement of symptom severity thresholds and addition of subitems and branching logic) over an 8-month period of system testing.

Patient advice development

Analysis of data from 15 routine care telephone consultations between a cancer nurse specialist and eight participants (men, n = 5; mean age, 62 years) identified four themes relating to post-operative advice and reassurance around pain, other physical symptoms, diet/nutrition and managing recovery sought by participants. 135 Data from seven end-of-study participant interviews (men, n = 4; mean age, 58 years) confirmed that the self-management advice was acceptable and relevant. This was combined with data from analysis of 28 patient information leaflets identified from 16 NHS trusts and three cancer support charities to develop draft self-management advice for 22 symptoms, tailored to individual patient symptom severity. Draft advice was reviewed and refined at two stakeholder meetings.

Piloting of the eRAPID surgery system

Consecutive patients from each centre who had undergone cancer-related upper gastrointestinal surgery were recruited. Eligible patients had undergone cancer-related upper gastrointestinal surgery, were ready for hospital discharge to their home, were aged ≥ 18 years, had access to a computer/mobile device and the internet at home, and were fluent in English. Participating patients were asked to prospectively complete the symptom self-report at discharge (baseline), twice in the first week (at approximately days 2 and 7) and weekly thereafter for 8 weeks. However, participants were told that they could complete the symptom report at additional time points if they wished (e.g. if they experienced new symptoms). Data analysis was conducted separately for the two sites and focused on symptom report response rates, reasons for non-completion, data completeness, frequency and severity of symptoms reported, and any system actions generated.

Rates of participation of relevant HCPs in monitoring the shared e-mail inbox to which alerts triggered by the eRAPID IT system were sent were evaluated. Although HCPs at the Bristol site were able to access eRAPID IT system patient symptom report data through EPRs during patient post-discharge follow-up consultations (telephone and/or face-to-face clinic appointments), which either took place as part of usual care or resulted from the eRAPID IT system alert, it was not possible to monitor rates of use owing to administrative and time constraints. However, weekly interviews were undertaken with participating HCPs to explore their engagement with and perspectives of the eRAPID IT system (described in Qualitative interviews with patients and health-care professionals).

Qualitative interviews with patients and health-care professionals

Semistructured qualitative interviews with participants were conducted weekly (all participants) to coincide with eRAPID symptom self-report time points and at the end of the study (a 10% purposive subsample). Interviews explored patients’ experiences of using the system and receiving tailored feedback and/or advice to contact a health-care professional. Weekly interviews were conducted with the lead cancer nurse specialist (Bristol site only) to examine engagement with the system and perspectives on the frequency, nature and relevance of patient contact resulting from eRAPID advice/alerts. End-of-study interviews with the lead nurse and hospital dietician (Bristol site only) were also conducted to explore views towards the practicality and usefulness of the eRAPID IT system in the context of routine clinical care. Interview topic guides were developed, informed by literature, nurse-led telephone consultations and study team discussion. Interviews containing data relevant to the study objectives were transcribed verbatim (targeted transcription) and textual data analysed in accordance with the principles of thematic analysis. Emerging codes were refined through iterative discussion with the study team as analyses progressed and until thematic saturation was reached.

Technical performance

The technical functionality of the eRAPID IT system, including integration with EPRs and the number of failed logins, HCP e-mail alerts and participant text/e-mail reminders, was monitored throughout the study.

Piloting of potential outcome measures for a main trial

The timing and acceptability of potential outcome measures for use in a future definitive main trial were examined. Participants completed paper-based FACT-G and EQ-5D questionnaires at baseline, week 3 (week 4 post surgery) and week 7 (week 8 post surgery). Methods for health-care resource use data collection were also examined. Patients completed a paper-based health resource use questionnaire at the end of the study, which recorded use of prescription and non-prescription medication and other costs associated with patients’ recovery from surgery. In addition, patients were telephoned by the study researcher at weekly intervals to record the frequency and reasons for HCP (e.g. GP and community nurse) contact.

Piloting of the eRAPID surgery system

Between August 2017 and March 2018 at the Bristol site, 109 patients were screened; 41 (38%) were eligible and 29 (71%) consented to participate (men, n = 19; mean age, 64.2 years). A total of 17 (25%) patients were ineligible because they did not undergo their planned surgical procedure and 15 (14%) were ineligible because they lacked a computer/mobile device/internet access. There did not appear to be any major differences between patients who did and patients who did not take part. Seven (24%) participants withdrew because they felt too tired/unwell or had experienced a prolonged re-admission to hospital. A total of 11 (55%) of the 20 Birmingham site patients screened between November 2017 and October 2018 consented to participate, of whom two (18%) withdrew. At the Bristol site, response rates to the eRAPID IT system symptom report questionnaire at key assessment time points exceeded 70% (range 72–93%), excluding day 2 to 3 post discharge (55%) when patients might have felt too unwell to access the system. The most frequent reasons for non-completion were participants starting adjuvant chemotherapy (12%) and not wanting to complete the symptom report at that time point (10%). These findings indicate that it may be beneficial to incorporate non-responses at critical time points as an indicator of potentially concerning symptoms that could trigger an alert to clinicians. At the Birmingham site, eRAPID symptom report response rates exceeded 60% at all time points (range 63–100%), excluding week 8 (response rate 50%) when participants are expected to have achieved a good level of recovery from surgery.

At the Bristol site, all four members of the upper gastrointestinal specialist team responsible for the care of patients post discharge after surgery, including three cancer nurse specialists and one dietician, agreed to participate in the study. All nurses participated in accessing and acting on alerts triggered by the eRAPID IT system and communicated to the shared e-mail account. Although exact frequencies are not available, all four team members utilised eRAPID patient symptom report data to inform their routine patient follow-up care (during nurse–patient telephone consultations and/or face-to-face clinic appointments). At the Birmingham site, all three upper gastrointestinal cancer nurse specialists in the team participated in the study and accessed and acted on eRAPID alerts. Ad hoc administrative discussions with nursing staff at the Birmingham site indicated that the clinical nurse specialist team considered the lack of integration with hospital EPRs a barrier to the usefulness of the eRAPID IT system for contributing to routine clinical care and limited their engagement with the system.

Between 30 August 2017 and 17 April 2018, 29 Bristol site participants completed the eRAPID symptom report a total of 197 times (median 9, range 1–11); of these reports, nine (5%) triggered a level-3 action (clinician alert), 72 (36%) triggered a level-2 action (advice to contact a clinician), 76 (39%) triggered a level-1 action (self-management advice) and 40 (20%) triggered no advice (Table 4). Seven (78%) of the nine level-3 complications triggering HCP feedback and patient advice to contact a HCP immediately were reported during the first 4 weeks post discharge. Potentially concerning symptoms decreased thereafter, coinciding with milder/expected symptoms. A further eight level-3 alerts triggered by baseline (i.e. pre-discharge) symptom report completion were removed from the data set following discussion with clinicians because they were considered irrelevant because the patients were still in hospital. A total of 60% (n = 43) of the 72 level-2 actions (i.e. participant advised to call a HCP) were triggered during the first 2 weeks post discharge. Just 15% (n = 11) of level-2 actions were triggered after 6 weeks post discharge. Most (n = 48, 63%) of the 76 level-1 actions (i.e. self-management advice) were triggered after 3 weeks post discharge, most commonly at weeks 4 and 5. A total of 27 (68%) of the 40 level-0 actions (i.e. no participant advice required) were triggered after 5 weeks post discharge. The findings support literature indicating that most complications and problematic/distressing symptoms occur within 30 days post surgery, with reduced symptoms from 4 weeks post discharge. 137 At the Birmingham site between 10 November 2017 and 22 November 2018, participants completed the eRAPID symptom report a total of 63 times (median 7, range 1–10); of these reports, one (2%) triggered a level-3 action, 20 (32%) triggered a level-2 action, 36 (57%) triggered a level-1 action and six (10%) triggered no advice. The pattern of timing of these actions was similar to the pattern at the Bristol site. One level-3 alert was triggered at week 3. Most (n = 14, 70%) level-2 actions were triggered in the first 3 weeks post discharge, most (n = 25, 69%) level-1 self-management advice was triggered after 3 weeks post discharge and the majority (n = 5, 83%) of level-0 feedback was triggered after 4 weeks post discharge.

At the Bristol site, most level-3 actions (n = 7, 78%) were generated by pain symptoms. Level-2 actions were most commonly triggered by problems with wounds (44%), appetite loss (36%), fever (24%), physical function (13%) and nausea/vomiting (11%). Level-1 actions were generated most frequently for problems with fatigue (n = 58) and/or pain (n = 27), physical function (n = 22), nausea or vomiting (n = 20) and constipation (n = 20). The eRAPID IT system ensures that patients are not overburdened with self-management by limiting advice to a maximum of the six most clinically concerning symptoms per eRAPID self-report completion. 135 Level 1 actions were triggered for a median of three (range 1–6) symptoms per level-1 action triggered.

A total of 109 weekly interviews were conducted with patient participants. Analyses indicated that some level-3 system actions had been triggered by symptoms for which the patient had already received treatment from another care team or symptoms relating to pre-existing conditions and, therefore, that no action needed to be taken by the participating cancer nurse specialist team. This indicated that further minor refinements to the level-2 and level-3 algorithms were needed to ensure that symptoms relating to underlying health conditions and relevant symptoms that are already being well-managed do not trigger HCP alerts. Corresponding weekly telephone interview data were available for 36 (50%, 17 participants) of the 72 level-2 actions triggered. In most cases (23/36, 64%), participants contacted a HCP for only new or previously unreported symptoms, in accordance with level-2 advice. In eight (35%) cases, all participants with new symptoms adhered to advice and contacted a HCP. Of these, four participants contacted a GP and four contacted the cancer nurse specialist team. This led to four participants receiving advice and/or reassurance from clinicians, three undergoing clinical interventions or investigations (e.g. appointments and blood tests) and one receiving antibiotics for a wound infection. In 15 cases (65%), participants appropriately followed advice not to contact a HCP, either because they had an upcoming clinical appointment and/or because their symptoms were already being appropriately managed. Among the 13 (36%) instances when participants did not adhere to level-2 advice, reasons for not contacting a HCP included feeling that their symptoms did not warrant reporting (e.g. because they already had an appointment scheduled, n = 6, 46%), not recalling receiving the advice (n = 6, 46%) and choosing not to contact a HCP (n = 1, 8%).

Five Bristol site participants were re-admitted to hospital for complications (fevers/infections, n = 3; nausea and vomiting, n = 1; and bile leak, n = 1). These participants completed the eRAPID symptom report a total of 14 times within 7 days prior to re-admission, resulting in two level-3 events (14%), nine level-2 events (64%) and one level-1 event (7%). This indicates that, in most cases, the eRAPID IT system produced appropriate advice regarding the symptoms reported by patients. Interviews indicated that some other participants may not have completed the symptom self-report prior to hospital re-admission because they had recognised the severity of their symptoms and instead contacted the care team without being prompted by the eRAPID IT system.

Qualitative interviews with patients and health-care professionals

Of the 173 potential weekly interviews, 109 (63%) were conducted, including at least one interview with all 29 patient participants. Of these, 35 interviews from 16 participants were selected because they contained data of relevance to the research questions, in accordance with the principles of targeted transcription. Most participants reported positively on their experience of using the eRAPID IT system. Data confirmed that patients found the system valuable and reassuring, particularly with regards to receiving confirmation that their symptoms were typical for their stage of recovery and having a method for objectively tracking progress with symptom improvement over time. Patients reported feeling empowered to take a central role in their own recovery because the system provided new and tailored symptom self-management advice and reminded them of and reinforced the guidance that they had previously received. Patients also stated that the system enabled them to overcome feelings of isolation and uncertainty post discharge136 (see Appendix 1, eRAPID radiotherapy developmental paper).

Health-care professional interviews (n = 10 nurses/dieticians) indicated that the system was a valuable adjunct to routine surgical follow-up care, particularly for interpreting the clinical significance of patients’ symptoms and progress in their individual recovery. Most system-initiated patient contact was regarded as appropriate and timely and, in some cases, directly informed patients’ subsequent clinical management. HCPs indicated that symptom self-report data collected by the system were useful to facilitate and enhance nurse–patient telephone consultations and acknowledged the value of the system to provide reassurance to patients outside ongoing nurse–patient consultations. The potential for technical problems with hospital systems (e.g. accessing EPRs) and problems with accessing computers in clinic were regarded as factors that may influence the use of the eRAPID surgery system in routine clinical practice.

Technical performance

At the start of the eRAPID research programme, a new Medway release was being installed, which delayed QStore IT integration. Establishing working relationships with key staff took time and significant input from the local eRAPID co-applicant/surgical lead. Between April 2016 and August 2017 there were four instances of loss of Medway integration, resulting in clinicians being unable to access eRAPID results through EPRs. There were two instances where integration was lost for 1 day and one instance where integration was lost for 6 days. A Medway software update resulted in the longest incident of integration loss, from 13 June 2016 to 28 October 2016. A separate ‘Administration and Report’ section in QStore was developed, allowing staff to log in directly to QStore using a secure login to administrate and see participant results and notification reports during periods of downtime. From May 2016 to March 2018, there were five instances of reminder system downtime, totalling 39 days.

Piloting of potential outcome measures for a main trial

The FACT-G and EQ-5D questionnaires were completed by 19 (66%) and 20 (69%) participants at week 4 post discharge and by 15 (52%) participants at week 8. Health resource use forms were completed by 15 (52%) participants.

Introduction

To help to understand the value of the eRAPID system in supporting patient care during systematic cancer treatment, a qualitative substudy was embedded within the RCT. Interviews and written feedback were obtained from patients and staff to gain insight into how the intervention was received and used, and its impact on clinical management. Summarised below are the main findings from this qualitative work, which helped to underpin and provide wider context to the main trial findings.

Methods

Analysis

Results

Conclusions

• The qualitative aspects of the eRAPID systemic RCT provided useful additional information on how the intervention worked and was well received by patients and staff.

• Both participant groups reported that eRAPID was easy to use/access.

• From the patient point of view, the positives of eRAPID were that it was a trusted information and support tool and helped to maintain a link with the hospital.

• Patients expressed a range of ways in which the system could be improved (through advances with the technology) and better use of the data (more engagement from the clinical teams).

From the staff perspective, some professionals were positive about the potential benefit of the system for assisting with consultation preparation and focusing discussions. In practice, we realise that the design of the RCT meant that some staff had limited exposure to patients allocated to the eRAPID intervention, which meant that they could forget to check for or use the patient reports.

The qualitative findings presented here are limited in a number of ways. For example:

• Interviews and feedback from patients were gathered at the end of the 18-week study period rather than routinely throughout. This may bias or limit what was recalled about how eRAPID was used.

• Clinicians were relied on to provide written summaries of how they had used the eRAPID data in clinical encounters. Observations or audio-recordings would have provided supplementary insight on how patient data were used in consultations.

As adoption of PROMs into current practice is becoming increasingly commonplace, it remains vital to explore how systems work in practice to ensure that they meet the needs of both patients and clinical teams. An important element of this will be to help support and train both patients and staff on methods to maximise the use and clinical value of PROMs data.

Background

An economic evaluation was conducted to estimate the cost-effectiveness of the eRAPID system for AE reporting from home or hospital compared with usual care for patients receiving systemic treatment (chemotherapy or targeted therapies) for colorectal, breast or gynaecological cancers. The analysis was guided by the recommendations of the NICE methods guide. 146

Methods

An embedded health economic evaluation was conducted alongside the eRAPID trial, consisting of a within-trial analysis evaluating the costs and benefits accruing to patients over the 18 weeks of the trial. An exploratory analysis was undertaken at 12 months post randomisation.

Results

Uncertainty analysis

Bootstrapped estimates of the incremental costs and incremental effects are plotted on the cost-effectiveness plane in Figure 7. This shows the joint distribution of the incremental costs and effects for patients using the eRAPID system compared with usual care. The majority of the points lie to the right of the y-axis, indicating that use of the eRAPID system is likely to increase QALYs gained. The spread of points both above and below the x-axis indicates the uncertainty around the impact of the eRAPID system on costs.

FIGURE 7.

Cost-effectiveness plane: eRAPID vs. usual care.

The probability that use of the eRAPID system is cost-effective compared with usual care is presented on the cost-effectiveness acceptability curve shown in Figure 8. Based on data collected over the 18 weeks of the main trial and a cost-effectiveness threshold of £20,000 per QALY, the eRAPID intervention has a 55% probability of being cost-effective.

FIGURE 8.

Cost-effectiveness acceptability curve: eRAPID vs. usual care.

Discussion

Introduction

Traditionally, radiotherapy AEs are reported by clinicians; however, the timing and accuracy of reporting and clinician–patient communication can be improved by asking patients to report their own symptoms via PROMs. Well-timed and appropriate documentation of AEs throughout the patient journey is essential to both understand the toxicity profile of the treatments and comprehensively support patients. 105 eRAPID is a programme to develop, evaluate and implement an online system for reporting and managing AEs during cancer treatment, developed for use with radiotherapy patients, including those receiving radical treatment to the prostate. Patients report AEs from home during and post treatment. Thus, the selection of appropriate self-report items to report treatment-related AEs was crucial to the success of the programme in capturing patient experiences.

In the preliminary work, we identified the AE and existing validated PRO questionnaires in radical prostate radiotherapy through a systematic review of RCTs, staff and patient interviews, and local clinical use. We found that PROMs were not used to report acute AEs (AEs occurring within 3 months of completion of radiotherapy) within trials and, for late AEs, the augmentation of existing questionnaires with additional items was required to provide comprehensive coverage. 110 The aim of this developmental phase was to reach consensus on appropriate items to include for online reporting of AEs using Delphi consensus methodology with an expert panel of HCPs and patients.

Method

The validated PROM questionnaires mapped against the identified AEs were MPQ,115 EPIC SF (the Expanded Prostate Cancer Index Composite Short Form),113,155 National Cancer Institute (NCI) PRO-CTCAE,33 EORTC QLQ-C30,82 EORTC QLQ-PR25,111 FACT-G,80 FACT-P (Functional Assessment of Cancer Therapy – Prostate),156,157 SF-36,112,158 eRAPID PRAE76 and Toxicity and Response Criteria of The Eastern Cooperative Oncology Group. 114 The MPQ was selected because of the comprehensive coverage of urinary and bowel symptoms. However, the MPQ did not provide full coverage and certain AEs needed more in-depth coverage (e.g. hormonal, sexual, social and emotional functioning). To enable the selection of appropriate PRO AE items to augment the MPQ, 45 participants took part in the Delphi process. This included 20 patients and 25 HCPs from St James’s Institute of Oncology in Leeds and The Christie Hospital Manchester.

Patient-reported outcome items relating to 11 symptom areas not covered by the MPQ were ranked by participants via a secure online questionnaire and survey tool (Bristol Online Surveys, now known as Online Surveys159) over a series of three rounds. Participants were sent a copy of the MPQ and advised that patients would complete the eRAPID questionnaire weekly/fortnightly. Participants were asked to rank questions corresponding to symptoms and side effects not covered in the MPQ and were asked to consider (1) how well the question reflected the symptom/side effect in question, (2) how well the questions would fit in with the MPQ and (3) the potential questionnaire burden for patients. Figure 9 below shows an example question from the survey on weight gain.

FIGURE 9.

Example question from the Delphi survey on weight gain; participants indicate preference via radio buttons and can comment in the expanding free-text box.

Typically, the number of PROM options varied from as small as two (Figure 9) to as many as six (when there were more PROM options to select from). Participants were advised to select whether some additional items should be included in the eRAPID questionnaire. Figure 10 below is a schematic representation of the process and shows which areas were put forward in each round. Where the pre-determined 55% consensus threshold was not achieved, the two highest options and a summary of free-text comments were presented back to participants to inform further selections.

FIGURE 10.

Schematic showing question areas put forward to participants on the Delphi panel in each round.

Results

Consensus was reached for 4 out of 11 areas in round 1 of the survey (range 62–77%), including weight gain, relationship/social impact, inclusion of questions on quality of orgasm (68%) and frequency of erections (56%). In round 2, consensus was achieved on the remaining seven areas, including depression, performance status and fatigue (range 55–89%). Generally, EORTC and eRAPID PRAE items were preferred owing to their conciseness and significance to patients. Interestingly, when HCPs were given feedback from patients, their subsequent selection of items converged more with the patient view (and vice versa). In round 3, weekly reporting of sexual issues was deemed excessive by all participants: 43% of patients suggested monthly reporting, and 57% of patients and 77% of HCPs recommended less frequent reporting. Table 16 below provides a summary of the results.

Conclusion

This work has contributed to the development of patient reporting of AEs during radiotherapy treatment. It highlighted the lack of patient-reported AEs in the acute setting in prostate radiotherapy RCTs (systematic review). The MPQ was selected and supplemented with additional items, resulting in a comprehensive questionnaire covering hormonal, emotional and sexual AEs experienced by prostate cancer patients. It was determined that a baseline assessment followed by a monthly or quarterly assessment would be welcomed, and online reporting of sexual problems seemed to be acceptable to patients. Endorsed by patients and staff, the included questions are a balance of items clinically relevant to HCPs and those reflecting patient voices. The items have been utilised in the pilot study of the eRAPID intervention during pelvic radiotherapy. We recommend including patient-reported outcomes to evaluate radical radiation therapy along the treatment trajectory and advocate working towards a consistent approach to PROM assessment of radiation therapy-related AEs.

Development of the eRAPID system for pelvic radiotherapy

Radiotherapy for pelvic malignancies typically involves daily attendance for treatment, often with surgery, chemotherapy and biological therapies. 160–163 These produce acute and late effects, peaking after treatment finishes (typically bowel and urinary toxicity). 106,110 To capture these late effects, radiotherapy patients need a longer and often multidisciplinary follow-up, and, for this reason, the pathways may be unclear. 164,165

Stage one: the development of the eRAPID symptom questionnaires, clinical alerts and self-management advice, mapping of patient care pathways and the integration of the system into the EPRs at both cancer centres. Clinical algorithms were developed and tested to establish symptom severity thresholds and symptom-specific management advice, including locally tailored self-management information, was developed on two specifically designed websites.

Stage two: a usability study at both the Leeds and the Christie centres was conducted and refinements were made to components of the intervention based on feedback from patients and staff. The intervention would then be ready for evaluation in the randomised pilot study.

Stage one of development of adverse event questionnaires and items for prostate, gastrointestinal and gynaecological cancers

A systematic review and mapping exercise with oncologists found that the MPQ116 was the most comprehensive PROM used in RCTs for acute and late AEs during and after radical radiotherapy for prostate cancer. A Delphi exercise with patients and staff was undertaken to reach consensus on which items from other questionnaires would augment the MPQ. Eleven items were added to cover hormonal, emotional, sexual, social and general domains. The outcome of the development work was an augmented eRAPID pelvic radiotherapy AE questionnaire fully addressing AEs associated with radiotherapy for prostate, anal, rectal, cervical, vaginal, endometrial and vulval cancer (Figure 11).

FIGURE 11.

Core questionnaire and additional items in the prostate and chemoradiation groups used to develop the eRAPID pelvic radiotherapy AE questionnaire for prostate, gastrointestinal and gynaecological cancer.

Stage two of development usability study of the erapid system in radiotherapy

The aim of usability testing was to evaluate how the eRAPID system worked in real-world clinical settings with patients and staff at Leeds Cancer Centre and the Christie Manchester. Patient and staff feedback adapted and improved system performance and integration into standard practice prior to the pilot.

Usability testing was carried out with patients receiving radical and post-surgical radiotherapy for prostate cancer (n = 10) and patients receiving pelvic chemoradiotherapy for cervical or endometrial cancer (n = 10). Patients were given a demonstration of eRAPID, a unique username and password, a user guide and a feedback booklet to complete during the evaluation. Patients were asked to complete the symptom monitoring questionnaire (electronic PRO radiotherapy AE) each week during radiotherapy treatment: 5 weeks for cervical and endometrial patients and 4 weeks for prostate patients.

Health professional staff involved as lead clinicians for the patient were provided with an eRAPID training session and a practitioner information sheet, which included instructions on how to retrieve patient data from the EPR. If a patient reported a severe toxicity, a nominated oncologist received an e-mail alert.

Usability of eRAPID and general feedback were collected in a range of methods:

• Patient evaluation – feedback form, semistructured interview and end-of-study system usability questionnaire.

• Patient health-care episodes – health economic questionnaire and collection of demographic and clinical data.

• Staff evaluation and system performance issues – interviews and ongoing feedback from oncologists and radiographers. Issues discussed and acted on at the regular radiotherapy workgroup meeting.

Results

Testing took place between March and August 2016. Twelve staff took part, including oncologists, a clinical nurse specialist and a radiographer. Ten prostate patients from Leeds and 10 gynaecological patients from the Christie participated, with a combined total of 92 completions.

Following feedback (Figure 12) and prior to the pilot, tweaks were made to the wording and timings of the, and significant changes to the, grading of some alerts.

FIGURE 12.

Type and amount of feedback collected from staff and patients during the study.

Feedback was positive and indicated that the system had value for patients. There was a high level of completion of symptom reports and the reports were not seen as burdensome. Reminders (text or e-mail) were developed for patients. Feedback from staff taking part in the usability study indicated that reminders, training and support were essential to the regular use of eRAPID, even for those staff who had had substantial involvement in the development.

Technical IT issues were encountered. Although the feasibility of developing further use of eRAPID in radiotherapy across two sites was established, there were significant issues throughout the development phase relating to the integration of IT systems, with both trust EPRs and radiotherapy-specific systems, such as MOSAIQ (radiotherapy delivery software). Issues with integration into the CWP at the Christie was a key cause of delay to the start of the recruitment for the pilot RCT.

There were limitations to the field testing. It was possible to evaluate the system using only questionnaires developed for gynaecological and prostate cancer. However, some of the lessons learned were general and applicable across the questionnaire design, irrespective of cancer site.

Conclusions

The outcome of the development work was an augmented eRAPID pelvic radiotherapy adverse event questionnaire that fully addressed AEs associated with radiotherapy for prostate, anal, rectal, cervical, vaginal, endometrial and vulval cancer. Extensive mapping of radiotherapy care pathways revealed a complex situation and required the development of new design features in the electronic questionnaire to take into the account the larger number of dependencies within pelvic radiotherapy, which encompassed the need to understand the side effects of multiple cancer types and combined systemic radiation treatment pathways.

Introduction

Radiotherapy in cancer care improves patient survival, but the potent multimodal regimes166 lead to significant treatment-related pelvic side effects. 167 PROMs can improve the timing and accuracy of symptom reporting,28 the timely preventative management and survival, particularly when captured electronically. 70 Self-management interventions are effective in reducing cancer symptom severity and improving QoL. 168

Patient-reported outcome measures in radiotherapy110,165 are becoming more integrated into clinical oncology practice. 169 Electronic reporting for post radiotherapy symptoms is being trialled for bowel and lung toxicity. 170,171 We are piloting the eRAPID system in pelvic radiotherapy in two cancer groups with differing treatment and side effect profiles. We aimed to determine the feasibility and acceptability of the eRAPID intervention for patients and staff. Primary end points included recruitment/attrition rates and adherence to symptom reporting. Secondary aims were to establish the number of hospital calls/admissions and aid the selection of outcome measures for a definitive trial.

Methods

Procedures

Procedures and randomisation are described in the published protocol. 75 Figure 13 is a summary of patient and process outcome measures collected. 74

FIGURE 13.

Study diagram. BL, baseline; PAM, patient activation measure; SUS, system usability scale.

Results

Recruitment, attrition, and completion rates

Over 15 months, a total of 253 patients were identified. Of the 228 fully eligible patients, 61 declined and 167 patients were consented (recruitment rate of 73.3%). A total of 103 males and 64 females took part [mean age prostate radiotherapy 70.0 (SD 7.0) years; chemoradiotherapy 52.1 (SD 15.1) years]. The chemoradiotherapy group had a lower education level and lower co-morbidity rates than the prostate group.

Sixteen patients withdrew: 10 (12.0%) from the eRAPID arm and six (7.1%) from the usual-care arm (one patient in the chemoradiotherapy usual-care arm died). Most eRAPID withdrawals (n = 8) were from the chemoradiotherapy arm (n = 7, gynaecological cancers) by 6 weeks. Three withdrawals from each treatment group were noted in the usual-care arm. Reasons for withdrawals included ‘too ill’ (n = 3), ‘too much to think about/too busy’ (n = 2), ‘being well, no symptoms’ (n = 1), ‘wanting to move on’ (n = 2) and ‘not confident using internet/prefer paper’ (n = 2).

A total of 88% of eRAPID patients and 92% of usual-care patients completed the study; completion was lower in the chemoradiotherapy arm (eRAPID, 80%; usual care, 90%) than the prostate radiotherapy arm (eRAPID, 95%; usual care, 93%). Completion of outcome measures at 24 weeks was excellent in the prostate radiotherapy arm (eRAPID, 95%; usual care, 95%) but less in the chemoradiotherapy arm (eRAPID, 63%; usual care, UC). Chemoradiotherapy patients had lower outcome completion rates across all time points (between 75% and 60%) than prostate radiotherapy patients (between 91% and 97%).

Adherence to eRAPID online symptom reports and algorithm generation

Prostate radiotherapy patients had high adherence to weekly symptom reporting but adherence was lower for chemoradiotherapy patients (Figure 14). This low adherence was because of the patients with cancers of the cervix/endometrium (11/23 patients completed 0–3 online reports). Non-adherent patients were younger (mean age 35.8 years; 9/11 were aged < 40 years) than adherent patients in the chemoradiotherapy arm (mean age 45.2 years; 4/12 were aged < 40 years).

FIGURE 14.

Adherence to eRAPID online symptom reports: proportion of QTool completions by protocol and treatment group.

The algorithms activated alerts for severe symptoms (4% for chemoradiotherapy patients and 0.5% for prostate patients) (Figure 15). Moderately severe symptoms were similar between treatments (40.5% and 38.3%, respectively) and just over 50% of reports generated self-management advice.

FIGURE 15.

Activated eRAPID algorithms during study: percentage of QTool algorithm generation by treatment group.

Clinical process measures

The mean number of calls to hospital staff in the 24 weeks was small [1.3, chemoradiotherapy arm (SD 1.9) vs. 1.6, usual care arm, (SD 2.2); 0.4, prostate radiotherapy arm (SD 1.1) vs. 0.1 usual care arm (SD 0.3)]. The mean number of unscheduled hospital visits was close to 0 in both treatment groups.

Data trends

Less deterioration over time was reported by participants in the eRAPID chemoradiotherapy arm than in the usual-care arm. Greater differences were seen at 6 weeks for FACT-G scores, EORTC QLQ-C30 summary scores, QLQ-C30 Global health/QoL score and EQ5D-VAS scores, remaining after adjustment for baseline scores. Figures 16 and 17 show graphically these trends over time. No trends were observed from the prostate radiotherapy arm.

FIGURE 16.

Mean EORTC QLQ-C30 Global QoL scores. Higher QLQ-C30 scores = better QoL.

FIGURE 17.

Mean FACT-G scores. Range 0–108; high score = good function.

Conclusion

This pilot established the feasibility and acceptability of online symptom reporting with severity-dependent immediate advice for patients undergoing two main modalities of pelvic radiotherapy (chemoradiotherapy for lower GI cancers) and radiotherapy for early prostate cancer in two major UK cancer centres. To the best of our knowledge, this is one of the first studies in pelvic radiotherapy to add immediate severity-dependent advice to address self-reported symptoms and enable clinicians to view online reports from within EPRs.

A recruitment rate of > 70% and a withdrawal rate of < 10% was achieved. Prostate radiotherapy patients were the most consistent and prolific symptom reporters, followed by patients with anorectal cancers consistent with previous chemotherapy studies. 36,72 Younger gynaecological patients did not adhere to the online symptom reporting; they received more intensive concurrent chemoradiotherapy and had a lower level of education. Those gynaecological patients who did report had more severe and prolonged problems (e.g. bowel and urinary urgency, pain, skin reactions) than those who did not, suggesting that when they were ill they were less likely to report. These findings suggest that eRAPID may not be useful for women aged < 40 years with lower education levels receiving intensive chemotherapy for advanced cervical cancer.

eRAPID did not generate extra hospital visits or calls. Alerts for severe symptoms were low. The potential outcome measures performed well. Trends towards less symptom deterioration over time were demonstrated for eRAPID chemoradiotherapy patients (particularly at 6weeks); however, baseline score imbalances and missing data from gynaecological patients must be considered. Limited benefits were seen for prostate radiotherapy patients; however, the willingness to report and self-manage symptoms could be translated into longer-term supportive care interventions. 172

The encouraging feasibility and patient adherence findings for prostate and anorectal cancers justify further studies exploring reporting and management of persistent late effects. This study adds to evidence on clinical benefits from regular online symptom monitoring in cancer and informs future randomised trials/health services development projects.

Introduction

The eRAPID radiotherapy feasibility pilot included an embedded qualitative element. Through this work we captured the views of patients and staff around the use and value of the eRAPID system to help to understand how the intervention was received, its impact on patient care and the scope for future refinement.

Methods and analysis

Qualitative feedback was gathered from several sources:

• End-of-study semistructured interviews with patients explored views about the practicalities of using the system, the relevance of the symptom reports and self-management advice, and the impact on their treatment or consultations.

• Free-text comments in end-of-study patient surveys in which a series of questions explored experiences of using eRAPID and suggestions for refinement.

• End-of-study semistructured staff interviews explored challenges and benefits of the practicalities of using eRAPID, as well as the impact on clinical work and relationship with patients.

• Staff end-of-study feedback captured further information on the challenges and benefits of the system.

Interviews were digitally recorded and transcribed verbatim, and data were subsequently managed using NVivo 11 software. Written feedback and comments from the end-of-study feedback questionnaires were analysed for content. A core group of the eRAPID research team conducted the qualitative analysis collaboratively173 using the principles of thematic analysis. 86

Results

Eleven patients and four staff from Leeds were interviewed, three staff from the Christie provided written feedback and 61 patients from across both sites returned the end-of-study survey (Table 18).

Conclusions

The qualitative data collected in the eRAPID radiotherapy pilot RCT adds nuance to our understanding of the feasibility and acceptability of online approaches to patient monitoring. Both staff and patients found the system easy to use. From the patients’ perspective, the reassurance the system provided was one of the key positive elements. However, there were issues with motivation in maintaining symptom report completions over the longer term, especially when feeling well. Staff, on the other hand, felt that it would be useful to consider extending the follow-up period to capture late effects of treatment. Further attention will need to be given to the patient views of the benefit of continuing monitoring over the longer term when they may want to move on from frequent surveillance.