Robot-assisted training compared with an enhanced upper limb therapy programme and with usual care for upper limb functional limitation after stroke: the RATULS three-group RCT

Article history

The research reported in this issue of the journal was funded by the HTA programme as project number 11/26/05. The contractual start date was in January 2014. The draft report began editorial review in December 2019 and was accepted for publication in May 2020. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Copyright statement

© Queen’s Printer and Controller of HMSO 2020. This work was produced by Rodgers et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2020 Queen’s Printer and Controller of HMSO

Problems after stroke

Stroke is the commonest cause of complex disability in the UK. 4 The potential consequences of stroke are broad and include problems with mobility, arm and hand function, speech, cognition, vision, swallowing, continence, mood, pain and fatigue. 5 There are 1.2 million stroke survivors in the UK and > 100,000 people have a stroke each year. 6 Almost two-thirds of stroke survivors leave hospital with a disability. 6

Upper limb problems after stroke

Upper limb problems frequently occur following stroke, comprising loss of movement, co-ordination, sensation and dexterity, which lead to difficulties with everyday activities. Approximately 80% of people with acute stroke have upper limb motor impairment, and, of those with limited arm function early after stroke, 50% still experience problems after 4 years. 7 The strongest predictor of recovery is severity of initial neurological deficit; patients with severe initial upper limb impairment are unlikely to fully recover their arm function, with a clear impact on their quality of life. 8 Patients report that loss of arm function is one of the most distressing long-term consequences of stroke. Improving upper limb function has been identified as a research priority by stroke survivors, carers and clinicians. 9

Repetitive functional task practice to improve upper limb recovery

At the time of designing the RATULS trial, a systematic review7 reported that treatments that focused on increased intensity and repetitive task-specific practice showed promise for improving motor recovery. Since then, the body of evidence has increased, with a 2014 Cochrane overview of systematic reviews10 reporting moderate-quality Grading of Recommendations Assessment, Development and Evaluation evidence showing that arm function following a stroke can be improved by the provision of at least 20 extra hours of repetitive task training. In addition, a 2016 Cochrane review11 found that repetitive functional task practice for stroke patients was associated with improved arm function [standardised mean difference (SMD) 0.25, 95% confidence interval (CI) 0.01 to 0.49], hand function (SMD 0.25, 95% CI 0.00 to 0.51) and ADL (SMD 0.28, 95% CI 0.10 to 0.45).

Robot-assisted training for upper limb recovery

First proposed in the late 1980s, robot-assisted training enables stroke patients with moderate or severe upper limb impairment to perform repetitive tasks in a highly consistent manner, tailored to their motor abilities. At the time of designing the RATULS trial, the most recent (2008) Cochrane review12 of electromechanical and robot-assisted arm training after stroke reported outcomes from 328 patients who participated in 11 trials. The largest study had 55 participants13 and four of the 11 studies evaluated the Massachusetts Institute of Technology (MIT)-Manus robotic gym system (InMotion commercial version, Interactive Motion Technologies, Inc., Watertown, MA, USA). 13–16 The systematic review12 reported improvements with electromechanical and robot-assisted arm training in terms of arm motor function (SMD 0.68, 95% CI 0.24 to 1.11) and motor strength (SMD 1.03, 95% CI 0.29 to 1.78), but no improvement in ADL (SMD 0.29, 95% CI –0.47 to 1.06).

In 2010, the Veterans Affairs (VA) RCT was published evaluating the MIT-Manus in four centres in the USA. 17 This trial was the largest trial of robot-assisted training at the time, recruiting 127 patients with moderate or severe upper limb impairment who were ≥ 6 months after stroke. Participants were randomised to receive robot-assisted training, intensive comparison therapy or usual care. Robot-assisted training and intensive comparison therapy both consisted of 36 sessions, each lasting 1 hour, over 12 weeks. The trial found that robot-assisted training did not improve upper limb motor function at 12 weeks compared with intensive therapy or usual care (the primary outcome). However, participants who received robot-assisted training had significantly better results at 12 weeks on the Stroke Impact Scale (SIS) than those who received usual care. In secondary analyses, improvements were seen in motor function for those who received robot-assisted training compared with those who received usual care, but not compared with those who received intensive comparison therapy, at 36 weeks. The added costs of delivering robot-assisted training or intensive comparison therapy were compensated for by the fact that the health-care costs were lower than for usual care. 18

The Robot-Assisted Training for the Upper Limb after Stroke trial

The RATULS trial evaluated robot-assisted training using the MIT-Manus robotic gym system. This was compared with an upper limb therapy programme [named the enhanced upper limb therapy (EULT) programme] of the same frequency and duration, and with usual post-stroke care. Robot-assisted training and the EULT programme involved repetitive task practice and were provided in addition to usual care.

Trial aim and objectives

Trial design

The RATULS trial was a three-group, pragmatic, observer-blind, multicentre RCT with an embedded economic analysis and a process evaluation. Participants were randomised to receive one of the following: robot-assisted training (in addition to usual NHS care), an EULT programme (in addition to usual NHS care) or usual NHS care in accordance with local clinical practice.

Trial setting

The trial was conducted in four NHS centres in the UK. Each centre comprised a hub site, which was a stroke service in an NHS hospital (Queen’s Hospital, Barking, Havering and Redbridge University Hospitals NHS Trust; Northwick Park Hospital, London North West University Healthcare NHS Trust; Queen Elizabeth University Hospital, NHS Greater Glasgow and Clyde; and North Tyneside General Hospital, Northumbria Healthcare NHS Foundation Trust) with a MIT-Manus robotic gym system, plus several spoke sites (a total of 18), which were stroke services in adjacent NHS trusts and community services.

Participants who were randomised to receive robot-assisted training or EULT travelled to a hub stroke unit to receive these interventions. Participants who were randomised to usual care were treated by their local stroke service.

Trial participants

Adults with a first-ever stroke who fulfilled the following criteria were eligible to participate in the RATULS trial.

Case ascertainment, recruitment and consent

Participants were recruited from both incident and prevalent stroke populations. Participants were sought from a number of settings in primary and secondary care, including stroke units, outpatient clinics, day hospitals, community rehabilitation services and general practices. The trial aimed to recruit similar numbers of participants within 3 months of stroke, > 3–12 months after stroke and > 12 months to 5 years after stroke.

Screening assessment

Once informed consent was obtained, a screening assessment was performed by the local trial centre co-ordinator or NIHR LCRN staff. The following data were collected: demography, stroke details, comorbidity and upper limb function (ARAT score19,20). If a patient fulfilled the trial inclusion and exclusion criteria, the local trial co-ordinator/NIHR LCRN staff proceeded with the baseline assessment. If it was not possible to complete the baseline assessment on the same day as the screening assessment, eligibility for the trial was reconfirmed on the day of the baseline assessment.

Baseline assessment

The following baseline data were collected: stroke severity (measured using the National Institutes of Health Stroke Scale21), cognitive function (measured using the Montreal Cognitive Assessment22), language skills (measured using the Sheffield Screening Test for Acquired Language Disorders23), upper limb impairment [measured using the Fugl-Meyer Assessment (FMA)20,24 (motor and sensory arm sections)], ADL (measured using the Barthel ADL Index25), quality of life [measured using the EuroQol-5 Dimensions, five-level version (EQ-5D-5L)26], upper limb pain (measured using a numeric rating scale27) and current upper limb rehabilitation treatments.

In addition, patients were given a self-completion questionnaire containing pre-trial resource use questions [an adaption of the Client Services Receipt Inventory (CSRI)28–30].

Randomisation

Randomisation was conducted by the local trial co-ordinator/NIHR LCRN staff following completion of the baseline assessment. A central independent web-based service, hosted by Newcastle University Clinical Trials Unit, was used. Participants were stratified according to trial centre, time since stroke (< 3 months, 3–12 months, > 12 months) and severity of upper limb functional limitation (ARAT score categories:19 0–7, 8–13, 14–19 and 20–39), and randomised, using permuted block sequences, 1 : 1 : 1 to receive robot-assisted training, EULT or usual care. The sequences were prepared by an independent statistician prior to the start of enrolment.

Randomisation groups

Outcome measures

Undertaking outcome assessments

Outcomes were assessed at 3 months (± 7 days) and 6 months (± 7 days) following randomisation, and were undertaken in two stages.

Stage 1 was a self-completion postal questionnaire consisting of the SIS42 (at 3 and 6 months) and the adapted CSRI28–30 resource use questions (at 6 months only).

Stage 2 was a face-to-face assessment with a researcher who was masked to the randomisation group. The following data were collected: ARAT,19 FMA24 (total upper-extremity score), Barthel ADL Index,25 EQ-5D-5L,26 upper limb pain27 and adverse events. At the end of the 6-month stage 2 assessment, participants were given a further self-completion questionnaire and were asked to return this by post. This questionnaire contained time and travel resource use questions. 48,49

Staff training

All staff received trial-specific training at the start of their involvement, with refresher training provided throughout the trial. Staff performing trial assessments (screening, baseline, outcomes) and delivering trial interventions received training in these aspects. In addition, manuals describing delivery of the interventions and instructions on how to perform the assessments were provided. Video demonstrations of the ARAT and FMA were also prepared and made available for ongoing reference.

Masking

Owing to the nature of the interventions, it was not possible to mask participants or treating therapists to treatment allocation. It was intended that stage 2 outcome assessments would be conducted by a researcher masked to treatment allocation. At each outcome assessment, the researcher was asked to record whether or not they had unintentionally become aware of treatment allocation in conversation with a participant.

Trial withdrawal

No specific withdrawal criteria were preset. Participants could withdraw from the trial at any time and for any reason. A reason for withdrawal was sought, but participants could withdraw without providing an explanation.

Investigators, GPs, stroke physicians and therapists could also withdraw participants from the trial at any time if they felt that it was no longer in their interest to continue, for example because of intercurrent illness or adverse events. Participants were informed that data collected prior to withdrawal would be used in the trial analysis, unless consent for this was specifically withdrawn. Participants who wished to receive no further intervention were not withdrawn from trial follow-up unless they specifically requested this.

Safety evaluation

The safety of robot-assisted training, EULT and usual care was evaluated by examining the occurrence of all adverse events and serious adverse events (SAEs) in accordance with the National Research Ethics Service guidance for non-Clinical Trials of an Investigational Medicinal Product. 50

All adverse events were captured by including the following question in the outcome pro formas: ‘are there any new medical problems since the last study assessment?’.

Events considered to be SAEs were subsequently documented on a separate trial SAE form, and a causality and expectedness assessment was performed. As trial investigators or other members of the research team could become aware of SAEs at times other than at outcome assessment appointments, the SAE form was also used to directly capture these events.

The standard definition for a SAE was used. 51 A SAE is an untoward occurrence that:

• results in death

• is life-threatening

• necessitates hospitalisation, or prolongation of existing hospitalisation

• results in persistent or significant disability or incapacity

• consists of a congenital anomaly or birth defect

• is otherwise considered medically significant by the investigator.

Data management

Trial data were collected on trial-specific paper case record forms, and subsequently entered onto an online database by trial administrators at participating centres.

Sample size

The target sample size was 762 participants (254 participants per group). Responses from 216 participants in each randomisation group would provide 80% power (significance level of 1.7% because of multiple comparisons) to detect a 15% difference in upper limb function ‘success’ between each of the three pairs of treatments (robot-assisted training, EULT and usual care). The target was inflated for 15% attrition, subsequent to protocol publication. The baseline estimate of success was estimated as 30% from the BoTULS trial,35,36 and a difference of between 45% and 30% corresponds to an odds ratio of 1.9.

Statistical analysis of primary and secondary outcome data

Economic analysis

The economic evaluation consisted of a microcosting analysis, a within-trial economic evaluation and a longer-term economic model. 57 Methods are described in Chapter 8.

Parallel process evaluation

A two-stage process evaluation was conducted to understand (1) participants’ and health service professionals’ experiences of robot-assisted training, EULT and usual care and (2) factors affecting the implementation of the trial within and across trial sites. Methods are described in Chapter 7.

Ethics and regulatory issues

The trial sponsor was Newcastle upon Tyne Hospitals NHS Foundation Trust. The trial was conducted in accordance with research governance framework for health and social care58 and good clinical practice. Ethics approval was granted by the National Research Ethics Committee Sunderland (reference number 13/NE/0274). Local NHS approvals were obtained from all participating NHS organisations. Monitoring of trial conduct and data collection was performed by regular visits to all participating trial centres. The trial was managed by a co-ordinating centre based at Newcastle University, Newcastle upon Tyne, UK. An independent Data Monitoring Committee and Trial Steering Committee were in place for the duration of the project.

Amendments made to the trial after it commenced

• Modification of the postal invitation procedure for potential participants by letter (from both primary and secondary care). When the trial commenced, potential participants who were invited by letter were asked to telephone their local trial team if they would like more information. The response rate to these invitations was low; it was suggested that this was because the letter and information sheet were too long, and that people may prefer to respond in writing initially. The letter was revised, a short summary leaflet designed and a postal reply slip created. These revised documents were then used for postal invitations. In addition, the option to re-contact potential participants who did not respond to the initial invitation letter was introduced. This was by follow-up letter or telephone call, according to local preference.

• Introduction of reminders to complete the trial arm rehabilitation logs. To attempt to promote completion of the trial arm rehabilitation logs, regular reminders via text message were introduced.

• Provision of trial newsletters. Regular newsletters were introduced for participants during their time in the trial. This was in response to noting that attrition was higher than anticipated, and aimed to encourage retention in the trial.

• Modification of the parallel process evaluation. Several changes were made to the parallel process evaluation:

  • The initial protocol included interviewing a larger proportion of participants in the robot-assisted training group. However, the need for equal data collection in both the robot-assisted training and EULT groups became apparent, owing to their equivalent complexity. Therefore, the requirement to interview more robot-assisted training group participants was removed.

  • The participant interview time points for the second period of data collection (August 2016 to April 2017) were changed from twice during the first 3 months to once during the first 3 months and once at around 6 months. This change was informed by the analysis of data collected in the first period of data collection (February to June 2016). It allowed continued exploration of views and experiences of treatments, but also additional exploration of the impact of therapy following the end of treatment, plus participants’ experiences of trial participation (e.g. completing trial documentation and outcome assessments).

  • A review of the quantitative outcome assessment data (ARAT score at baseline and at 3 and 6 months) was added to the process evaluation for some participants. Comparison of quantitative data with participants’ subjective experiences captured in the interview data aimed to inform interpretation of the trial.

  • The range of health service professionals to be interviewed was broadened. Rather than interviewing only health service professionals who delivered therapy to RATULS trial participants (as per the original trial protocol), other staff who were involved in the trial were included in interviews. These included principal investigators, NIHR LCRN staff, local site trial co-ordinators and trial administrators. This allowed more information to be captured about factors affecting the implementation of the trial.

• Revision of recruitment target. In the original trial protocol, the target sample size was 720 participants, which included 10% attrition for the primary outcome. As the trial proceeded, it became apparent that dropout was > 10%, and nearer to 15%. The target sample size was therefore increased to 762 participants, to allow for 15% attrition.

Participant recruitment and randomisation

Between 14 April 2014 and 30 April 2018, 770 participants were recruited from the four trial centres, and were randomised as follows: robot-assisted training, 257 participants; EULT, 259 participants; and usual care, 254 participants.

Original predicted, revised predicted and actual cumulative recruitment are shown in Figure 1. Recruitment predictions were revised in July 2016, when target recruitment was increased to 762 participants. Monthly recruitment by trial centre is tabulated in Appendix 2, Table 28.

FIGURE 1.

Predicted and actual cumulative recruitment. The initial recruitment target was 720 participants, to allow for 10% attrition, but the sample size was increased to 762 participants to allow for 15% attrition.

Participant retention and follow-up

Participant flow through the trial is shown in a Consolidated Standards of Reporting Trials (CONSORT) flow diagram (Figure 2).

FIGURE 2.

The CONSORT flow diagram.

Of the 257 participants randomised to robot-assisted training, 233 (91%) completed a 3-month assessment and 223 (87%) completed a 6-month assessment. Of the 259 participants randomised to EULT, 236 (91%) completed a 3-month assessment and 222 (86%) completed a 6-month assessment. Of the 254 participants randomised to usual care, 207 (81%) completed 3-month assessment and 190 (75%) completed a 6-month assessment. There was a lower follow-up rate in the usual care group and, therefore, differential attrition between randomisation groups. Seven participants did not complete the ARAT at their 3-month assessment, but remained in the trial.

The numbers of participants who did not have assessments, and the reasons why, are reported in Appendix 2, Table 29. There were 60 withdrawals by 3 months, and a further 22 withdrawals by 6 months. The main reasons given for withdrawal were not wanting to be randomised to usual care [19/60 (32%) by 3 months], the burden of participation [19/60 (32%) by 3 months and 4/22 (18%) by 6 months] and being unwell [10/60 (17%) by 3 months and 6/22 (27%) by 6 months]. There were two deaths before 3 months and a further two deaths before 6 months. Assessments were not conducted for a further 32 people at 3 months and 49 people at 6 months. The main reasons for this were non-response to attempts to arrange the assessment [14/32 (44%) at 3 months and 37/49 (76%) at 6 months], missing the assessment and non-response to attempts to re-arrange [6/32 (19%) at 3 months and 5/49 (10%) at 6 months], and being unwell and unable to complete the assessment [8/32 (25%) at 3 months and 4/49 (8%) at 6 months].

Timing of participant outcome assessments

Of the 676 assessments scheduled for 3 months, 545 (81%) were carried out at 3 months ± 14 days; the median time from randomisation was 96 days [interquartile range (IQR) 91–102 days]. Of the 635 assessments scheduled for 6 months, 575 (91%) were carried out at 6 months ± 28 days; the median time from randomisation was 187 days (IQR 182–193 days) (Figure 3).

FIGURE 3.

Compliance with time of assessments at (a) 3 months and (b) 6 months. The vertical lines indicate the allowable window of time within which the assessment could take place (± 14 days for 3 months and ± 28 days for 6 months).

Missing data in the measurement scales

There were very few data missing across the scales, and simple imputation did not add many participants to the analyses. For example, the ARAT total score was fully completed for 668 out of 708 (94%) of expected participants at 3 months; after simple imputation, only one more participant was included, as this participant was the only one with missing data who had < 20% of items missing. The number (percentage) of participants with no missing data on the measurement scales is reported by measure and time point in Appendix 2, Tables 30–32. These tables also show how many participants were included after simple imputation. As < 20% of participants were missing data on the primary outcome after simple imputation, multiple imputation techniques were not used.

Participant baseline characteristics

Baseline demographics and stroke characteristics are shown in Tables 1 and 2; they appear balanced across groups. The mean age of participants was 61 years (SD 13.5 years), and 468 out of 770 (60.8%) participants were men. Most participants had experienced a cerebral infarction [613/770 (79.6%)]. The median time from stroke to randomisation was 240 days (IQR 109–549 days), and participants had a mean ARAT score of 8.4 points (SD 11.8 points), a mean FMA motor score of 18.1 points (SD 13.7 points) and a mean Barthel ADL Index score of 14.4 points (SD 3.9 points). Typically, the ARAT score was low [median 3 points (IQR 0–11 points)] and the distribution was positively skewed. The upper limb rehabilitation treatments that were being received at baseline are given in Appendix 2, Table 33. A total of 409 out of 768 (53.3%) participants were receiving physiotherapy and/or occupational therapy at the time of randomisation.

Primary outcome

The proportion of participants who achieved upper limb functional recovery ‘success’ at 3 months in the ITT analysis was 103 out of 232 (44%) in the robot-assisted training group, 118 out of 234 (50%) in the EULT group and 85 out of 203 (42%) in the usual care group (Table 3 and Figure 4). There was little evidence of a statistically significant difference in the incidence of ‘success’ at 3 months when comparing the robot-assisted training group with the usual care group [adjusted odds ratio (aOR) 1.2, 98.33% CI 0.7 to 2.0], the EULT group with the usual care group (aOR 1.5, 98.33% CI 0.9 to 2.5), or the robot-assisted training group with the EULT group (aOR 0.8, 98.33% CI 0.5 to 1.3). Although the unadjusted success rate was 8% higher in the EULT group than in the usual care group, the trial was powered to detect a difference of 15% in success rates between groups. The results of both sensitivity analyses and the PP analysis were consistent with the ITT analysis (see Table 3).

FIGURE 4.

Upper limb functional recovery ‘success’ at 3 and 6 months for robot-assisted training, EULT and usual care. (a) The proportion of participants achieving upper limb functional recovery ‘success’ by randomisation group; and (b) pairwise comparisons of group upper limb functional recovery ‘success’ presented as ORs (98.33% CIs) (ITT population). RT, robot-assisted training; UC, usual care.

The secondary outcome results of upper limb functional recovery ‘success’ at 6 months are shown in Table 3 and Figure 4. The number of participants who achieved upper limb functional recovery ‘success’ at 6 months in the ITT analysis was 103 out of 221 (47%) in the robot-assisted training group, 118 out of 218 (54%) in the EULT group and 81 out of 185 (44%) in the usual care group; all groups showed slight improvement from the 3-month assessment. The unadjusted success rate was 10% higher in the EULT group than in the usual care group, but there was little evidence of a difference in the incidence of ‘success’ at 6 months when comparing the robot-assisted training group with the usual care group (aOR 1.2, 98.33% CI 0.7 to 2.1), the EULT group with the usual care group (aOR 1.6, 98.33% CI 0.9 to 2.8) or the robot-assisted training group with the EULT group (aOR 0.8, 98.33% CI 0.5 to 1.3). The results of both sensitivity analyses and the PP analysis were consistent with the ITT analysis (see Table 3).

Secondary outcomes

Robot-assisted training versus usual care

Action Research Arm Test total score

In the robot-assisted training group, the mean ARAT total scores were 8.5 points at baseline, 15.5 points at 3 months and 16.5 points at 6 months. For the usual care group, the mean ARAT total scores were 8.1 points at baseline, 14.2 points at 3 months and 16.4 points at 6 months. There was little evidence of a difference in the ARAT total score between these groups at 3 months [adjusted mean difference (aMD) 1.4, 98.33% CI –1.1 to 3.8] and 6 months (aMD 1.0, 98.33% CI –1.9 to 3.7) (Table 4 and Figure 5). The MCID for the ARAT total score is 6 points,59 so the changes in mean ARAT scores between baseline and 3 months were consistent with a clinically meaningful difference on this scale for both groups, but the differences between 3 and 6 months, and between groups at 3 and 6 months, were not. The results of the ARAT subscales are given in Appendix 3, Table 34.

TABLE 4 - Comparison of the mean ARAT score at 3 and 6 months between randomisation groups

For time since stroke (to the power of –0.5), baseline ARAT score and trial centre.

This number is positive.

Note

Total score: 0–57.

Time point	Robot-assisted training			EULT			Usual care			Mean difference (98.33% CI)
										Robot-assisted training – usual care		EULT – usual care		Robot-assisted training – EULT
	n	Mean (SD)	Median (IQR)	n	Mean (SD)	Median (IQR)	n	Mean (SD)	Median (IQR)	Unadjusted	Adjusteda	Unadjusted	Adjusteda	Unadjusted	Adjusteda
Baseline	256	8.5 (11.9)	3 (0–12)	259	8.7 (11.9)	3 (0–13)	254	8.1 (11.5)	3 (0–11)	NA	NA	NA	NA	NA	NA
3 months	232	15.5 (19.1)	5 (0–28)	234	17.3 (20.1)	5 (1–33)	203	14.2 (19.6)	3 (0–22)	1.2 (–3.3 to 5.7)	1.4 (–1.1 to 3.8)	3.0 (–1.6 to 7.5)	2.5 (0.0b to 5.1)	–1.8 (–6.2 to 2.6)	–1.2 (–3.6 to 1.2)
6 months	221	16.5 (19.7)	4 (0–29)	218	17.2 (19.9)	5 (0–36)	185	16.4 (21.3)	3 (0–34)	0.1 (–4.8 to 5.0)	1.0 (–1.9 to 3.7)	0.8 (–4.3 to 5.6)	0.3 (–2.6 to 3.0)	–0.6 (–5.2 to 3.9)	0.7 (–1.8 to 3.3)

FIGURE 5.

The ARAT total score at baseline and at 3 and 6 months for robot-assisted training, EULT and usual care. (a) The ARAT total score by randomisation group; and (b) pairwise comparisons of the ARAT total score presented as mean differences (98.33% CIs). RT, robot-assisted training; UC, usual care.

Fugl-Meyer Assessment total upper-extremity score

A similar pattern was seen in the mean FMA total upper-extremity scores, which were 68.9 points at baseline, 76.6 points at 3 months and 78.2 points at 6 months in the robot-assisted training group. By comparison, they were 68.9 points at baseline, 74.2 points at 3 months and 77.9 points at 6 months in the usual care group. There was little evidence of a difference between these groups on this scale at 3 months (aMD 3.1, 98.33% CI –0.0 to 6.5) or at 6 months (aMD 1.6, 98.33% CI –1.8 to 5.2) (Table 5). There is no published MCID for the FMA total upper-extremity score.

TABLE 5 - Comparison of the upper limb impairment (FMA score) at 3 and 6 months between randomisation groups

For time since stroke (to the power of –0.5), baseline FMA score and trial centre.

FMA and time point	Robot-assisted training		EULT		Usual care		Mean difference (98.33% CI)
							Robot-assisted training – usual care		EULT – usual care		Robot-assisted training – EULT
	n	Mean (SD)	n	Mean (SD)	n	Mean (SD)	Unadjusted	Adjusteda	Unadjusted	Adjusteda	Unadjusted	Adjusteda
Total upper-extremity score (0–126)
Baseline	254	68.9 (16.5)	259	69.0 (17.9)	254	68.9 (17.4)
3 months	232	76.6 (22.1)	234	77.8 (22.8)	202	74.2 (23.6)	2.4 (–2.9 to 7.6)	3.1 (–0.0 to 6.5)	3.6 (–1.8 to 8.9)	3.7 (0.5 to 6.8)	–1.3 (–6.2 to 3.7)	–0.5 (–3.4 to 2.6)
6 months	221	78.2 (22.8)	218	79.4 (24.1)	186	77.9 (23.2)	0.3 (–5.2 to 5.8)	1.6 (–1.8 to 5.2)	1.5 (–4.3 to 7.0)	1.8 (–1.8 to 5.3)	–1.1 (–6.5 to 4.2)	–0.2 (–3.6 to 3.4)
Motor function (0–66)
Baseline	255	18.0 (13.1)	259	18.2 (14.1)	254	18.2 (13.9)
3 months	232	26.2 (17.7)	234	27.1 (18.3)	202	24.2 (18.4)	2.0 (–2.2 to 6.2)	2.8 (0.7 to 5.0)	2.9 (–1.3 to 7.1)	3.0 (0.9 to 5.0)	–0.9 (–4.8 to 3.1)	–0.2 (–2.2 to 1.9)
6 months	221	27.2 (18.4)	217	28.2 (19.3)	186	26.3 (18.9)	0.9 (–3.6 to 5.3)	2.5 (0.1 to 5.1)	1.9 (–2.7 to 6.4)	2.4 (–0.2 to 4.8)	–1.0 (–5.3 to 3.3)	0.1 (–2.3 to 2.7)
Range of motion and joint pain (0–48)
Baseline	254	41.5 (6.1)	259	41.5 (6.2)	254	41.0 (6.6)
3 months	232	40.8 (6.8)	234	41.2 (6.2)	204	40.0 (8.0)
6 months	221	41.5 (6.1)	218	41.6 (5.9)	186	41.6 (6.2)
Sensation (0–12)
Baseline	253	9.4 (3.2)	258	9.4 (3.3)	252	9.8 (2.9)
3 months	231	9.6 (3.1)	234	9.5 (3.0)	202	9.9 (2.9)
6 months	221	9.5 (3.0)	218	9.6 (2.9)	186	10.0 (2.7)

Fugl-Meyer Assessment motor function subscale

This pattern of improvement was also seen for the FMA motor function subscale. For the robot-assisted training group, the mean scores were 18.0 points at baseline, 26.2 points at 3 months and 27.2 points at 6 months. In comparison, for the usual care group the mean scores were 18.2 points at baseline, 24.2 points at 3 months and 26.3 points at 6 months. The robot-assisted training group had a higher average FMA motor function subscale score than the usual care group at 3 months (aMD 2.8, 98.33% CI 0.7 to 5.0), and this difference was maintained at 6 months (aMD 2.5, 98.33% CI 0.1 to 5.1) (Figure 6; see also Table 5). The MCID for this subscale is 4 points for acute stroke patients and 5.25 points for chronic stroke patients, which makes it difficult to interpret the results from a mixed patient group. 59 Further details and descriptive statistics for the ‘range of motion and joint pain’ and ‘sensation’ subscales are given in Table 5.

FIGURE 6.

The FMA motor function subscale score at baseline and at 3 and 6 months for robot-assisted training, EULT and usual care. (a) The FMA motor function subscale score by randomisation group; and (b) pairwise comparisons of the FMA motor function subscale score presented as mean differences (98.33% CIs). RT, robot-assisted training; UC, usual care.

Barthel Activities of Daily Living Index

There was little change in the Barthel ADL Index score over time. In the robot-assisted training group, the mean was 14.5, 15.5 and 15.6 points at baseline, 3 months and 6 months, respectively. This was similar to the usual care group, in which the mean was 14.4, 15.3 and 15.3 points at baseline, 3 months and 6 months, respectively. There was little evidence of a difference between these groups on this scale at 3 months (aMD 0.2, 98.33% CI –0.4 to 0.8) or at 6 months (aMD 0.5, 98.33% CI –0.1 to 1.1) (Table 6). The MCID on this scale is 1.85 units,59 and so the changes in the mean Barthel ADL Index scores between baseline, 3 months and 6 months were not large enough to be consistent with a clinically meaningful difference on this scale for both groups (i.e. robot-assisted training and usual care), nor were the comparisons between group means at 3 or 6 months.

TABLE 6 - Comparison of the ADL (Barthel ADL Index scores) at 3 and 6 months between randomisation groups

For time since stroke (to the power of –1), baseline Barthel ADL Index score and trial centre.

Time point	Robot-assisted training		EULT		Usual care		Mean difference (98.33% CI)
							Robot-assisted training – usual care		EULT – usual care		Robot-assisted training – EULT
	n	Mean (SD)	n	Mean (SD)	n	Mean (SD)	Unadjusted	Adjusteda	Unadjusted	Adjusteda	Unadjusted	Adjusteda
Barthel ADL Index (0–20)
Baseline	255	14.5 (3.8)	259	14.3 (4.0)	254	14.4 (3.9)
3 months	233	15.5 (3.4)	236	15.9 (3.4)	207	15.3 (3.8)	0.2 (–0.6 to 1.0)	0.2 (–0.4 to 0.8)	0.6 (–0.2 to 1.5)	0.7 (0.2 to 1.2)	–0.5 (–1.2 to 0.3)	–0.5 (–1.0 to –0.0)
6 months	223	15.6 (3.4)	222	16.0 (3.5)	190	15.3 (3.7)	0.3 (–0.5 to 1.1)	0.5 (–0.1 to 1.1)	0.6 (–0.2 to 1.5)	0.9 (0.3 to 1.5)	–0.4 (–1.2 to 0.4)	–0.4 (–1.0 to 0.1)

Stroke Impact Scale

There was little change over time on any SIS subscale. The mean scores on the ADL subscale were 50.8 points at 3 months and 50.4 points at 6 months in the robot-assisted training group, but they were 50.8 and 51.9 points at 3 and 6 months, respectively, in the usual care group. The results were consistent with the Barthel ADL Index, in that they showed little evidence of a difference between these groups at 3 months (aMD 0.7, 98.33% CI –4.2 to 5.8) or at 6 months (aMD –0.6, 98.33% CI –5.6 to 4.5) (Table 7). The MCID on this SIS subscale is 5.9 units,59 so the differences between groups were not consistent with a clinically meaningful difference.

TABLE 7 - Comparison of the quality of life (SIS scores) at 3 and 6 months between randomisation groups

For transformed ‘time since stroke’ and trial centre. Time since stroke was to the power of –0.5 for all subscales except ADL and social participation, where –1 and 3 were used, respectively.

This number is positive.

SIS subscale and time point	Robot-assisted training			EULT			Usual care			Mean difference (98.33% CI)
										Robot-assisted training – usual care		EULT – usual care		Robot-assisted training – EULT
	n	Mean (SD)	Median (IQR)	n	Mean (SD)	Median (IQR)	n	Mean (SD)	Median (IQR)	Unadjusted	Adjusteda	Unadjusted	Adjusteda	Unadjusted	Adjusteda
ADL (0–100)
3 months	220	50.8 (22.5)	48 (33–70)	223	55.9 (19.8)	55 (40–70)	194	50.8 (21.1)	50 (33–65)	0.0b (–5.1 to 5.2)	0.7 (–4.2 to 5.8)	5.1 (0.3 to 9.9)	5.6 (0.9 to 10.2)	–5.1 (–10.0 to –0.3)	–4.8 (–9.5 to –0.1)
6 months	212	50.4 (22.3)	50 (33–68)	216	52.5 (22.3)	50 (38–68)	179	51.9 (21.3)	53 (35–70)	–1.4 (–6.7 to 3.9)	–0.6 (–5.6 to 4.5)	0.6 (–4.7 to 5.8)	1.1 (–3.9 to 6.1)	–2.0 (–7.2 to 3.1)	–1.7 (–6.7 to 3.3)
Hand function (0–100)
3 months	219	15.5 (24.4)	0 (0–20)	223	21.4 (27.9)	5 (0–40)	193	14.3 (22.9)	0 (0–25)	1.3 (–4.3 to 6.8)	2.6 (–2.6 to 7.9)	7.2 (1.1 to 13.1)	7.9 (2.2 to 13.5)	–5.9 (–11.9 to 0.1)	–5.4 (–10.9 to 0.2)
6 months	213	15.7 (25.2)	0 (0–25)	216	18.4 (26.2)	0 (0–35)	179	14.8 (23.5)	0 (0–25)	0.9 (–5.0 to 6.8)	1.9 (–3.6 to 7.5)	3.6 (–2.4 to 9.5)	3.9 (–1.8 to 9.6)	–2.7 (–8.6 to 3.3)	–2.0 (–7.4 to 3.6)
Mobility (0–100)
3 months	219	61.6 (25.1)	64 (44–83)	223	66.4 (23.2)	69 (50–86)	192	61.0 (24.6)	61 (46–81)	0.6 (–5.3 to 6.5)	1.3 (–4.3 to 6.9)	5.4 (–0.1 to 11.1)	5.8 (0.4 to 11.2)	–4.8 (–10.4 to 0.7)	–4.5 (–9.9 to 0.9)
6 months	213	61.7 (24.8)	64 (44–83)	216	63.9 (23.7)	68 (47–83)	180	62.9 (23.8)	65 (44–83)	–1.2 (–7.1 to 4.7)	–0.5 (–6.0 to 5.1)	1.1 (–4.6 to 6.8)	1.4 (–4.1 to 6.8)	–2.3 (–7.9 to 3.3)	–1.9 (–7.3 to 3.5)
Social participation (0–100)
3 months	217	47.7 (24.7)	47 (29–66)	221	51.7 (23.0)	53 (38–69)	193	47.1 (23.7)	47 (31–63)	0.6 (–5.1 to 6.3)	0.5 (–5.2 to 6.2)	4.6 (–0.9 to 10.1)	4.7 (–0.7 to 10.1)	–4.0 (–9.5 to 1.5)	–4.2 (–9.6 to 1.3)
6 months	210	47.0 (25.9)	44 (25–66)	216	50.2 (24.4)	50 (31–69)	179	49.2 (23.8)	50 (31–63)	–2.2 (–8.3 to 3.9)	–2.2 (–8.2 to 3.9)	1.0 (–4.8 to 6.9)	1.2 (–4.7 to 7.0)	–3.3 (–9.1 to 2.6)	–3.4 (–9.2 to 2.4)
Strength (0–100)
3 months	220	43.8 (20.4)	44 (31–56)	220	48.1 (22.2)	50 (31–63)	193	40.1 (19.1)	38 (25–50)
6 months	213	40.9 (21.7)	38 (25–56)	216	44.3 (22.7)	44 (25–63)	176	40.3 (21.6)	38 (25–56)
Emotion (0–100)
3 months	220	68.3 (18.2)	69 (56–82)	222	69.9 (16.5)	69 (58–83)	194	67.4 (20.5)	67 (53–86)
6 months	211	66.9 (18.3)	67 (53–81)	216	68.3 (18.8)	69 (56–83)	179	67.4 (19.7)	69 (56–83)
Memory (0–100)
3 months	219	76.0 (22.4)	82 (61–96)	222	80.3 (18.8)	86 (68–96)	194	76.8 (22.2)	82 (61–96)
6 months	213	74.6 (22.7)	79 (58–93)	215	77.4 (21.2)	82 (64–96)	178	77.8 (23.4)	86 (64–96)
Communication (0–100)
3 months	220	80.1 (23.4)	89 (68–100)	222	85.3 (20.5)	96 (79–100)	194	80.9 (24.3)	93 (68–100)
6 months	213	79.3 (21.7)	86 (68–100)	216	83.7 (22.7)	96 (75–100)	179	82.2 (22.5)	93 (71–100)
Stroke recovery (0–100)
3 months	217	47.9 (18.9)	50 (30–60)	223	50.3 (18.9)	50 (40–60)	190	45.2 (19.9)	50 (30–60)
6 months	213	51.4 (20.4)	50 (35–70)	215	50.9 (21.5)	50 (35–70)	180	48.5 (20.3)	50 (33–60)

The mean scores on the SIS hand function subscale were 15.5 points at 3 months and 15.7 points at 6 months in the robot-assisted training group, but were 14.3 and 14.8 points at 3 and 6 months, respectively, in the usual care group. There was also little evidence of a difference in hand function between groups at 3 months (aMD 2.6, 98.33% CI –2.6 to 7.9) or 6 months (aMD 1.9, 98.33% CI –3.6 to 7.5). The MCID on this SIS subscale is 17.8 units,59 so the results were not consistent with a clinically meaningful difference.

The mean scores on the SIS mobility subscale were 61.6 points at 3 months and 61.7 points at 6 months in the robot-assisted training group, but were 61.0 and 62.9 points at 3 and 6 months, respectively, in the usual care group. As before, there was little evidence of a difference between groups for mobility at 3 months (aMD 1.3, 98.33% CI –4.3 to 6.9) or 6 months (aMD –0.5, 98.33% CI –6.0 to 5.1). However, the MCID on this SIS subscale is 4.5 units,59 which was included in these CIs, so a clinically meaningful difference between groups cannot be ruled out.

The mean scores on the SIS social participation subscale were 47.7 points at 3 months and 47.0 points at 6 months in the robot-assisted training group, but were 47.1 and 49.2 points at 3 and 6 months, respectively, in the usual care group. There was also little evidence of a difference at 3 months (aMD 0.5, 98.33% CI –5.2 to 6.2) or 6 months (aMD –2.2, 98.33% CI –8.2 to 3.9) between the robot-assisted training group and the usual care group, as measured by the SIS. More details and descriptive statistics for strength, emotion, memory, communication and stroke recovery are given in Table 7.

Pre-planned subgroup and exploratory analyses

Exploratory subgroup analysis

There was little evidence of a difference in the mean ARAT total score between pairs of randomisation groups when looking at the subgroups of participants for time since stroke, baseline ARAT score, trial centre and age (Table 9 and Figure 7). This was also the case for the FMA motor function subscale score (Table 10 and Figure 8a–c) and the Barthel ADL Index score (Table 11 and Figure 8d–f), which were examined by the time since stroke categories. Note that, by splitting the data across subgroups, the sample size is reduced for any comparison, so the CIs are wide. The CI includes results that are often consistent with a clinically meaningful difference (as per the MCID) on the scales, sometimes in both directions.

TABLE 9 - Exploratory subgroup analysis of the total ARAT score (0–57)a

Comparison of the mean ARAT score at 3 months between trial randomisation groups and subgroups (time from stroke to randomisation, baseline ARAT score, trial centre and age).

This number is positive.

Subgroup	Robot-assisted training			EULT			Usual care			Unadjusted mean difference (98.33% CI)
	n	Mean (SD)	Median (IQR)	n	Mean (SD)	Median (IQR)	n	Mean (SD)	Median (IQR)	Robot-assisted training – usual care	EULT – usual care	Robot-assisted training – EULT
Time since stroke
< 3 months	47	25.6 (22.2)	23 (3–48)	39	34.2 (23.9)	46 (5–57)	50	22.4 (24.3)	7 (0–48)	3.1 (–8.2 to 14.4)	11.8 (–0.9 to 23.5)	–8.7 (–20.2 to 3.6)
3–12 months	94	16.2 (19.8)	4 (0–29)	110	16.5 (19.1)	5 (3–30)	77	13.4 (18.5)	3 (0–21)	2.8 (–4.4 to 9.7)	3.1 (–3.9 to 9.4)	–0.3 (–6.6 to 6.3)
> 12 months	91	9.5 (13.6)	3 (0–11)	85	10.5 (14.6)	3 (0–15)	76	9.6 (15.4)	3 (0–8)	–0.2 (–6.0 to 5.0)	0.8 (–5.0 to 6.3)	–1.0 (–6.2 to 4.0)
Baseline ARAT score
0	84	5.5 (12.1)	0 (0–4)	89	5.1 (12.0)	0 (0–3)	87	3.4 (10.6)	0 (0–3)	2.1 (–2.1 to 6.3)	1.7 (–2.3 to 5.9)	0.4 (–4.1 to 4.8)
1–7	77	6.5 (10.3)	3 (1–7)	69	10.3 (15.0)	4 (3–10)	51	7.7 (13.1)	3 (0–6)	–1.1 (–7.3 to 3.5)	2.6 (–3.8 to 8.6)	–3.7 (–9.3 to 1.0)
8–13	16	22.3 (15.6)	19 (11–29)	21	24.1 (14.7)	23 (14–29)	20	17.5 (13.7)	16 (8–22)	4.8 (–6.6 to 17.0)	6.6 (–4.5 to 16.7)	–1.8 (–13.0 to 11.1)
14–19	12	32.8 (14.2)	29 (24–43)	7	38.9 (14.2)	38 (32–49)	11	34.2 (22.0)	33 (14–55)	–1.3 (–18.7 to 18.1)	4.7 (–16.1 to 24.7)	–6.0 (–20.3 to 12.7)
20–39	43	43.6 (11.0)	45 (34–54)	48	43.7 (11.9)	47 (33–54)	34	43.5 (12.3)	46 (36–53)	0.1 (–6.0 to 6.9)	0.2 (–6.1 to 6.9)	–0.1 (–5.7 to 5.7)
Trial centre
1	65	16.2 (19.6)	5 (1–29)	70	18.7 (19.9)	7 (3–33)	57	16.2 (21.5)	4 (0–39)	0.1 (–9.0 to 8.7)	2.5 (–6.6 to 11.1)	–2.4 (–10.5 to 5.7)
2	76	12.3 (18.2)	3 (0–17)	72	13.6 (20.0)	1 (0–25)	68	9.6 (17.0)	0 (0–10)	2.7 (–4.4 to 9.5)	4.0 (–3.6 to 11.5)	–1.3 (–8.9 to 6.1)
3	34	19.7 (22.9)	6 (0–45)	28	26.1 (23.8)	20 (3–54)	30	24.1 (24.0)	18 (0–48)	–4.4 (–18.3 to 9.7)	2.0 (–12.9 to 17.0)	–6.4 (–20.6 to 7.9)
4	57	16.3 (16.9)	6 (3–28)	64	16.0 (17.9)	4 (3–32)	48	12.3 (15.4)	5 (3–16)	4.0 (–3.9 to 11.3)	3.7 (–4.1 to 11.0)	0.2 (–7.3 to 7.9)
Age
< 55 years	81	17.0 (20.8)	5 (1–29)	90	17.0 (19.8)	5 (3–30)	53	12.0 (19.1)	3 (0–14)	5.0 (–3.9 to 12.9)	5.0 (–3.7 to 12.5)	0.0b (–7.3 to 7.5)
55–70 years	97	14.3 (17.9)	5 (0–26)	84	16.6 (20.9)	3 (0–34)	83	11.3 (17.6)	3 (0–15)	3.1 (–3.5 to 9.3)	5.4 (–1.9 to 12.4)	–2.3 (–9.3 to 4.6)
> 70 years	54	15.2 (18.6)	5 (0–28)	60	18.6 (19.9)	9 (2–34)	67	19.7 (21.4)	6 (0–39)	–4.5 (–13.0 to 4.3)	–1.1 (–9.7 to 7.8)	–3.4 (–11.9 to 5.2)

FIGURE 7.

Forest plots for the subgroup analysis of the unadjusted ARAT score by time since stroke, baseline ARAT score, trial centre and age categories, comparing (a) robot-assisted training and usual care; (b) EULT and usual care; and (c) robot-assisted training and EULT. RT, robot-assisted training; UC, usual care.

TABLE 10 - Exploratory subgroup analysis of the mean FMA scorea

Comparison of the mean FMA motor score at 3 months between randomisation groups and subgroups.

Time since stroke	Robot-assisted training			EULT			Usual care			Unadjusted mean difference (98.33% CI)
	n	Mean (SD)	Median (IQR)	n	Mean (SD)	Median (IQR)	n	Mean (SD)	Median (IQR)	Robot-assisted training – usual care	EULT – usual care	Robot-assisted training – EULT
< 3 months	47	36.0 (19.8)	37 (16–55)	39	38.8 (21.0)	42 (21–58)	50	30.3 (22.3)	26 (6–51)	5.7 (–4.6 to 15.9)	8.5 (–2.7 to 19.2)	–2.8 (–13.1 to 7.9)
3–12 months	94	26.4 (17.8)	23 (10–41)	110	26.4 (17.6)	23 (11–42)	76	24.0 (18.4)	17 (9–40)	2.3 (–4.5 to 8.9)	2.4 (–4.2 to 8.6)	–0.0 (–5.9 to 6.0)
> 12 months	91	21.1 (14.2)	19 (8–30)	85	22.7 (15.7)	20 (10–32)	76	20.4 (14.2)	18 (8–29)	0.7 (–4.7 to 5.9)	2.3 (–3.3 to 8.0)	–1.6 (–7.1 to 3.7)

FIGURE 8.

Forest plots for the subgroup analysis of the unadjusted FMA motor function subscale score and Barthel ADL Index by time since stroke categories. (a) FMA, robot-assisted training vs. usual care; (b) FMA, EULT vs. usual care; (c) FMA, robot-assisted training vs. EULT; (d) Barthel ADL Index, robot-assisted training vs. usual care; (e) Barthel ADL Index, EULT vs. usual care; and (f) Barthel ADL Index, robot-assisted training vs. EULT. RT, robot-assisted training; UC, usual care.

TABLE 11 - Exploratory subgroup analysis of the mean Barthel ADL Index scorea

Comparison of the mean Barthel ADL Index score at 3 months between randomisation groups and subgroups.

This number is positive.

Time since stroke	Robot-assisted training			EULT			Usual care			Unadjusted mean difference (98.33% CI)
	n	Mean (SD)	Median (IQR)	n	Mean (SD)	Median (IQR)	n	Mean (SD)	Median (IQR)	Robot-assisted training – usual care	EULT – usual care	Robot-assisted training – EULT
< 3 months	47	15.8 (3.3)	17 (15–18)	39	17.3 (2.8)	18 (15–20)	50	15.7 (4.1)	17 (13–19)	0.1 (–1.7 to 2.0)	1.6 (–0.1 to 3.4)	–1.5 (–3.1 to 0.1)
3–12 months	95	15.6 (3.1)	16 (14–18)	110	15.6 (3.6)	16 (14–18)	80	14.9 (3.9)	16 (13–18)	0.7 (–0.5 to 2.0)	0.7 (–0.6 to 2.0)	0.0b (–1.1 to 1.2)
> 12 months	91	15.2 (3.8)	16 (13–18)	87	15.8 (3.2)	17 (14–18)	77	15.4 (3.5)	16 (13–18)	–0.2 (–1.6 to 1.1)	0.4 (–0.9 to 1.7)	–0.6 (–1.9 to 0.6)

Masking of treatment allocation

At 3 months, the researchers conducting the outcome assessments reported that they became aware of the treatment group allocation for 101 out of 676 (15%) participants with non-missing data (see Appendix 4, Table 35). The rate in the robot-assisted training group, at 21%, was almost twice that in the EULT group (12%) and the usual care group (12%). The researcher was correct in 86 out of 94 (91%) cases; for 58 out of 101 (57%) cases, the researcher reported that the treatment was known before the outcome assessment was completed (see Appendix 4, Table 36). Becoming aware of the treatment group before outcome assessment was more common in the intervention groups than in the usual care group [robot-assisted training group, 34/50 (68%); EULT group, 15/26 (58%); and usual care group, 9/25 (36%)]. The results were similar at 6 months: the researchers thought that they knew the allocation of 88 out of the 633 (14%) assessments that were undertaken, and awareness of the treatment group before the outcome assessment was completed was also similar across the groups [robot-assisted training group, 22/39 (56%); EULT group, 15/25 (60%); and usual care group, 15/24 (63%)] (see Appendix 4, Tables 37 and 38).

Participant safety data

Introduction

This chapter includes a description of the robot-assisted training programme and details of delivery fidelity.

The robot-assisted training programme

Robot-assisted training used a MIT-Manus robotic gym system, which was available at each of the four trial hub sites. The MIT-Manus robotic gym comprises three components: a shoulder–elbow (InMotionARM robot), a wrist (InMotionWRIST robot) and a hand attachment (InMotionHAND robot). The shoulder–elbow and wrist components are standalone robot units, whereas the hand component is an attachment that fits onto the shoulder–elbow unit. The training programme was designed to be progressive and challenging. Over the course of the programme, all participants progressed from exercises that involved gross movements (e.g. shoulder–elbow movement) to exercises that involved more complex movements (e.g. hand movement). A detailed description of the robot-assisted training programme is provided in the following paragraph, and a completed TIDieR checklist34 is provided in Appendix 1, Table 27.

The robot-assisted training programme aimed to provide therapy sessions three times per week for 12 weeks, predominantly in an outpatient setting (36 sessions in total). This was divided into 18 sessions on the shoulder–elbow (± hand attachment) component and 18 sessions on the wrist robot component. A session lasted for up to 1 hour, which included 45 minutes of face-to-face therapy for each individual participant (the aim was to provide 27 hours of face-to-face therapy in total). Transport was arranged by the local trial co-ordinator if required. A senior therapist (physiotherapist or occupational therapist) undertook the initial training session on each robot unit (shoulder–elbow and wrist), and at the session in which the hand attachment was introduced. The remaining therapy sessions were designed to be delivered by a therapy assistant.

The 12-week robot-assisted training programme was divided into three consecutive blocks (Figure 9).

FIGURE 9.

Summary of the robot-assisted training programme.

Block 1 lasted for 2 weeks; the shoulder–elbow and wrist robot components were used on alternate training sessions for three sessions each. Participants used the robot’s ‘playback protocol’, whereby the arm (placed in the robot arm support) was moved rhythmically towards the flashing red targets that appear sequentially on a clock face on the robot computer screen (Figure 10). This block allowed participants to become gradually accustomed to robot-assisted training.

FIGURE 10.

Robot-assisted training clock face exercise. The small white circle is the cursor the patient attempts to move; the large white circle is the target.

Block 2 lasted for 6 weeks; the shoulder–elbow and wrist robot components were used on alternate training sessions for nine sessions each. Participants used the ‘adaptive protocol’, whereby the arm (placed in the robot arm support) was ‘assisted as needed’ to reach the flashing red targets appearing sequentially on the clock face.

For the final 4 weeks (block 3), the shoulder–elbow and wrist robot units were used on alternate training sessions for six sessions each. In this block, if a participant was physically able to grasp the hand attachment, then the hand attachment was introduced. If a participant was physically unable to grasp the hand attachment, they continued the programme on the shoulder–elbow robot and did not use the hand attachment. Participants who used the hand attachment in this block used the ‘adaptive protocol’, whereby the arm is ‘assisted as needed’ by a robot to complete the exercises. On the wrist robot, and if a participant was unable to use the hand attachment on the shoulder–elbow robot, the ‘random protocol’ was used. The ‘random protocol’ was ‘assisted as needed’ as participants attempt to move towards flashing red targets on the clock face, but this time targets were presented in random order.

In addition to the robot training exercises described above, the robot-assisted training programme also involved regular ‘robot evaluations’ whereby kinematics (i.e. related to the movement pattern) and kinetics (i.e. related to the causes of movement) were measured by the robot. These were completed every 2 weeks.

The robot-assisted training programme was prescriptive, in that every session had a defined number and type of exercises that a participant should complete. Each exercise comprised a prespecified number of movement attempts. Movement attempts were ‘assisted’ for playback, adaptive and random protocols and ‘unassisted’ for robot evaluations.

The total number of potential movement attempts differed per session, as this depended on the robot (shoulder–elbow robot alone, shoulder–elbow robot with hand attachment, or wrist robot) and the training block. If a participant was able to use the hand attachment, the total number of possible shoulder–elbow robot movement attempts over the 18 sessions was 20,060 (a mean of 1114 movement attempts per session). For those participants who were unable to use the hand attachment, the number of possible movement attempts was 19,496 (a mean of 1083 movement attempts per session). There was no variation in training for the wrist robot; therefore, for each participant, the total number possible of wrist robot movement attempts over 18 sessions was 19,196 (a mean of 1066 movement attempts per session). Overall, the number of possible movement attempts for the robot-assisted training programme was 38,692 for those participants unable to use the hand attachment and 39,256 for those able to use the hand attachment (a mean 1075 and 1090 movement attempts per session, respectively).

All therapy movement attempts were ‘assisted’. The total possible number of ‘assisted’ movement attempts for each participant on each robot was 17,280 over the 18 sessions (a total of 34,560 over the 36 sessions and a mean of 960 movement attempts per session).

Robot evaluations were ‘unassisted’. The total possible number of ‘unassisted’ robot movement attempts for each participant was 2216 (for those unable to use the hand attachment) or 2780 (for those able to use the hand attachment) for the shoulder–elbow robot over the 18 sessions. The total possible number of ‘unassisted’ movement attempts for each participant on the wrist robot was 1916 over the 18 sessions. Over the 36 sessions, this equated to a total target of 4132 (for those unable to use the hand attachment) or 4696 (for those able to use the hand attachment) ‘unassisted’ movement attempts.

Training was provided to senior therapists and therapy assistants who delivered the robot-assisted training programme with updates throughout the trial. Six manuals provided guidance on intervention delivery. 60 The first described the purpose, principles and structure of the robot-assisted training programme, as well as staff roles and responsibilities; the second provided instructions on how to operate the MIT-Manus robotic gym system. The remaining manuals provided guidance on how to conduct the robot-assisted training programme on each robot component, with accompanying step-by-step flow charts to guide trial staff.

Recording the robot-assisted training programme

Details from all sessions were recorded on the trial robots. This included the number of times a participant used a trial robot, a participant’s duration on the robot and the number and type of exercises practised. The anonymised data from the trial robots were uploaded to the Newcastle University server. Data about non-attendance were recorded on the trial database.

Analysis of robot data

Descriptive analyses of numeric data were undertaken and presented as mean (SD) or median (IQR), as appropriate. Categorical data are presented as n (%).

Fidelity of the robot-assisted training programme

A total of 257 participants were randomised to receive robot-assisted training. Two participants withdrew from the trial before receiving any robot-assisted training, as they felt that it was too much of a burden. Data for the remaining 255 (99%) participants were available.

A total of 255 out of 257 (99%) participants received at least one robot-assisted training session, with 254 out of 257 (99%) of participants receiving one or more shoulder–elbow robot-assisted training sessions and 243 out of 257 (95%) participants receiving one or more wrist robot-assisted training sessions. Of the 254 participants who received training on the shoulder–elbow robot, 156 (61%) received training using the hand attachment.

Overall, 8026 out of a possible 9252 sessions (87%) were attended by participants. The number of sessions attended for the wrist robot was lower [3812/4626 (82%)] than the number of shoulder–elbow sessions attended [4214/4626 (91%)]. The median number of robot-assisted training sessions received per participant was 35 (IQR 31–36 sessions), with a median of 18 sessions (IQR 16–18 session) on the shoulder–elbow robot and 18 sessions (IQR 14–18 sessions) on the wrist robot. The most common reason for lack of attendance at sessions was because participants withdrew from the trial [429/1226 (35%) sessions that were not attended]. Details of reasons for all missed sessions can be found in Appendix 6, Table 45.

The median total duration of training (time on robot) per participant over the 12-week intervention period was 23 hours and 30 minutes (IQR 19 hours and 13 minutes to 25 hours and 47 minutes). The median total duration of training per participant was similar on both robots [11 hours and 40 minutes (IQR 9 hours and 54 minutes to 12 hours and 54 minutes) on the shoulder–elbow robot and 11 hours 56 minutes (IQR 9 hours and 35 minutes to 13 hours and 14 minutes) on the wrist].

The median duration of face-to-face training in each attended session was 41 minutes (IQR 35–47 minutes). Similarly, the median duration of face-to-face training in each attended session was similar for both robots [40 minutes (IQR 34–46 minutes) on the shoulder–elbow robot and 42 minutes (IQR 36–48 minutes) on the wrist robot].

In terms of movement attempts, participants achieved 15,002 out of 20,060 (75%) movement attempts on the shoulder–elbow robot with the hand attachment, 11,141 out of 19,496 (57%) on the shoulder–elbow robot without the hand attachment and 13,460 out of 19,196 (70%) on the wrist robot. Over the entire robot-assisted training programme, the median total number of movement attempts per participant was 27,407 (IQR 22,900–32,187 movement attempts). There was a median of 808 movement attempts (IQR 668–1040 movement attempts) per participant, per attended session. The numbers of movement attempts achieved for the shoulder–elbow robot (with and without the hand attachment), the wrist robot and overall are shown in Appendix 6, Table 46.

Movement attempts by baseline Action Research Arm Test score and time since stroke

The impact of baseline ARAT score and time since stroke on the number of assisted, unassisted and total movement attempts was investigated. The predefined categories of baseline ARAT score of 0–7 and 20–39 were used, but, owing to the small numbers of participants randomised with baseline ARAT scores in the ranges of 8–13 or 14–19, these two categories were combined. For time since stroke, the pre-defined categories of < 3 months, 3–12 months and > 12 months were used.

Participants with the low baseline ARAT scores (0–7) achieved the lowest median total number of movement attempts [25,368 (IQR 21,662–29,153)]. Participants with baseline ARAT score in the ranges of 8–19 and 20–39 achieved median total movement attempts of 32,187 (IQR 27,745–36,624) and 33,358 (IQR 29,704–38,348), respectively.

Time since stroke did not have an impact on the number of movement attempts achieved, with all subgroups achieving a similar number. Participants for whom it was < 3 months since their stroke achieved a median of 28,359 movement attempts (IQR 18,611–35,070 movement attempts), compared with a median of 26,706 movement attempts (IQR 23,319–32,163 movement attempts) for those for whom it was 3–12 months since their stroke, and a median of 27,412 movement attempts (IQR 23,660–30,061 movement attempts) for participants for whom it was > 12 months since their stroke. Appendix 6, Table 47, shows the number of assisted, unassisted and overall movement attempts by baseline ARAT score and time since stroke for the shoulder–elbow and wrist robots.

Descriptive analysis of the relationship between treatment received and total Action Research Arm Test score

The total ARAT score at 3 and 6 months was plotted against the number of sessions attended, total duration of therapy received and total robot movement attempts. For participants who did not attend an appointment, zero duration and zero movement attempts were assumed. There is little evidence of an association between total ARAT score and amount of treatment received (Figure 11).

FIGURE 11.

Scatterplots for the robot-assisted training group for ARAT total score at 3 and 6 months. (a) Number of sessions attended vs. ARAT score at 3 months; (b) total time on robot vs. ARAT score at 3 months; (c) total robot movement attempts vs. ARAT score at 3 months; (d) number of sessions attended vs. ARAT score at 6 months; (e) total time on robot vs. ARAT score at 6 months; and (f) total robot movement attempts vs. ARAT score at 6 months.

Introduction

This chapter includes a description of the EULT programme and details of delivery fidelity, dose, goals selected and goal achievement.

The enhanced upper limb therapy programme

The EULT programme consisted of repetitive functional task practice, directed towards achieving participant-centred goals. The aim of the EULT programme was to promote long-term improvement in arm function that was useful to individual participants by focusing on their personal goals, through an intervention that was engaging and challenging, yet achievable. The intervention structure and principles were standardised, while its application allowed tailoring to individuals’ goals and abilities. A detailed description of the EULT programme is provided in this chapter, and a completed TIDieR checklist34 is provided in Appendix 1, Table 27.

To ensure that this comparator was matched to the robot-assisted training in terms of frequency and duration, the EULT programme aimed to provide therapy sessions three times per week for 12 weeks, predominantly in an outpatient setting (36 sessions in total). A session lasted for up to 1 hour, which included 45 minutes of face-to-face therapy for each individual participant (the aim was 27 hours of face-to-face therapy in total). Transport was arranged by the local trial co-ordinator if required. A senior therapist (physiotherapist or occupational therapist) undertook the initial therapy session and reviewed the participant every 4 weeks. The remaining therapy sessions were designed to be delivered by a therapy assistant.

At the start of their first session, participants were invited to identify personally relevant goals; these were not prespecified, other than that they should involve a functional task involving the affected arm. Participants were advised to select no more than four goals at each review session (up to 12 across the EULT programme), based on previous work. 62 Goals from previous arm rehabilitation studies were prepared as examples to facilitate goal selection. 35–38 The goal selection process was not formalised, but was undertaken according to each senior therapist’s clinical judgement. In each session, participants practised functional activities to work towards their goals. Activities could involve whole- or part-task practice, at the discretion of the senior therapist. Whole-task activity practice consisted of practising all of the components of the task in sequence. Part-task activity practice consisted of practising a specific part of a task. Part-task practice was undertaken when a participant had difficulty with a specific part of a task, as it enabled them to concentrate on this particular aspect while working towards completing the task as a whole. The order in which activities were practised, and the time spent on each, were at the discretion of the therapist and participant. Participants were encouraged to undertake as many repetitions as possible in each session. There was no set target for the number of repetitions.

At the start of each session, participants undertook a brief ‘warm-up’, consisting of focusing attention on their affected arm, gently stretching soft tissues and mobilising joints to stimulate sensation and proprioception, before commencing practice, which focused on an individual’s goals.

At therapy sessions 12 (end of week 4) and 24 (end of week 8), progress towards goals was reviewed with a senior therapist. If a participant had achieved a goal, a new goal and a new activity to practise were selected. If a participant found a goal or activity too challenging, or experienced other problems, an alternative goal and activity were chosen. At the final therapy session, practice continued and progress towards goals was reviewed with a senior therapist. Part of the session was also dedicated to providing feedback to the participant about progress over the course of the programme and advice about maintaining arm function in the longer term.

Training was provided to the senior therapists and therapy assistants who delivered the EULT programme, with updates throughout the trial. In addition, three manuals provided guidance on intervention delivery. 60 The first described the purpose, principles and structure of the EULT programme, as well as staff roles and responsibilities; the second provided guidance on how to structure each session (including assessment, warm-up and stretching, demonstration and education, progression, monitoring compensatory movements and feedback). The third manual provided examples of soft tissue stretches prior to task practice, and an overview of commonly selected goals, with accompanying step-by-step flow charts to guide and progress practice of functional tasks, each with whole-task and part-task options.

Recording the enhanced upper limb therapy programme

The following were recorded: session attendance, duration of session, duration of face-to-face therapy during a session, number of repetitions per task practised in each session, goals selected and type of task practice (whole or part task), and goal achievement. In terms of task repetitions, for whole-task practice, completion of the whole task counted as one task repetition, that is from the start position to a ‘return to the start position’ or to completion of the task (if different from the start position). For part-task practice, completion of the component of the part task counted as one task repetition.

Details about the goals selected were recorded at the initial therapy session and at the 4- and 8-week review sessions by the senior therapists. Information about whether or not a participant had achieved their goals (recorded as ‘yes’/‘no’, according to the senior therapist’s clinical judgement) was recorded at the 4-, 8- and 12-week review sessions. A formal goal attainment scale was not used.

Analysis of enhanced upper limb therapy data

Descriptive analyses of numeric data were undertaken and are presented as mean (SD) or median (IQR), as appropriate. Categorical data are presented as n (%).

Recorded goal descriptions were reviewed by a research physiotherapist and retrospectively coded into the following categories, as defined by the Canadian Occupational Performance Measure (COPM):63 self-care, productivity and leisure, and their respective subcategories and activity categories. An additional ‘other’ category was developed for when the described goal did not fit into one of the three COPM categories. The ‘other’ subcategories were coded as generic pick up/grasp/reach/place object; range of movement; upper limb strengthening; weight-bearing; and, when there was insufficient information to code a goal, unclassified.

Fidelity and dose of the enhanced upper limb therapy programme

A total of 259 RATULS trial participants were randomised to receive the EULT programme across the four trial centres. Data for all 259 participants were uploaded to the trial database.

Overall, 7877 of the 9324 (84%) total possible sessions were attended. A median of 34 sessions (IQR 29–36 sessions), out of the possible 36, were attended, per participant. The most common reason for lack of attendance at sessions was being unwell [406/1447 (28%) sessions that were not attended]. Reasons for all missed sessions can be found in Appendix 6, Table 45. Participants attended 941 out of 1036 (91%) possible senior therapist review sessions, with a median of four reviews (IQR 4–4 reviews) per participant. The overall median time spent at therapy sessions by each participant was 30 hours and 32 minutes (IQR 24 hours and 42 minutes to 34 hours and 5 minutes). The overall median duration of face-to-face therapy at the sessions was 24 hours and 40 minutes (IQR 20 hours and 24 minutes to 26 hours and 15 minutes) per participant. The median duration of each session was 60 minutes (IQR 45–60 minutes). The median duration of face-to-face therapy in each session was 45 minutes (IQR 45–45 minutes).

To further describe dose, the number of task repetitions was recorded. A median of 127 (IQR 70–190) task repetitions were achieved per participant per session attended. Overall, a median of 4121 (IQR 2395–5727) task repetitions per participant were achieved across the EULT programme.

Goal selection and goal achievement

A median of 12 goals (IQR 9–12 goals) were selected per participant during the 12-week EULT programme; the overall total was 2664 goals. Table 13 shows the goal choices and goal achievement in the EULT programme. Further detail about the activities practised is given in Appendix 6, Table 48. The most popular category of goal choice was ‘self-care’ [1449/2664 (54%)], followed by the ‘other’ category [661/2664 (25%)], ‘productivity’ [374/2664 (14%)] and ‘leisure’ [180/2664 (7%)]. In the ‘self-care’ category, the subcategory of ‘personal care’ was the most frequently selected, with 1283 out of 1449 (89%) goals. Of the 1283 personal care goals, the majority [622 (48%)] related to eating whereas 254 out of 1283 (20%) related to dressing. All 374 goals in the productivity category related to household management, with ‘cleaning’ and ‘cooking’ being chosen for 206 (55%) goals and 125 (33%) goals, respectively. In the leisure category, the most popular subcategory was ‘socialisation’ [93/180 (52%)], with the activity of ‘correspondence’ being chosen for 92 out of 93 (99%) of socialisation goals.

Despite the specification that goals should involve a functional task using the arm, the ‘other’ category predominantly comprised impairment-based upper limb goals, with ‘range of movement’ being the most commonly chosen category [413/661 (62%)]. Range of movement was the second most commonly chosen goal overall. A total of 163 out of 661 (25%) goals in the ‘other’ category comprised generic reach, grasp, pick-and-place activities; although these were functional, they were not specified sufficiently to be allocated to any particular COPM category. There was insufficient information to subcategorise 26 out of 661 (4%) of the ‘other’ goals.

In terms of type of practice, participants predominantly used whole-task practice to work towards their goals [2213/2664 (83%) whole task, 451/2664 (17%) part task].

Of the 2664 goals selected, goal achievement data were recorded for 2501 (94%) goals. In total, 1287 out of 2501 (51%) goals were achieved, ranging between 47% and 100% for each COPM subcategory (see Table 13). A median of 5 goals (IQR 2–7 goals) were achieved per participant. Of the three most commonly chosen goals, 234 out of 587 (40%) related to eating were achieved, 196 out of 390 (50%) related to range of movement were achieved and 141 out of 242 (58%) related to dressing were achieved.

Goal selection and goal achievement by baseline Action Research Arm Test score and time since stroke

The impact of baseline ARAT score and time since stroke on goal selection and goal achievement was investigated. The predefined categories of baseline ARAT scores of 0–7 and 20–39 were used; however, because of the small numbers of participants randomised with baseline ARAT scores in the ranges of 8–13 and 14–19, these two subgroups were combined. For time since stroke, the predefined categories of < 3 months, 3–12 months and > 12 months were used.

For all subgroups, the most commonly selected goal was self-care; however, goal achievement varied according to baseline ARAT score and time since stroke (see Appendix 6, Table 49). Participants who had the lowest baseline ARAT score (0–7) had the lowest goal achievement, with 684 out of 1637 (42%) goals being achieved. For participants with a low baseline ARAT score, goals that related to personal care, range of movement and functional mobility had low achievement rates. Participants who had ARAT scores in the ranges of 8–19 or 20–39 had goal achievements of 201 out of 318 (63%) and 402 out of 546 (74%) goals, respectively. In terms of time since stroke, participants who were recruited > 1 year after stroke had the lowest goal achievement, with 336 out of 882 (38%) of goals being achieved. Those participants who had the lowest baseline ARAT score (0–7) and who were recruited > 1 year after stroke had the lowest goal achievement, with only 167 out of 616 (27%) goals being achieved. Participants randomised within between 3 and 12 months of their stroke achieved 616 out of 1176 (52%) goals, whereas those recruited < 3 months after stroke had the highest goal achievement, with 335 out of 443 (76%) goals being achieved.

Descriptive analysis of the relationship between treatment received and total Action Research Arm Test score

The total ARAT score at 3 and 6 months was plotted against the number of sessions attended, total duration of therapy received and total number of therapy repetitions. For participants who did not attend an appointment, zero duration and zero therapy repetitions were assumed. There is little evidence of an association between total ARAT score and amount of treatment received (Figure 12).

FIGURE 12.

Scatterplots for the EULT group for ARAT total score at 3 and 6 months. (a) ARAT score at 3 months vs. number of sessions attended; (b) ARAT score at 3 months vs. total session duration; (c) ARAT score at 3 months vs. total therapy duration; (d) ARAT score at 3 months vs. total number of repetitions; (e) ARAT score at 6 months vs. number of sessions attended; (f) ARAT score at 6 months vs. total session duration; (g) ARAT score at 6 months vs. total therapy duration; (h) ARAT score at 6 months vs. total number of repetitions.

Introduction

It was intended that all randomisation groups should continue to receive usual post-stroke care during participation in the trial. Trial interventions (robot-assisted training and the EULT programme) were delivered in addition to usual care. Trial involvement and/or randomisation group should not have had an effect on the usual care received. This chapter provides a description of the usual post-stroke care that trial participants reported that they received for their arm or hand, during their time in the trial.

What is usual care in the NHS?

The current NICE quality standard39 states that stroke patients in hospital or in the community should receive a minimum of 45 minutes of therapy, 5 days per week, and that this therapy should continue in the longer term, depending on a patient’s need. However, it is widely accepted that this is not met, especially once patients are discharged from hospital or early supported discharge services. 40 Furthermore, this guidance relates to all therapy; it does not distinguish upper limb rehabilitation from other aspects of rehabilitation. There is no specific standard about the amount or content of upper limb rehabilitation that should be provided post stroke.

Recording information about usual care arm therapy

All RATULS trial participants were given trial-specific arm rehabilitation therapy logs, and were asked to record any upper limb rehabilitation, other than the robot-assisted training or the EULT programme, that they received during the course of the trial. Two arm rehabilitation therapy logs were given to each participant: the log for trial weeks 1–12 (the intervention period) was given to participants at the baseline assessment and the log for trial weeks 13–24 (the follow-up period) was given to participants at the 3-month outcome assessment. The logs were collected from participants at the 3- and 6-month outcome assessment visits.

The bespoke self-completion logs were intended to be completed weekly. If a participant received upper limb therapy that was not a trial intervention, they were asked to record the frequency, duration and the type of therapy received. For type of therapy, participants, with help from their therapists, were asked to select from the following categories: passive stretching, range of motion, functional strengthening, weight-bearing through hand, trunk control, repetitive task-specific practice, personal ADL, extended ADL or other. Participants were also asked to record whether or not they undertook any self-practice upper limb exercises.

Usual care received by each randomisation group

Usual care arm therapy according to baseline arm function and time since stroke

In clinical practice, the intensity of therapy usually decreases as time since stroke increases, and varies according to severity of the upper limb neurological deficit. Therefore, to describe the therapy received by participants in the usual care arm, the arm rehabilitation log data were analysed according to participants’ baseline ARAT scores and time since stroke. The prespecified categories of baseline ARAT score of 0–7 and 20–39 were used; however, because of the small numbers of participants randomised with baseline ARAT scores in the ranges 8–13 and 14–19, these two categories were combined. For time since stroke, the prespecified categories of < 3 months, 3–12 months and > 12 months were used.

Introduction

Process evaluation can contribute to the interpretation of the results of a RCT of a complex intervention. 64 Trial designs tend to prioritise the objective assessment of clinical outcomes, leaving the process of change (the space between the provision of the intervention resources and the outcomes) unclear. 34,65,66 Process evaluation can lead to an understanding of participants’ experiences and responses to trial interventions (acceptability), reflection on the mechanisms of impact of the intervention(s) and descriptions of the range of individual and contextual factors that affect how the interventions are delivered during the trial. It can also inform how interventions may need to be adapted and/or delivered in routine practice, after the trial, if proven efficacious.

Qualitative reviews and meta-syntheses of patient experiences of recovery after stroke demonstrate the multifaceted impact that stroke can have on a survivor, on their life after stroke and trajectories of recovery. 67–69 Rehabilitation for stroke patients is, therefore, a complex intervention. Often, interventions are multicomponent, target different domains, use multiple outcome measures and depend on engagement from both the participant and the health service professional delivering the treatment.

In line with the UK Medical Research Council’s guidance for the design and evaluation of complex interventions,70 a qualitative process evaluation was conducted to explore the experiences and delivery of two complex interventions (robot-assisted training and EULT) [both described in detail in Chapters 4 and 5, and using the TIDieR framework34 (see Appendix 1, Table 27)] and usual care for upper limb rehabilitation following stroke.

The objectives of this qualitative study were to:

• seek the views and experiences of patients and health service professionals about the upper limb rehabilitation that they have received or provided

• seek the views of patients and health service professionals about the factors that affected the implementation of the trial.

Qualitative process evaluation design and methods

The study design comprised semistructured interviews with both patients and health service professionals, conducted during two time periods (phases) in the trial (Figure 13). During phase 1, patients who were randomised to receive an intervention were interviewed at baseline and at 3 months post randomisation (at the end of the intervention period). During phase 2, different patients were interviewed at 3 months (post intervention) and 6 months (follow-up) post randomisation. Interviews took place in two phases to minimise the likelihood that reporting of the experiences of participants was unduly shaped by factors relating to delivery of the trial at the time of interviewing, which can change over the course of the trial, and to enable an iterative process of data collection and analysis between phases, providing scope to follow up initial findings in greater depth and/or explore questions not targeted in interviews from phase 1. The phase 1 interim analysis also informed phase 2; instead of interviewing patients at the beginning and end of therapy, we interviewed them at the end of therapy and 3 months after completing therapy (6 months post randomisation).This provided a post-intervention period of 3 months during which patients could reflect on any perceived impacts of therapy, before we interviewed them.

FIGURE 13.

Qualitative process evaluation study design. a, Some health service professionals were interviewed twice (n = 5).

In both phases, patients who were randomised to receive a trial intervention (robot-assisted training or EULT) were interviewed twice. This was to allow for exploration of changes in patients’ expectations and experiences at different points during the therapy programme (and to provide more detailed follow-up of emerging key issues at a participant’s second interview). In combination, the two data collection phases allowed us to collect qualitative data corresponding to the three assessment time points in the trial (baseline, 3 months and 6 months), without interviewing any patient more than twice.

Patients from the usual care group were interviewed only once to provide insight into the type of therapy received outside the trial context. Different health service professionals were interviewed up to two times in total, across phases 1 and 2.

Results

Patients’ experiences of robot-assisted training and enhanced upper limb therapy

In this section, we describe the intervention participants’ experiences in terms of treatment acceptability (attendance, intensity and duration), participation challenges, physical and functional impacts, and social and psychological impacts, and we present case studies of participants who made large gains on the ARAT.

Experiences of participants allocated to usual care

Twelve participants who were randomised to receive usual care were interviewed. All had been discharged from hospital; four were receiving ongoing rehabilitation from community-based NHS services, one of whom was supplementing these services with a private physiotherapist. Two participants were accessing ongoing physiotherapy solely through privately funded services. The most intensive rehabilitation received was by Irene (aged 62 years, 5 months post stroke), who reported that she was visited three times a week by a community stroke team. This team was providing therapy to improve Irene’s mobility and the use of her upper limb. The community team had also provided speech and language therapy, which had ended when Irene felt that she was no longer making progress. Although the rehabilitation received by Irene was comparable to either of the RATULS trial therapies in terms of number of visits (three times a week), the focus was not solely on the upper limb.

In comparison with Irene, the least intensive therapy received was by Gillian (aged 64 years, 2 weeks post stroke), who described attending one outpatient group at her local hospital every 3 weeks, where she exercised her upper limb under the supervision of a physiotherapist.

The type and amount of upper limb rehabilitation received by usual care participants varied. Other types of therapy undertaken at the time of the RATULS trial included mirror therapy, electrical stimulation and botulinum toxin. From this small number of interviews, we can conclude that, although it is unlikely that the other therapies and treatments were comparable to the RATULS trial therapy programmes in the targeting of the upper limb, some participants in the usual care group of the trial engaged in (NHS-provided and non-NHS-provided) activities to support their recovery after stroke.

Unsurprisingly, participants randomised to usual care expressed disappointment, particularly as they had had a 66% chance of receiving additional treatment, and because of the length of the screening process, which they felt had built their expectations of receiving treatment:

Then I was disappointed when, well my husband was as well. I don’t think we could quite believe it and, you know, we’d been through all that and why didn’t we just get told that in January instead of waiting all those weeks, ‘cause that was a long time to wait.

Nicola, aged 70 years, 14 months post stroke, usual care

It should be noted that two patients interviewed (Alison, aged 29 years, 5 years post stroke; and Clive, aged 79 years, 4 months post stroke) reported that they were confused by the three randomisation groups and thought that they would be receiving an intervention in time. These findings were reported back to the trial centre teams, who called the participants to provide clarification on what allocation to the usual care group entailed.

Factors affecting implementation of the Robot-Assisted Training for the Upper Limb after Stroke trial

The RATULS trial was complex, evaluating two forms of upper limb rehabilitation; 770 participants, across four trial centres, were recruited over a period of 49 months.

This section focuses on the processes that contributed to the successful delivery and completion of the trial, from the perspective of therapists and trial staff involved in delivering the RATULS trial. We include here, also, reference to professionals’ views about delivery of the different therapies themselves. The results are framed around the constructs of NPT. 71 NPT proposes that the embedding of a practice or intervention, in other words ‘becoming normalised’, is dependent on four kinds of work that must be undertaken: coherence (ways in which an intervention ‘makes sense’ and has clear purpose and objective), cognitive participation (willingness and ability to commit the time and energy needed to make it work), collective action (the resources, arrangements, skills, etc. required to make an intervention work) and reflexive monitoring (formal and informal mechanisms for appraising the process and outcomes of an intervention).

Discussion

This qualitative study was designed to help illuminate the complexities of the RATULS trial and its therapy programmes, through in-depth exploration of patients’ and professionals’ experiences of participating in the trial and of providing robot-assisted training and the EULT programme. Rehabilitation requires the active participation of patients and professionals; therefore, any intervention’s success or failure is partly shaped by their experiences and their engagement with the therapies provided. Participant experiences provide important insight into the acceptability of the treatment, and challenges in intervention delivery that can inform future implementers and decision-makers, and facilitate the interpretation of the trial results.

Our study reports that robot-assisted training and EULT were generally acceptable to patients. Both interventions were physically and mentally challenging and, in some cases, frustrating for patients, when they struggled to complete a task. Some of these challenges were related to severity of the stroke in general (e.g. fatigue, mobility issues and cognitive impairment), and of arm impairment in particular, including sensory and proprioceptive impairment, as well as participants’ expectations of themselves and the intervention. Support from RATULS trial staff, and the patients’ families in trying to overcome these challenges was appreciated. Some patients expressed a wish for therapy to be extended beyond 12 weeks to support their achievement of goals for their upper limb and ongoing recovery more generally.

The RATULS trial reported that robot-assisted training and EULT did not improve upper limb function (the primary outcome) after stroke, compared with usual care. However, improvement in impairment was seen for both robot-assisted therapy and EULT, compared with usual care. Additional improvements in ADL and mobility were found in the EULT group. This study explored patients’ self-reported impacts of therapies. Patients who received either treatment reported physical improvements related to their upper limb, and many reported that these improvements were maintained at 6 months. However, some of the self-reported changes (e.g. better arm positioning, improved sensation and reduced stiffness) would not have been captured by the trial outcome measures, and indicate a potential mismatch (or limited focus) of objective assessments, relative to qualitative exploration of impacts of trialled interventions. This mismatch has been reported previously and highlights the importance of capturing participants’ own experiences of intervention impacts. 75 In addition, although patients generally expressed some level of satisfaction with physical improvements resulting from therapy, they self-assessed their progress with reference to their more general hopes and expectations for complete recovery, which were focused on regaining ‘before stroke’ capabilities and functioning.

A more detailed analysis of two phase 2 interviewees’ (who achieved a ‘successful’ outcome according to the RATULS trial criteria) experiences of therapy impact indicated that improvement is more complex than assessment scores indicate. There is a need to ensure that patient and practitioner understandings of therapy interventions are aligned as much as possible. Our findings highlight the importance of what ‘clinically assessed improvement’ means to patients, in terms of functionality and practical gains, and in terms of what impacts of therapy they experience as valuable. This point was further emphasised, as participants reported a range of positive impacts that crossed social and psychological domains, for example social engagement and increased confidence that arose from taking part in a trial (but these did not seem to be associated with the intervention content per se). These are additional benefits of therapy that are important for patients in their recovery process, but that would not be expected to be captured in measures included in a trial.

Although the RATULS trial demonstrated improvement in impairment following robot-assisted training, those who received EULT additionally showed improvement in ADL. This was not surprising, given the specificity of learning. In robot-assisted training, movements were stereotypical, identical and did not involve everyday objects, and movement patterns were tightly controlled by the device. In contrast, health-care professionals reported that the EULT programme offered more scope for working on a range of activities that linked to patients’ personal goals, and they also felt that there was more opportunity to monitor and facilitate motor control in this type of treatment. Activities could be varied, tailored and progressed, and feedback could be used selectively; all of these are components for optimising skill acquisition and transfer. Patients in the EULT group indicated that they valued some of the functional goals they had achieved.

Some participants (in both intervention groups) were engaging in other treatments (e.g. botulinum toxin, other forms of physiotherapy and other physical rehabilitation) outside the RATULS trial provision, as were some participants in the usual care group. This was allowed in the protocol. Furthermore, some participants in both groups expressed a need for therapy to be extended beyond 12 weeks, to support their rehabilitation potential in their upper limb and more general recovery. This resonates with the findings of a qualitative study of Australian stroke survivors’ (n = 19) and spouses’ (n = 9) perceptions of upper limb recovery after stroke, that emphasised survivors’ approach as one of ‘keeping the door open’ – maintaining hope for and continuing to work towards improvement. 76

Although broader conclusions about the therapies cannot be made for patients collectively on the basis of specific examples, our findings indicate that rehabilitation for upper limb recovery is complex, with the arm often being only part of the picture and patients engaging in multiple treatments (when possible) to achieve their rehabilitation potential. Effective engagement in upper limb therapy (both robot-assisted training and EULT) is likely to be influenced by factors related to stroke, such as fatigue, cognition and communication abilities, as well as emotional factors, motivation, self-efficacy and social support. These factors also influence other aspects of a patient’s life that constitute rehabilitation, such as participation in social activity and return to work. It must also be emphasised that, even for those indicating good improvement from upper limb therapy, the end of treatment does not mean the end of the recovery journey.

The second objective of this qualitative study was to identify factors affecting the implementation of the trial, and the trial therapies, increasingly an important focus in process evaluation of complex interventions. 64 It is important to understand two different objects of implementation here: the trial in its broader sense (which includes the therapies and delivery contexts, in combination with trial processes) and the therapies themselves, which may be subsequently provided in routine service provision in the absence of the trial. We used NPT to present the results about implementation of the trial and the RATULS trial therapies in that context. 71 The findings indicated that successful delivery of the trial required continuous investment in NPT framed ‘work’ to maintain commitment, engagement and reflection/adaptation to some of the trial processes, to achieve the tasks (primarily recruitment, therapy delivery and data collection) required for the RATULS trial.

Early engagement of trial centres in planning the trial enabled development of understanding of the trial and of the therapies being delivered (shared by most trial centres) (coherence), which facilitated engagement by staff (cognitive participation). In terms of achieving the work of the trial (collective action), allocation of therapy staff (dedicated RATULS therapists when possible) was highly facilitative, as was a level of flexibility in therapy provision and organisation (e.g. allowing patients to express preferences for days of therapy). The level of support, monitoring and feedback from the co-ordinating centre throughout the trial (regarding recruitment, completion of assessments, etc.) (reflexive monitoring) ensured ongoing engagement of sites, staff and patients.

Findings suggest that both robot-assisted training and EULT could be delivered successfully in routine service provision, if adequately resourced and optimally organised to allow for reasonable flexibility in delivery. Apart from some frustrations expressed by staff in relation to robot equipment failures, both therapies were acceptable to staff after adequate training in the treatment protocols.

Methodological reflections indicate key strengths, and some limitations, of this qualitative process evaluation. This is a large qualitative study of a multicentre trial, providing experiential accounts of trial participation and delivery from staff and patients across multiple time points during a patient’s trajectory through the trial. Conducting the interviews in two phases allowed some iterative development of fieldwork focus, based on issues arising during the trial (and from phase 1). This approach also enabled coverage of three different time points during therapy (commencement, completion and maintenance), while limiting the number of interviews for individual patients; in particular, the extent to which patients felt that they had been able to maintain any impacts at 6 months (3 months after therapy completion) could be explored in phase 2. As a qualitative process evaluation study, the absence of an observational component to supplement (or triangulate) interview findings may be a limitation, in that we have focused on participants’ accounts of therapy rather than having directly observed the sessions they spoke of. Observational methods can be valuable, but also intrusive to those involved. If we had taken this approach in the RATULS trial, it may have affected therapy negatively, for limited benefit in data collection. Other data were collected during the trial to document therapy provision, which can be used for some level of data triangulation, and these suggest that therapy was generally delivered as intended. The findings from the subset of trial participants who took part in the qualitative process evaluation have highlighted a range of experiences, and differing levels of engagement in therapy, that were able to be achieved by patients with different needs, and at different points in their recovery process after stroke. This suggests that adequate variability of participants in the trial was achieved.

Overall, the RATULS trial participants’ accounts of stroke were strongly aligned with the existing qualitative literature concerning the multifaceted impacts of stroke,68,69,77 variable nature of recovery over time76,78,79 and expression of hopes and expectations concerning fuller recovery over time. 76,80 Participating in the RATULS trial was viewed by patients as an opportunity to extend or supplement their recovery process, particularly in relation to upper limb rehabilitation. Some participants expressed disappointment on being allocated to the usual care group. Participants who received robot-assisted training and EULT tended to be highly motivated and engaged well with their therapies, and reported a range of benefits that represented improvements.

From the perspective of professional staff who delivered the RATULS therapies, both therapies could be delivered to patients as a 12-week programme, providing the necessary resources were available and staff engagement was high, and provided there was scope to make adjustments to therapy delivery to accommodate patient, therapist, equipment and organisation factors affecting therapy provision. Provision of both robot-assisted training and EULT in routine care (beyond the RATULS trial) may be more flexible in terms of delivery and organisation than was possible in the context of the trial, in which adherence to therapy and trial protocols were of paramount importance. Routine provision of these therapy treatments may differ, however, in ways that affect efficacy, and should be further explored in this context.

Conclusion

Participants in both the robot-assisted training and EULT groups generally found the therapies to be acceptable, and were able to complete their therapy programmes. Not surprisingly, some challenges were experienced by some participants relating to their upper limb capabilities and other stroke-related limitations that affected their engagement in the programmes. Participants reported a range of benefits from participating in either therapy programme, and many reported (self-judged) maintenance of some of the benefits at 6 months.

Delivering the RATULS trial required continuous investment of effort by trial centres and the co-ordinating centre, with high levels of engagement from the staff involved. At times, flexibility and adaptation of some of the processes, within the constraints of the trial protocol, were necessary to support continued engagement of patients and professionals. The findings of this qualitative process evaluation can be used to inform the delivery of robot-assisted training and the EULT programme for upper limb rehabilitation, for service provision beyond the RATULS trial.

Introduction

The health economic evaluation undertaken as part of the RATULS trial assessed the relative cost-effectiveness of the robot-assisted training using the MIT-Manus robotic gym, compared with the EULT programme and usual care. This evaluation included a within-trial analysis and an extrapolation of the results to estimate costs and outcomes beyond the time frame of the trial.

The analyses took the perspective of the UK NHS and PSS. A wider societal perspective was also explored by analysing the costs borne by participants and their carers. These costs included out-of-pocket expenses incurred for hospital admissions, GP visits and outpatient appointments and time away from usual activities such as work, study or caring responsibilities.

This chapter includes the main results of the within-trial economic evaluation and the extrapolation exercise beyond the trial time frame. For the within-trial analysis, the following outcome measures are reported:

• NHS and PSS resource use

• health-care costs per participant to the NHS and PSS over 6 months for each area of resource use

• utility scores derived from responses to the EQ-5D-5L26 questionnaire at baseline and at 3 and 6 months post randomisation

• average QALYs per participant at 6 months post randomisation

• incremental cost-effectiveness ratios (ICERs)

• cost-effectiveness acceptability curves (CEACs) to assess the probability of each of the interventions being considered most cost-effective at different willingness-to-pay (WTP) thresholds for a gained QALY.

For the extrapolation exercise, the following outcome measures are reported

• average health-care costs per participant at 12 months post randomisation

• average QALYs per participant at 12 months post randomisation

• ICERs and CEACs derived by extrapolating costs and QALYs from the data observed during the trial.

Results are also reported for the prespecified per-protocol and sensitivity analyses. The sensitivity analyses explored uncertainty surrounding the level of resource use and the assumed lifespan of the MIT-Manus robotic gym.

A stochastic sensitivity analysis was carried out for all analyses to address potential variance in the outcome measures.

Methods

The economic analysis was conducted following best-practice guidelines conforming to the Consolidated Health Economic Evaluation Reporting Standards. 81

Results

Results from the within-trial economic evaluation

Incremental cost-effectiveness analysis

The unadjusted CEA shows that usual care was less costly than robot-assisted training and EULT, but that usual care was less effective than EULT. There was no evidence of a difference in QALYs between groups (Table 25). The deterministic results from the adjusted analysis show an ICER of £74,100 for the EULT group, compared with usual care. Robot-assisted training was dominated by EULT as it was, on average, both more costly and less effective, although this was not statistically significant.

TABLE 25 - Results from base-case and stochastic cost–utility analysis for robot-assisted training, EULT and usual care based on an ITT analysis

Notes

Unadjusted analysis: costs – usual care, n = 178; EULT, n = 259; robot-assisted training, n = 257. QALYs – usual care, n = 254; EULT, n = 259; robot-assisted training, n = 257). Adjusted analysis: usual care, n = 171; EULT, n = 254, robot-assisted training, n= 247.

Randomisation group	Unadjusted mean (98.33% CI)		Adjusted incremental (98.33% CI)		Adjusted ICER (£)	Probability that each therapy is cost-effective at WTP of
	Cost (£)	QALY	QALY	Cost (£)		£0	£10,000	£20,000	£30,000	£50,000
Usual care	3785 (2801 to 4770)	0.21 (0.19 to 0.23)	–			0.90	0.85	0.81	0.74	0.62
EULT	4451 (3548 to 5354)	0.23 (0.21 to 0.24)	0.010 (–0.005 to 0.025)	741 (–461 to 1943)	74,100	0.10	0.15	0.19	0.26	0.38
Robot-assisted training	5387 (4777 to 5996)	0.21 (0.19 to 0.23)			Dominated by EULT	0.00	0.00	0.00	0.00	0.00

The results of the deterministic analysis do not reflect the statistical imprecision surrounding estimates of cost-effectiveness. This is provided by the stochastic sensitivity analysis. The results from this sensitivity analysis (Figure 14) suggest that there was approximately a 20% chance that EULT was cost-effective at a £20,000 WTP threshold value, and the probability that EULT was cost-effective is never > 40% over the range of WTP values considered. Robot-assisted training has a 0% chance of being cost-effective at all the WTP values considered in the analysis.

FIGURE 14.

Cost-effectiveness acceptability curve (base-case analysis): adjusted bootstrapped replications for CEA.

Additional graphical representation of the bootstrapped incremental costs and QALYs for EULT and robot-assisted training, compared with usual care, can be seen in Figures 15 and 16. Figure 15 represents a comparison between EULT and usual care, showing the bootstrapped incremental costs and QALYs, taking usual care as the reference point. As seen in Table 25, the majority of bootstrapped values show that EULT was more costly and more effective than usual care. The majority of these bootstrapped points fall above the £20,000 WTP threshold line, confirming that EULT has a low probability of being cost-effective except when society’s WTP for a QALY is very high.

FIGURE 15.

Incremental costs and QALYs for EULT in relation to usual care.

FIGURE 16.

Incremental costs and QALYs for robot-assisted training in relation to usual care.

For completeness, we have also graphically represented the bootstrapped incremental costs and QALYs for robot-assisted training compared with usual care. Figure 16 shows how the majority of bootstrapped values for robot-assisted training confirm that it was, on average, both more costly and as effective as usual care. A very small number of bootstrapped values fall below the £20,000 WTP threshold line, confirming that the probability of robot-assisted training being cost-effective is very close to zero, even when society’s WTP for a QALY is higher than might normally be considered to be the case.

Deterministic sensitivity analysis

The results of each scenario explored in the deterministic sensitivity analysis are detailed in Appendix 8, Deterministic sensitivity analysis. Table 26 provides a summary of the main findings. Robot-assisted training remains dominated in all the scenarios explored.

TABLE 26 - Results from adjusted sensitivity analysis

Estimates based on adjusted analysis.

Sensitivity analysis	Incremental (98.33% CI)a		ICER (£)	Probability of EULT being considered cost-effective at a WTP of £20,000
	Cost (£)	QALY
Extreme analysis	1892 (823 to 2961)	0.011 (–0.002 to 0.024)	172,000	0.00
Multiple imputation	550 (–503 to 1604)	0.011 (–0.002 to 0.024)	50,000	0.27
Useful life of robot extended to 7 years	741 (–460 to 1943)	0.010 (–0.005 to 0.025)	74,100	0.19
Physiotherapy sessions excluded	486 (–702 to 1674)	0.010 (–0.005 to 0.025)	48,600	0.34

When missing costs were changed to zero, the resulting ICER between EULT and usual care increased to £172,000 (see Appendix 8, Tables 74 and 75, and Figure 17). This increase was to be expected because all participants with missing total costs belonged to the usual care group. By imputing zero for missing costs, the mean costs for the usual care participants were reduced; hence, the ICER increased when compared with EULT (see Appendix 8, Tables 76 and 77, and Figure 18).

Results from extending the life of the robotic gym are shown in Appendix 8, Tables 78 and 79, and Figure 19. The lowest ICER resulted from the exclusion of physiotherapy sessions from the analysis (£48,600) (see Appendix 8, Tables 80 and 81, and Figure 20).

Subgroup analysis

The results from the adjusted and unadjusted subgroup analysis are outlined in Appendix 8, Subgroup analysis. Adjusted and unadjusted results suggest that robot-assisted training was, on average, more costly and less effective than EULT across all subgroups considered. The ICER, therefore, relates to the comparison between EULT and usual care at all times. The highest ICER (£126,143) relates to those participants who, at randomisation, had experienced a stroke between 1 and 5 years ago. The subgroup of participants who were < 3 months post stroke at randomisation had the lowest ICER (£31,400). The stochastic sensitivity analysis for this group suggests that EULT has a 41% probability of being cost-effective at the £20,000 WTP threshold.

The subgroup analysis results seem to mimic those found in the base-case economic evaluation. The results, however, need to be interpreted with caution because of the reduced sample size in each subgroup.

Per-protocol analysis

Results from the per-protocol analysis are reported in Appendix 8, Per-protocol analysis. Both the unadjusted and adjusted results suggest that robot-assisted training is, on average, more costly and less effective than EULT, and more costly and slightly more effective than usual care.

The resulting ICER (£68,000) relates to the comparison of EULT with usual care. The stochastic sensitivity analysis suggests that EULT has a 17% probability of being cost-effective at the £20,000 WTP threshold.

Discussion

The main strength of this economic evaluation is that it was conducted alongside a rigorously run pragmatic RCT and followed guidelines for best practice throughout. 87,91 As a result, the economic evaluation was based on individual patient data collected during the trial and benefited from small numbers of missing health-care resource and quality-of-life data. Completion of baseline questionnaires was close to 100% across all randomisation groups. Although response rates for the health service utilisation questionnaire remained high at 6 months, at 79% overall, this decreased to 70% for participants in the usual care group. This number of missing responses is, however, to be expected. To assess the impact of the missing data, multiple imputation methods were applied in a sensitivity analysis. The cost-effectiveness results obtained using the imputed variables were consistent with those from the base-case analysis.

Self-reported quality-of-life data were collected at three points during the trial using the EQ-5D-5L questionnaire. 26 This enabled us to accurately measure QALY gains for participants across all groups using the preference-based instrument recommended by NICE. 88 One important strength is that assessing quality of life in this way will enable cost-effectiveness comparisons not only for this patient group, but across different disease areas. Following NICE guidance,88 responses to the EQ-5D-5L questionnaire were mapped to the EQ-5D-3L descriptive system to generate the utility values.

A noteworthy limitation of the economic evaluation is associated with the time frame of the trial. The within-trial economic evaluation assessed the cost-effectiveness of the interventions at 6 months. A longer-term perspective would have been achieved through the development of the long-term full economic model that was originally planned. However, given the results of the trial and the dominant status of usual care, the development of this type of model was no longer considered appropriate. As an alternative, a shorter-term model was conducted to derive cost-effectiveness results at 12 months. The results, however, need to be interpreted with caution because of the restricted assumptions made on both costs and utility values.

The economic evaluation fills a significant evidence gap, as no economic evaluation comparing robot-assisted training with usual care had been conducted in the UK NHS setting before, to our knowledge. A scoping review found little evidence of cost-effectiveness studies in the UK, and the only economic evaluation assessing the cost-effectiveness of robot therapy was US based. 18 Differences in the health-care systems between both countries mean that, unlike our economic evaluation, the results from this US study18 lack generalisability to the UK setting. Furthermore, there were differences in the therapy received by the randomisation groups, which reduces the grounds for comparability with the care received in the NHS.

The use of multiple sites across the UK contributed to the generalisability of the economic evaluation. The analysis controlled for differences in trial centres, hence minimising the chance of obtaining biased results from differences in costs and effects driven by location.

Overall, our economic evaluation was designed and conducted following NICE’s guidance for best practice,87,91 which resulted in robust and generalisable results.

The results from the economic evaluation create opportunities for further research. In particular, further analyses could explore the potential effect on both costs and QALYs of reconfigurations to the delivery of EULT and robot-assisted training.

Conclusion

The CEA results suggest that, on average, robot-assisted training was more costly than both EULT and usual care, and that robot-assisted training was less effective than EULT and as effective as usual care. EULT was, on average, more effective and more costly than usual care in both the adjusted and unadjusted CEA. The balance of probabilities favoured usual care as the preferred upper limb rehabilitation therapy over the range of WTP values considered for the participants included in the trial. Focusing on society’s WTP for a QALY, the stochastic analysis suggests that the balance of probabilities does not favour EULT as being cost-effective over any of the WTP values considered, despite EULT being more effective than the other randomisation groups.

A subgroup analysis in which the participants were categorised according to time since stroke did not change the direction of the cost-effectiveness results.

The lowest ICER (£48,600) between EULT and usual care in the sensitivity analysis was reported when the costs of physiotherapy sessions were excluded. Extrapolating within-trial results to 12 months produced the lowest ICER (£6095) for the comparison between EULT and usual care overall; however, high uncertainty surrounds the assumptions made on costs and utilities for the 12-month period. Further studies are needed to confirm if these results hold true.

Key findings

Robot Assisted Training for the Upper Limb after Stroke trial results in the context of other studies

Mechanisms of action of robot-assisted training and enhanced upper limb therapy

Theories of neuroplasticity and motor learning99,100 support an approach to rehabilitation based on repetitive practice of tasks. 101,102 Both interventions were, therefore, underpinned by established theories of motor learning, including the stages of learning103 and motor programme theory,100,104 to structure the learning processes. Evidence from translational studies105,106 further supports the use of progressive, repetitive training protocols. Further information can be found in the RATULS interventions TIDieR table (see Appendix 1, Table 27).

It is important to consider why the improvements in impairment seen with robot-assisted training in the RATULS trial did not translate into improved function. The RATULS trial provided integrated training with all three components of the MIT-Manus robotic gym (shoulder–elbow robot, wrist robot and hand attachment), which adapt to participants’ abilities (providing more/less assistance as required). The robot-assisted training programme did not include grip or pinch activities. Participants trained specific movements of the affected arm in a spatially controlled manner, but it is possible that these did not resemble movements in everyday activities and, therefore, improvements in motor control (as shown on the FMA) did not transfer into ADL. Furthermore, there may have been a lack of guidance for participants about making the best use of any improvement in impairment in day-to-day activities. These suggestions are supported by data from the qualitative study, in which some interviewees commented that robot-assisted training focused on achieving gross arm movements rather than delicate tasks.

A trial107 published in 2019 (45 participants) randomised participants to receive robot-assisted training using the MIT-Manus robotic gym or robot-assisted training plus therapist-assisted transition-to-task training. Both interventions were provided for 1 hour, three times per week, for 12 weeks. There was no difference in the primary outcome: FMA score at 12 weeks. However, participants who received robot-assisted training plus therapist-assisted transition-to-task training showed a statistically significant improvement in motor performance (as measured by the Wolf Motor Function Test96) and hand function (as measured by the SIS42) at 12 weeks, compared with those who received robot-assisted training alone.

Many daily activities involve both upper limbs, including the hands, in bilateral (e.g. opening a drawer) or bimanual (e.g. making a sandwich) co-ordinated action. Furthermore, daily activities (e.g. buttoning a shirt) often involve a range of objects that require different types of grasp and/or manipulation. Such activities involve diverse and complex patterns of behaviour involving sensory, perceptual, cognitive and motor functions. This could explain why robot-assisted training resulted in a less favourable outcome in self-reported ADL than the EULT programme, in which training specifically focused on complex daily activities and functional tasks. The improvement in mobility seen following EULT would also support this theory, as the EULT programme included tasks involving the upper limbs in balance and sit-to-stand activities, which were not components of robot-assisted training. The EULT programme may also have addressed learned non-use of the affected arm by encouraging participants to use their arm in day-to-day activities; one participant who was interviewed in the qualitative study stated ‘I had completely forgotten what this hand did’.

The results of the RATULS trial provide a further example of the specificity of learning. 100 This is also supported by the results of the NIHR HTA Clinical and cost-effectiveness of aphasia computer treatment versus usual stimulation or attention control long term post-stroke (Big CACTUS) RCT,108 which found that, although patients with dysphasia due to stroke who undertook a self-managed programme of computer exercises showed improved word-finding abilities, this did not translate into improvement in general conversation ability.

The Robot-Assisted Training for the Upper Limb after Stroke trial interventions

Usual care

The RATULS trial was a pragmatic trial; therefore, usual care was chosen as a comparator group. Obtaining accurate information about the usual care that patients receive is a challenge to this type of study, and the assumption that usual care is uniformly provided is rarely true. 116 Although there are UK guidelines about the amount of therapy that a stroke patient should receive, they refer to contact time with a specific type of therapist (e.g. physiotherapist) rather than specifying the focus (e.g. upper limb) or intensity of the intervention itself. 117

Usual care records are not standardised and, often, basic information about content and dose is not recorded. Collecting data about the upper limb therapy provided by a large number of individuals from a range of services and organisations (stroke units, early supported discharge teams, community rehabilitation teams, outpatient therapists from hub and spoke sites), many of whom were not directly involved in the RATULS trial, was not feasible. 2 In addition, some patients arranged private rehabilitation. Therefore, participants were asked to record the usual care they received and to seek help if needed from their therapist(s) to complete their log books. However, this approach has limitations, including data completeness. Although the number of participants who returned a therapy log was low (66% at 3 months), there was little variation between randomisation groups for the number of log books returned or the completeness of the logs returned, for example of those returned, > 80% had all 12 weeks completed.

The usual care logs in the RATULS trial indicated that the most common type of therapy undertaken was passive stretching, whereas the least common was practising extended ADL. This finding resonates to some extent with findings from a recent survey of upper limb treatment reported by physiotherapists and occupational therapists across the UK. 118 This survey reported that, although there was considerable variation in treatments provided for stroke patients with severe upper limb deficits, the most common were range of movement exercises, mirror-box therapy and functional electrical stimulation. For those with less severe upper limb deficit, most interventions comprised active, task-specific practice. In terms of dose, upper limb therapy was reportedly provided for an average of 29 minutes, three times per week. 118 However, this amount exceeds that of observational studies of upper limb treatment. 76,111,119

Recording, measuring and monitoring the treatment received by individuals, and reporting of the rehabilitation services provided, should be integral to routine clinical practice and should not be seen as an added burden. 92 If this was achieved, these data could be available to research teams (with appropriate information governance), and would greatly facilitate the reporting of usual care in rehabilitation trials.

Disappointment about group allocation may have resulted in some usual care participants seeking or being provided with additional therapy, or increasing the amount of self-practice exercises undertaken, thereby introducing a competitive therapy bias. We did consider offering either robot-assisted training or EULT to usual care participants following the 6-month outcome assessment, but this was not feasible.

Methodological considerations

To our knowledge, the RATULS trial is the largest trial of upper limb robot-assisted rehabilitation undertaken to date. The RATULS trial is the first multicentre trial with adequate statistical power to compare robot-assisted training with both an evidence-based therapy programme of the same frequency and duration and with usual care.

Patient and public involvement

Our involvement with patient groups, clinical experience and research has consistently shown that patients feel that stroke rehabilitation focuses on mobility rather than recovery of upper limb function. Improving upper limb function has been identified as the fourth most important rehabilitation research priority by stroke survivors, carers and clinicians. 9

The Guidance for Reporting Involvement of Patients and Public 2147 was published in 2017, towards the end of the RATULS trial recruitment period; therefore, we were unable to be fully compliant. We have not systematically recorded how patient and carer involvement enhanced the research process throughout the trial.

Stroke survivors were involved in developing the outline and full NIHR proposal. At the grant application stage, the trial was presented to the NIHR Stroke Research Network (SRN) Lay Member Panel and comments from the panel were incorporated into the application. Input was sought from the North East SRN Patient and Carer Panel for the design and content of trial documents, including the trial information sheets and consent forms, and to help to develop resources to train staff who undertook trial assessments. Stroke survivors contributed to the development of the intervention programmes used in the RATULS trial and ran through the programme with researchers. The programmes were adjusted following their feedback.

One of the co-applicants is a stroke survivor and contributed to design, delivery and reporting of this work. One of the members of the Trial Steering Committee had had a stroke and gave a patient perspective about the trial.

Trial participants were sent newsletters throughout the trial and were sent a letter that included a lay summary of the results of the trial, following publication of the main results in The Lancet. 1 The results of the trial have been presented and discussed with the NIHR Clinical Research Network: Stroke North East Patient and Carer Panel (formerly the SRN Panel).

Implications for health and social care

The results of the RATULS trial do not support the routine use of robot-assisted training using the MIT-Manus robotic gym, as implemented in this trial, for patients with moderate or severe upper limb functional limitation due to stroke.

The RATULS trial provides some evidence of the potential benefits of the EULT programme, although, as delivered in this trial, it is unlikely to be cost-effective. Delivering EULT as group/classroom therapy would lead to a reduction in costs, which has the potential to make EULT a cost-effective intervention. However, the impact of this change on clinical effectiveness and QALYs is unknown and should be explored.

Recommendations for further research

The RATULS trial has demonstrated that a large multicentre trial to evaluate robot-assisted rehabilitation, which includes a clear description of the interventions received and high levels of intervention fidelity, is achievable. An editorial in The Lancet entitled ‘Robot-assisted training after stroke: RATULS advances science’125 was supportive of the trial, and we suggest that future rehabilitation trials could build on our experience.

Future trials may wish to stratify patients into groups with differing probabilities of upper limb recovery using techniques such as shoulder abduction and finger extension52 and/or advanced neuroimaging and transcranial magnetic stimulation to improve the targeting of therapies towards participants with the potential to respond. 148 Ongoing biomarker research may help in patient selection or treatment monitoring of future stroke rehabilitation trials. 110 Future trials may also wish to consider having a patient-reported outcome measure and one of the standard activity scales as co-primary outcome measures. 75 Future trials should be aligned with recommendations made by the Stroke Recovery and Rehabilitation Roundtable. 139,149

Recording, measuring and monitoring the treatment received are important components of research, clinical practice and audit. By following checklists and guidance to describe research interventions, the content of rehabilitation interventions evaluated in clinical trials could be accurately reported. 34,110,150 Unfortunately, collecting standardised minimum data about the usual care received remains a challenge in many settings, especially when it is provided by many services in a single study centre. 92 There needs to be a culture change in the recording and reporting of usual care in clinical practice to enable key data to be used to inform clinical practice and research.

The dose of therapy is an important component of any stroke rehabilitation trial. Therapy dose is a multifactorial concept. 127 Dosage can include the number of sessions attended, session frequency and duration, duration of face-to-face therapy, and the number of repetitions of tasks undertaken. Equally important is reporting of the type of therapeutic activities undertaken, and the way these are progressed. There is converging evidence that more therapy may result in better outcomes,50 but, in future, adequately powered dose-finding studies of promising interventions, tailored to targeted subgroups, are needed, that also take into account potential cost-effectiveness.

Despite the neutral primary outcome of the RATULS trial, robot-assisted training in upper limb rehabilitation after stroke still has potential. Further research is needed to determine how improvement in upper limb impairment seen with the MIT-Manus gym can be translated into meaningful improvement in upper limb function and ADL. For example, trials could be designed whereby robot-assisted training could be utilised at the start of an intervention session as a ‘primer’,151 with the aim to focus attention on the affected arm, assist movement and generate sensory, proprioceptive and visual feedback. The EULT intervention could then be utilised (in the same session or after a number of robotic sessions) to enable translation of improved motor control into meaningful, daily activities. 107,152

Robotics is a rapidly advancing field and a number of other innovative devices are available, some of which may be suitable for home use. In some parts of the world, the clinical use of robots has preceded evidence of effectiveness. It is important that there is partnership between developments in bioengineering, neuroscience and clinical evaluation to optimise improvement in patient outcomes and care.

Contributions of authors

Professor Helen Rodgers (https://orcid.org/0000-0003-3433-4175) (Professor of Stroke Care) was the chief investigator and lead grant holder. She was involved in the study design and delivery, interpretation of data and drafting the report.

Dr Helen Bosomworth (https://orcid.org/0000-0002-4670-8827) (Research Associate) was a researcher working on the project who managed the project. She was involved in the study design and delivery, interpretation of data, analysis of intervention content data and drafting the report.

Dr Hermano I Krebs (https://orcid.org/0000-0002-9317-0900) (Principal Research Scientist and Lecturer) was a co-investigator, involved in the study design and delivery, design of the robot-assisted training programme, interpretation of data and drafting the report.

Professor Frederike van Wijck (https://orcid.org/0000-0003-0855-799X) (Professor in Neurological Rehabilitation) was a co-investigator, involved in the study design and delivery, design of the EULT programme, interpretation of data and drafting the report.

Ms Denise Howel (https://orcid.org/0000-0002-0033-548X) (Senior Lecturer) was a co-investigator and the senior statistician. She was involved in the study design and delivery, interpretation of data and drafting the report, and supervised the main patient and carer analyses.

Dr Nina Wilson (https://orcid.org/0000-0001-5908-1720) [Research Associate (statistician)] was the study statistician who conducted the main data analyses and contributed to the drafting of Chapter 3.

Professor Tracy Finch (https://orcid.org/0000-0001-8647-735X) (Professor of Healthcare & Implementation Science) was a co-investigator and lead for the parallel process evaluation. She was involved in the study design, delivery, interpretation of data and drafting the report, and supervised the process evaluation.

Dr Natasha Alvarado (https://orcid.org/0000-0001-9422-4483) (Research Fellow) was a researcher working on the project who undertook the parallel process evaluation data collection and analyses, and drafted Chapter 7.

Dr Laura Ternent (https://orcid.org/0000-0001-7056-298X) (Senior Lecturer in Health Economics) was a co-investigator and senior study health economist. She was involved in the study design and delivery, interpretation of data and drafting the report, and supervised the health economic analyses.

Dr Cristina Fernandez-Garcia (https://orcid.org/0000-0002-7113-225X) (Health Economist and Bioethicist) was the health economist who conducted the main data analyses, and contributed to the drafting of Chapter 8.

Mrs Lydia Aird (https://orcid.org/0000-0002-0578-9309) (Specialist Physiotherapist in Stroke) was a co-investigator and was involved in the study design and delivery, interpretation of data and drafting the report.

Dr Sreeman Andole (https://orcid.org/0000-0001-8356-9320) (Clinical and Research Lead in Stroke) was a co-investigator and was involved in the study design and delivery.

Dr David L Cohen (https://orcid.org/0000-0001-7318-8567) (Consultant Physician) was a co-investigator and was involved in the study delivery, interpretation of data and drafting the report.

Professor Jesse Dawson (https://orcid.org/0000-0001-7532-2475) (Professor of Stroke Medicine and Consultant Physician) was a co-investigator and was involved in the study design and delivery, interpretation of data and drafting the report.

Professor Gary A Ford (https://orcid.org/0000-0001-8719-4968) (Professor of Clinical Pharmacology) was a co-investigator and was involved in the study design and delivery, interpretation of data and drafting the report.

Dr Richard Francis (https://orcid.org/0000-0002-6568-1295) (Research Associate) was a researcher working on the project who contributed to data management and analyses during the study delivery period.

Mr Steven Hogg (Stroke Survivor) was a co-investigator and is a service user; he was involved in the study design and delivery, and interpretation of data.

Dr Niall Hughes (https://orcid.org/0000-0001-5555-6442) (Consultant Physician) was a co-investigator and involved in the study design and delivery, interpretation of data and drafting the report.

Dr Christopher I Price (https://orcid.org/0000-0003-3566-3157) (Reader) was a co-investigator and involved in the study design and delivery, interpretation of data and drafting the report.

Professor Duncan L Turner (https://orcid.org/0000-0001-8916-4025) (Professor of Neurorehabilitation Sciences) was a co-investigator and was involved in the study design and delivery, interpretation of data and drafting the report.

Professor Luke Vale (https://orcid.org/0000-0001-8574-8429) (Health Foundation Chair in Health Economics) was a co-investigator and senior study health economist. He was involved in the study design, delivery, interpretation of data and drafting the report, and supervised the health economic analyses.

Professor Scott Wilkes (https://orcid.org/0000-0003-2949-7711) (Professor of General Practice and Primary Care) was a co-investigator and involved in the study design, delivery, interpretation of data and drafting the report.

Dr Lisa Shaw (https://orcid.org/0000-0002-3435-9519) (Principal Research Associate) was a co-investigator and managed the project. She was involved in the study design and delivery, interpretation of data and led the drafting of the report.

Publications

Rodgers H, Shaw L, Bosomworth H, Aird L, Alvarado N, Andole S, et al. Robot Assisted Training for the Upper Limb after Stroke (RATULS): study protocol for a randomised controlled trial. Trials 2017;18(1):340.

Rodgers H, Bosomworth H, Krebs HI, Van Wijck F, Howel D, Wilson N, et al. Robot Assisted Training for the Upper Limb after Stroke (RATULS): a multicentre randomised controlled trial. Lancet 2019;394:51–62.

Rodgers H, Bosomworth H, Van Wijck F, Krebs HI, Shaw L. Usual care: the big but unmanaged problem of rehabilitation evidence – authors’ reply. Lancet 2020;395:337–8.

Bosomworth H, Rodgers H, Shaw L, Smith L, Aird L, Howel D, et al. Evaluation of the enhanced upper limb therapy programme within the Robot-Assisted Training for the Upper Limb after Stroke (RATULS) trial: descriptive analysis of intervention fidelity, goal selection and goal achievement. Clin Rehabil 2020; in press.

Data-sharing statement

All data requests should be submitted to the corresponding author for consideration. Access to available anonymised data may be granted following review. A data-sharing agreement will need to be signed by data requestors.

Patient data

This work uses data provided by patients and collected by the NHS as part of their care and support. Using patient data is vital to improve health and care for everyone. There is huge potential to make better use of information from people’s patient records, to understand more about disease, develop new treatments, monitor safety, and plan NHS services. Patient data should be kept safe and secure, to protect everyone’s privacy, and it’s important that there are safeguards to make sure that it is stored and used responsibly. Everyone should be able to find out about how patient data are used. #datasaveslives. You can find out more about the background to this citation here: https://understandingpatientdata.org.uk/data-citation.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health and Social Care.