Cognitive remediation therapy to enhance cognition and improve recovery in early psychosis: the ECLIPSE research programme including an RCT

Design

Two focus groups of service users (each of n = 6–8) from EIS each met twice, the second time for the purpose of respondent validation. This was facilitated by service user researchers, digitally recorded and transcribed. The same process was carried out with EIS staff members.

Sample

Purposive sampling was used to recruit EIS participants from a single site, aged 18–35 years, within the first 3 years of their illness. All service users were considered eligible if they could give written consent and communicate in English. All EIS staff were eligible if they gave written consent.

Procedure

Repeated focus groups35 were facilitated by service user researchers, digitally recorded and transcribed for both service users and staff. At the outset, the service user group was shown a live demonstration of CIRCuiTS and the staff group a presentation. A topic guide generated discussion on the content and format of the intended measure. We used an existing service user satisfaction measure, previously designed for a paper-and-pencil evaluation20 and built an interview guide (Report Supplementary Material 1) for assessing satisfaction with CIRCuiTS with that measure. 24 The service user researchers conducted a thematic analysis on the first focus group, and these data were used to develop a new draft measure. This draft measure was taken to the second focus groups for checking. The data were thematically analysed a second time and the final measure amended accordingly.

Participants

All eight service user participants had a diagnosis of psychosis and were from minority ethnic communities; 87% were men with a median age of 28 years. Eleven clinical staff (69%) were women, 10 (63%) were of white ethnic origin and ranged in age from 20 to 40 years. The majority (n = 14; 88%) were care coordinators.

Using CIRCuiTS

Although CIRCuiTS appealed to many of the service users, some initially felt daunted by its apparent complexity and thought that it would be too hard to complete. Service users felt it was important to capture the levels of initial confidence as it had a bearing on their motivation and ability to undertake the therapy, as well as being a potential indicator of the level of support they were receiving.

You’d be like ‘no, this is too much for me’. Some of it I’d be trying to do but I think to myself ‘you’ve got to be like a rocket genius to actually get it all right’.

A new item was added: ‘I felt confident when I first saw the CIRCuiTS programme’. Support from the therapist was seen as vital. Clear explanations and support would help to address their lack of confidence, and this led to the generation of another new item: ‘I valued the support my therapist gave me during CIRCuiTS’. Support to work on the computer was discussed, leading to the inclusion of: ‘I needed extra computer support during the therapy sessions’.

Measure format

On many occasions, participants mentioned difficulties they had with attention, concentration, memory and planning (the cognitive targets of CR). Their first impressions were that the self-report measure was too complicated. Participants also expressed their difficulties verbalising their thinking and writing responses. They felt frustrated by open-ended questions which took too long to answer and preferred closed questions with set response options.

Some people’s brains work better than others. It just frustrates me knowing that it took me like ten minutes just to think of the reason. I’ve got it in my head but it’s how I’m going to word it.

SU2

Although this was the majority opinion, one participant found forced choice responses frustrating. A decision was made to keep the closed questions, but to add a comments box at the end of the questionnaire so that participants could answer more fully if they wished to.

It’s a bit of an ultimatum. I just think the English language qualitatively can reflect a plethora of emotions and ideas and limiting a person to two answers that you want to hear is sort of not conducive to a scientific study.

SU5

During the second wave of focus groups, there was consensus amongst service user participants that the measure was comprehensive and of an appropriate length and wording.

Measures

The resulting self-report measures contained 31 items each. The majority (n = 23) were brief statements with a six-point Likert response scale; six were categorical items and two open-ended items. Optional free-text space was provided for each question. The measure was grouped into four domains: (1) Therapy Sessions: eight items on the understanding and use of different components of the CIRCuiTS therapy. (2) The Therapist: five items about understanding the role of the therapist and about levels of engagement and support. (3) The Effects of Therapy: 12 items about the ability to apply skills learnt from CIRCuiTS into everyday life and the personal impact of undertaking and ending the therapy. (4) Using the Computer: four items asking about the ease of use of the computer or tablet and the CIRCuiTS programme. The final two items concern satisfaction with the delivery method and overall satisfaction. The staff survey followed the same format but with items asking about the client rather than themselves. There was consensus that the measures were comprehensive and of an appropriate length and wording. A higher total score indicated better satisfaction.

Design

Using the cognitive remediation satisfaction (CRS) created in Study 1, an online survey was conducted at the end of therapy (WP3, Study 9 – the cognitive remediation trial) to ensure independence from the therapist so that researchers would not know who had received therapy. To test reliability, participants completed the CRS twice with at least a 1-week gap. Data from these repeat ratings are reported to assess reliability and validity. Satisfaction data for the different methods of CR implementation are provided together with the trial data for context.

Recruitment

Potential participants were those recruited to take part in the main trial (see WP3) where they had consented to be randomised to one of the four groups: Intensive CR, Independent CR, Group CR or Treatment as Usual (TAU). They were included if they were aged 16 to 45 years, attending EIS for at least 3 months, had a diagnosis of non-affective psychosis and were able to give informed consent. Exclusion criteria were an inability to communicate in English, an underlying organic or neurological condition or a comorbid diagnosis of learning disability.

Staff participants were NHS healthcare staff working in EIS where the CR trial took take place. Participants were required to give informed consent. Due to involvement in other non-ECLIPSE studies, only three trial sites were able to take part in Study 2a.

Sample size calculation for the differences between treatment arms

For a comparison of two arms with n = 60 participants, there is an estimated 60% power to detect a (standardised) effect size of 0.4 (or larger), which reduces to 39% for a comparison between an arm with n = 60 (Intensive and Group arms) and an arm with n = 25 (Independent therapy) (using Stata sampsi procedure). This revised calculation resulted from lower-than-expected recruitment that was even lower for the staff, and so we decided not to carry out any formal testing of staff satisfaction between arms. Recruitment issues are discussed in WP3.

Procedures

All those receiving CR treatment were asked to complete the satisfaction measure at the end of the 12-week CR intervention in addition to the Working Alliance Inventory (WAI). 33 Staff were asked to complete the satisfaction measure by trial therapists who were in contact with the teams.

Measures

Cognitive remediation satisfaction: The 31-item measure had 23 items, scored on a 1–6 Likert scale with 1 being ‘Strongly disagree’ to 6 being ‘Strongly agree’; 6 categorical items; and 2 open-ended questions. There was also space for participants to comment.

Working Alliance Inventory. 36 The WAI is a 36-item validated measure of the relationship between client and therapist. Items are rated on a 1–7 Likert scale. As the WAI measures therapy engagement, it was used to establish convergent validity for the measure under development. We assumed that there would be a correlation between the two measures as they both measure different aspects of satisfaction with therapy. For these analyses, we removed the therapist items from the overall satisfaction total score to prevent direct overlap.

Psychometric analysis

Total satisfaction scores were calculated after reverse scoring some items so that higher scores indicated higher satisfaction. The overall score was prorated if fewer than 20% of the items scores were missing.

We also used a 19-item satisfaction scale that excluded satisfaction with the therapist, open-ended and categorical items and the overall satisfaction’item. Frequencies were reported for the categorical items not included in the scale.

Reliability: Cronbach’s alpha was calculated for the total satisfaction score to assess internal consistency. Test–retest reliability was explored for each item from the repeated assessments using a weighted κ-statistic and an intraclass correlation calculated for the overall scale. Items were dropped from the scale where they had either low test–retest reliability, low internal consistency or considered to have low face validity.

Concurrent validity: Assessed by correlating CRS and WAI scores.

Service users

One hundred and fifty participants completed the CRS once and 60 (40%) twice. Their mean age was 26.5 years [standard deviation (SD) = 5.99], 38 (25%) were women and 81 (54%) were from minority ethnic communities. The repeat assessment sample was broadly similar with a mean age of 26.6 years, 25% female and 36% white people, 32% black people, 15% Asian and 17% other.

After pro-rating, 144 of 150 participants had a total satisfaction score with a mean of 85.43 (SD 9.04), and a median of 86.00 and a range of 55.00–108.00.

Overall, Cronbach’s alpha was 0.703 (range item alpha: 0.673–0.736), suggesting reasonable reliability. Item Q10 (I learnt how to use a computer/tablet during therapy) and Q12 (I needed extra computer support during the therapy sessions) were dropped as they had negative item-rest correlations. They were also about the use of computers rather than CR and so were judged to have low face validity. This left a 17-item measure and alpha increased to 0.755. With this change, the mean was 77.91 (SD 8.83), median was 79 (72–83) and the range was 48–100.

Q4 (I found some of the tasks or exercises too difficult) also had a negative item-rest correlation but was not judged to have low face validity and was not removed from the scale.

The single overall satisfaction item (separate to the scale) did not have a high correlation with the whole scale score (Pearson’s r = 0.55).

Test–retest analyses

Sixty participants completed the measure twice with an average gap of 9.23 days (SD 6.21). Intraclass correlation (a measure of overall inter-rater agreement) for the satisfaction scale was estimated as 0.77 [95% confidence interval (CI) 0.65 to 0.86] within participants.

Non-scale items

Most participants thought the sessions were the right length (121; 90%) and there were the right number of sessions (129, 89%). Most (99%) reported that they got on well with their therapist and valued their support, thought their therapist was a good teacher, and that they were able to provide feedback to each other during the therapy.

Concurrent validity

One hundred and twenty-nine participants completed both CRS and WAI (214.88; SD = 26.06). Pearson’s correlation between the WAI and CRS scores was estimated as 0.51 (95% CI 0.38 to 0.63).

Staff

Thirty-nine staff completed the CRS staff version. The survey was anonymous; therefore, no demographic information was collected. Scores were prorated if more than 80% items were completed, leaving n = 28. Mean staff scores were 70.51 (SD = 12.83) with a median of 73.72, a range of 42.29–88.00.

Overall Cronbach’s alpha for the satisfaction score was 0.900, indicating very high reliability. There are negative item-rest correlations for Q29 and Q30, but face validity would not suggest dropping them from the scale.

The correlation between the scale score and the overall item score was 0.860.

Half the staff reported that they believed the clients found the length of sessions about right (n = 18, 50%), 17 (47%) did not know and one thought it was too long. Sixteen (43%) staff thought their client found the frequency of sessions about right, 18 (4%) did not know and 3 (3%) thought there were too many sessions per week.

Thirty-three (87%) staff reported that their client seemed to engage well with the therapist, 30 (87%) agreed that the client felt the therapist was a good teacher, 33 (92%) reported their client was able to communicate openly with their therapist and 31 (87%) felt the client was able to value the support the therapist gave.

Design

Focus groups with EIS users (three groups) and EIS staff (two groups) were convened and met twice to discuss the domains of interventions most important to individuals. Each focus group lasted up to 2 hours and was facilitated by two trained researchers. The focus groups were recorded digitally and transcribed.

Participants

We aimed to recruit approximately 18–24 service users (6–8 per group) and 12–16 (6–8 per group) staff to take part in the focus groups. The study was carried out in EIS which were not taking part in the trial (Study 9).

Service users

Service users taking part in the focus groups were using EIS, had a diagnosis of non-affective psychosis and aged 18–35 years. They were excluded if they were not able to give written informed consent or unable to communicate in English.

Staff

All staff participants were working in EIS. There were no exclusion criteria.

Focus groups

We collected information about what they would value in a therapy session. Example questions were the amount of therapist involvement, the importance of the context in which therapy is provided (e.g. individual/group). Through probing, the focus group members teased out the range of key attributes for the modes of CR delivery (Intensive/Independent/Group) employed in the trial (WP3 Study 9). See Report Supplementary Material 2 for the topic guide.

Data analysis

Thematic analysis was used with constant comparison in initial coding and identification of the main themes and subthemes.

Sample characteristics

Twenty-three participants took part in the study: 10 service users and 13 staff members. The majority of service user participants were from ethnic minority groups (80%), half were men and the mean age was 27 years (range 20–35 years). In the staff groups, just over half were male (54%), the majority were white people (85%) and the mean age was 44 years (range 32–55 years).

Thematic analysis

Thematic analysis was used with constant comparison in initial coding and identification of the main themes and subthemes. This was based on Braun and Clarke’s (2006)37 widely used methodology of six steps: (1) becoming familiar with the data, (2) generating initial codes, (3) searching for themes, (4) revising themes, (5) defining themes and (6) writing up.

Both groups preferred intensive, one-to-one sessions. Group delivery was seen by service users as a valuable opportunity for interaction and support. There was consensus that independent delivery would be unlikely to provide sufficient support. Service users made a strong case for a flexible, staged approach in delivery, moving towards independence. Staff and service users showed a preference for regular, hourly sessions, once a week that continued for as long as needed. The consensus across participants was that the venue should be local, for ease of access, in a confidential environment such as a community mental health team. Service users and staff valued the provision of verbal and written information on the CIRCuiTS therapy and its benefits. Both staff and service users wanted a flexible and personalised therapy, with choice and collaboration over the delivery method and therapist, but also emphasised strong engagement, personal and professional therapist skills.

There were both staff and user perception that the level of commitment expected from CR was too onerous. Staff worried that there would be little time left for normal care and were particularly concerned about the impact on risk monitoring. The accessibility of CR was raised as a concern, particularly for those with limited access to computers.

Participants

Participants were staff working in EIS at ECLIPSE sites. Inclusion criteria were the ability to communicate in English and give informed consent.

Analysis

Preliminary coding and categorisation into primary and secondary themes were undertaken independently by two raters using NVivo 12 (QSR International, Warrington, UK). The researchers then reinterrogated and discussed their individual analysis to derive further themes and categories to produce a final framework using inductive themes. 43 This process was carried out for each team separately.

Participants

Forty-two interviews across four teams were carried out pre trial. The teams were labelled A–D and consisted of a mix of nurses, psychiatrists, psychologists, occupational therapists and support workers. Team A (N = 11) was in an urban/suburban location and team participants had an average of 6.7 years in the service. Team B (N = 10) participants were in a suburban location and had an average length of service of 4.8 years. Team C (N = 11), an urban/inner city team, had an average of 3.2 years of service. Team D (N = 11), an urban service, had an average of 4.6 years of service.

Main themes and analysis

The pre-trial data suggest that teams were interested in research and innovation. One driver for the successful implementation of CR was a recognition of the need and potential benefit to service users. Staff consistently acknowledged the clinical need for the treatment of cognitive difficulties in psychosis. These themes were universal across all participating teams.

Democratic decision-making allowed staff to be able to work effectively and was seen as key to successful implementation.

The culture of this team is, I’d say it’s more of a flattened hierarchy than a lot of other teams, as in when they have the clinical meeting everyone attends, you know, regardless of your band and everyone’s input is just as valid.

Teams who felt supported by senior staff and had shared experiences in decision-making were most willing to take on extra responsibility. Staff who felt involved in individual care plans reported a more positive attitude to incorporating new therapies to routine care. On a practical level, teams with a wide variety of professional skills and experience were seen as an advantage when bringing in new therapeutic activities. A new intervention that has an evidence base to back it was also seen as an activity worth trying.

I’m up for that. Yes, I’m always happy to try things. I mean if there’s an evidence to it and it works, then yes, why not, you know, I don’t see anything wrong with that

Absolutely, yes, and everyone’s keen to sort of, you know, add another arrow to the bow in terms of their own professional development.

Staff members who had few opportunities for open communication and who felt less involved in decision-making within their team had more negative attitudes and were more resistant when discussing CR implementation. High staff turnover was also an inhibiting factor in addition to limited resources.

And because there have been so many changes within the team in terms of the way that we work and the people that we work with, I think it might be that that could continue and that the ball would keep on rolling or it might mean I guess that people get fatigued and overwhelmed with too much change happening at one time and get a bit exhausted …

Design

An online survey was conducted with staff involved in the wider trial at the start of recruitment. The survey contained validated measures of implementation leadership and climate, attitudes to EBP and implementation citizenship that have been used in other studies of mental health services in the USA and Europe.

Participants

Staff members from EIS across 6 UK sites (10 teams) were recruited through opportunity sampling. Teams were selected based on their participation in the ECLIPSE trial. All staff members were eligible to participate if they gave informed consent.

Procedure

Website links were sent to participants and team leaders, and after following the URL link, participants provided electronic consent. The survey included questions about an individual’s demographic characteristics (e.g. age) and was followed by four measures that took 15 minutes to complete. This was completed both before the start of the CR interventions (wave 1) and afterwards (wave 2).

Measures

Implementation Leadership:63 The Implementation Leadership Scale (ILS) was developed by the investigative team and has excellent reliability and convergent and discriminant validity and predicts climate. The ILS is a brief 12-item measure with four subscales: proactive leadership (α = 0.95), knowledgeable leadership (α = 0.96), supportive leadership (α = 0.95), perseverant leadership (α = 0.96) and a total score (α = 0.98). All items referred to CR as the EBP of interest.

Implementation Climate Assessment64 assesses individual, team and organisational support for implementation and is assessed by the 18-item Implementation Climate Assessment Scale (ICS) adapted for motivational enhancement therapy (MET) andcognitive–behavioural therapy (CBT). The ICS assesses the degree to which EBPs are expected, supported and rewarded and has excellent internal consistency (Cronbach’s alpha = 0.91) and predicts behavioural health staff outcomes. The six subscales are as follows: focus on EBP (α = 0.91), educational support for EBP (α = 0.84), recognition for EBP (α = 0.88), rewards for EBP (α = 0.81), selection for EBP (α = 0.89) and selection for openness (α = 0.91). All items referred to CR as the EBP of interest.

Evidence-Based Practice Attitude Scale-50 (EBPAS-5065) was adapted for mental health providers. For these analyses, we used subscales from the original 15-item measure with four subscales to assess attitudes towards EBP as a function of perceived appeal of EBP, requirements to use EBP, provider openness and perceived divergence between EBP and usual care. EBPAS total scores (α = 0.76) represent global attitudes towards adopting EBP and subscale alphas range from 0.66 to 0.91. Responses are scored on a 5-point scale (0 = not at all, 4 = to a very great extent); scores are associated with individual provider attributes and organisational characteristics. All items referred to CR as the EBP of interest.

Analyses

Measures were scored according to recommendations by the measure developers. Bivariate correlations were used to examine hypotheses 1–3, and multilevel path analyses accounting for the nested data structure were used to assess hypothesis 4.

We tested whether a leadership style that supports EBP affects implementation climate and subsequent staff attitudes towards CR. Path analyses controlled for the nested data structure (i.e. clinicians nested in teams) and the model tested were whether the effects of leadership on implementation outcomes (i.e. attitudes towards EBP, fidelity) are partially or fully mediated through implementation climate. We used maximum likelihood estimation with robust standard errors (SEs) to account for the nested design. We used the Sobel test for mediation. 66,67

Design

We carried out four focus groups with targeted clinicians from mental health services. We aimed for four to nine participants in each group, which was facilitated by two researchers, digitally recorded and transcribed. Each focus group lasted up to 2 hours.

Procedures

We ran two focus groups with CR trainees and with senior clinicians familiar with CR. To facilitate discussion, both groups were shown the training slides and materials used in face-to-face training.

Topic guide

Following the presentation (1), CR trainee participants were asked to give feedback on the following:

• Will this training be:

  • Effective in achieving the learning aims, objectives and outcomes?

  • Feasible for clinicians (or clinical researchers) working within the NHS or in services abroad to complete?

  • Acceptable to the trainees and their managers?

Following the presentation (2) Training for Trainers, participants were asked to give feedback regarding the following questions:

• Will this model of training be feasible to deliver within the NHS and abroad?

• Will it be:

  • Effective in achieving the learning aims, objectives and outcomes?

  • Feasible for clinical psychologists working within the NHS or in services abroad to complete?

  • Acceptable to the trainees and their managers?

• Can you foresee any potential barriers or challenges?

Participants

Training to deliver CR: Participants were mental health nurses, occupational therapists, assistant psychologists, nursing assistants or other mental health professionals who were experienced in working with people with a diagnosis of schizophrenia. They were identified by NHS managers or members of the research team.

Training for Trainers groups: Participants were clinical or counselling psychologists who were experienced in working with people with a diagnosis of schizophrenia. They were identified by NHS managers or members of the research team.

All participants were provided with written information consent forms for the study. The consent was taken by researchers, who also facilitated the focus groups.

Analysis

Feedback was noted and used in amending planned training programmes. Formal analyses were not necessary.

Trainee focus group

Focus group attendees found it very helpful to have an overview of CIRCuiTS before starting the training itself. An accompanying manual glossary and training slides were recommended and contact details of a trainer in case of questions. They felt that 5 weeks might be too short to complete training and suggested 6–8 weeks instead. Some were worried that if the training was too quick, then they would forget the information. A recap and introduction at the start of each module were felt to be helpful. Trainees also preferred intuitive software so that they could maximise their time on the CR training rather than learning new software.

Focus group attendees believed it was important to incorporate service user views about what is important to them and what topics they would like addressed. Over the years, our team developed and refined this approach and trainees agreed it was important.

Training for Trainers: focus groups feedback

Overall feedback from the Training for Trainers focus group indicated that the model of training is acceptable. To maximise success, they recommended that at least three people from each trust should be included to provide each other with support, as well as gain opportunities for professional development. Participants proposed that training be expanded to include a variety of staff, such as occupational therapists. They also suggested that trainees should not be spoon-fed and that some input from them is required for ownership of the training, such as giving them some flexibility (e.g. a list of exercises to choose from to present/teach). It was also important to think about their own clinical examples. Interaction between trainees was recommended to maintain motivation, perhaps via some assessment and presentations in front of colleagues, which sets a level of expectation. It was recommended that accreditation should require trainees to have a test of specific trainer skills at the end of the course. To maintain momentum and establish CR services, it was suggested that an online discussion forum should continue after the end of the training for support and networking.

Study design

All recruited participants had access to the training and completed measures to evaluate the feasibility, acceptability and potential benefits of the programme.

Measures

1. sociodemographic information;

2. feasibility:

   1. training dates and times were used to calculate the total time required to complete each module;

   2. participants who completed the training within 6 weeks were classified as having completed the programme in the recommended time window;

3. potential benefits;

Participants were required to complete a knowledge questionnaire (minimum of 80% to pass) to show competencies.

1. acceptability.

Participants completed a satisfaction questionnaire.

Lastly, those who had completed the training were offered a face-to-face (which later moved to online because of the COVID-19 pandemic) clinical workshop.

Data analysis

We used descriptive statistics for sample characteristics and to understand training completion times. We carried out logistic regression to investigate whether professional role, education or experience affected the ability to pass the knowledge questionnaire. The potential predictors were as follows: (1) psychologist versus other professional groups, (2) university degree, (3) years practicing since qualification and (4) years working with psychosis.

Thematic analysis of free-text feedback was carried out by two researchers independently who adhered to Braun and Clarke’s (2006) six steps. 37,78 This informed how the programme could be further modified and improved.

Participants

A total of 135 participants from 8 NHS trusts consented to take part in the study. Of these, 70 (67.3 %) completed the training. Out of the 70 completers, 60 (85.7%) also successfully passed the knowledge questionnaire at the end of the training (full completers), leaving 10 (14.3 %) who completed the training but failed the knowledge questionnaire (partial completers), giving some indication of the potential benefits of the training.

Sociodemographic characteristics

See Report Supplementary Material 4.

The majority completing the course had an undergraduate degree, worked in outpatient settings and qualified < 10 years ago with around 7 years’ experience of working with psychosis.

Full completers tended to be younger. The mean average years since qualifying, time in the current NHS Trust and working in psychosis was shortest among full completers who were mainly psychologists and nurses.

Training completion time

The median to complete training for full and partial completers was 34 days, but there was a wide range. Although the median time was 5 weeks (the expected training time), partial completers often took much longer.

Time to complete the knowledge questionnaire was predicted by years since qualifying and years working with psychosis with significantly fewer years on average in the group that passed the knowledge questionnaire. Participants’ profession or education did not significantly affect the ability to pass the knowledge questionnaire (Report Supplementary Material 5).

Training satisfaction

Fifty-six per cent (N = 34) of full completers and 50% (N = 5) of partial completers felt they had made a lot of progress, but more full completers felt they would make a lot of further progress in implementing what they had learnt during training (partial: N = 3; 30%; full: N = 45; 75%). Most full and partial completers reported understanding the programme well (N = 40; 57%). The majority (80%) of partial and full completers said they would recommend the training programme to others.

Thematic analysis

Two researchers conducted a thematic analysis on the free-text responses in the satisfaction questionnaire to examine the different perspectives and identify common themes.

Emerging themes and exemplar quotes.

Bringing cognitive remediation to life

Participants valued applying their learning to real-life experiences. Living with cognitive difficulties from a service user’s perspective was understood more empathically through the three course examples of service user journeys.

It was helpful following the three case studies throughout the course and seeing videos of therapy sessions – this brought CR to life.

Psychologist, inpatient service

Course design and content

The course and modules were considered well designed and easy to follow, having a bite-sized structure, logical flow and clear focus.

Interactive, split into discrete modules, learning clarified at the beginning of each module.

Occupational therapist, specialist service

Technical issues

Difficulties related to using technology fell into two categories: (1) software issues and (2) local technology limitations causing barriers to learning:

… [the] programme is very frustrating to use if internet is slow because you have to load so many pages.

Allied health professional, Early Intervention Service

Course content

Course content emerged as a suggested area for improvement, particularly the final knowledge quiz.

The multiple answers … was a bit demoralising. You might have had all but one point correct, but the answer was marked as wrong. Some of the answers were open to interpretation.

Psychologist, inpatient service

Online versus in-person

Practical challenges of online delivery of the training included difficulty reading or remembering on-screen text, with a preference for more audio, video or hard-copy presentation of information.

I do find online learning more challenging than face-to-face learning because when I read things on a screen, I don’t tend to retain them as easily … I find it easier to watch videos or to have the information narrated as I go along.

Student, community mental health team

Could be supplemented with a face-to-face training session to give the opportunity to ask a qualified therapist any questions around delivering the program.

Allied health professional, community mental health team

Appreciation

This theme conveyed a strong sense of gratitude for the online training.

I am extremely grateful for the opportunity to complete this training … this was a great alternative under the circumstances – COVID-19, cost and practical difficulties of travelling away from home … I think CIRCuiTS is an amazing tool!

Psychologist, inpatient service

Primary objective

To determine the best CR implementation method as assessed by the participant’s stated goals using the Goal Attainment Scale (GAS).

Secondary objectives

To determine the best implementation method to improve cognitive function, social function, self-esteem and negative symptoms.

To determine which CR implementation method is most cost-effective.

Design

This was a multiarm multisite (MAMS) randomised controlled trial with outcome assessors blind to group allocation. Participants were recruited from EIS in 12 mental health trusts, linked to 6 universities, serving urban/rural diverse populations. In each trust, CR was delivered in Independent, Intensive and Group formats. A fourth group, TAU, was included. An interim analysis was planned based on health economic outcomes, with the possibility of arms being closed that showed higher costs and worse clinical outcomes compared with other arms. Full details can be found in Wykes et al. 80

Participants

Members of the clinical team obtained permission from participants to be approached for consent.

Inclusion criteria: (1) attending an EIS and at least 3 months from the onset of the first episode of psychosis with clinical stability as judged by the clinical team; (2) aged between 16 and 45 years; (3) research diagnosis of non-affective psychosis, that is, schizophrenia, schizoaffective or schizophreniform disorder according to the Mini-International Neuropsychiatric Interview 5.0.0;81 (4) ability to give informed consent.

Exclusion criteria were as follows: (1) not able to communicate in English sufficiently to participate in cognitive testing; (2) suffering from an underlying organic condition affecting cognition; (3) have a comorbid diagnosis of intellectual disability.

Cognitive function

At each assessment, we collected The Cambridge Neuropsychological Test Automated Battery (CANTAB83) subtests of Single and 5-Choice Reaction Time; Paired Associated Learning; Spatial Working Memory; One-touch Stockings of Cambridge; Emotion Recognition Task; Rapid Visual Information Processing; Attention Switching Task, the Computerised Wisconsin Card Sorting Task,84 the Rey Auditory Verbal Learning Test85 and the Rey Osterrieth Complex Figure. 86

Symptoms

Clinical Assessment Interview for Negative Symptoms (CAINS). 87

Social function

Time Use Survey88; Social and Occupational Functioning Assessment Scale (SOFAS73).

Self-esteem

The Rosenberg Self-Esteem Scale (RSE). 89

Recruitment (randomisation) over time

Recruitment fell below target resulting in (1) changes to randomisation from group to single participant to avoid long waits for therapy and (2) the interim analysis being undertaken before reaching the intended sample size.

The interim analysis used end-point data from 100 participants: TAU (n = 20), Group (n = 33), Independent (n = 22) and Intensive (n = 25). The Data Monitoring Committee recommended closing the Independent arm based on these data and [together with the National Institute of Health and Care Research (NIHR) panel] closing the TAU arm based on the logistical need to maximise expected future recruitment to the Intensive and Group arms. Revised power calculations for the subsequent feasible target participant numbers used a sample size of 438 participants giving an expected total of 158 and 141 participants to the Group and Intensive arms at the end of the trial. For the contrast of Group versus Intensive, assuming 80% with end-point and follow-up data, with a plausible correlation structure, making no allowance for clustering and retaining the effect size of 0.3 for a comparison of active arms gave 79% power (two-tailed alpha = 0.05). The final analysis plan included assessment of the impact of the interim analysis on treatment effect estimates.

The first participant was consented on 1 June 2016 and randomised on 14 September 2016 and the last on 2 December 2019 and randomised on the 9 January 2020.

Consolidated Standards of Reporting Trials diagram

Figure 4 shows the Consolidated Standards of Reporting Trials (CONSORT) diagram for the trial. Three hundred and seventy-seven participants were randomised and all randomised participants were included in the ITT analyses.

FIGURE 4.

Consolidated Standards of Reporting Trials diagram. Present at FU, present at follow-up; a, GAS; b, SOFAS; c, CAINS total score; d, Composite cognitive score as measured using the CANTAB; Wisconsin Card Sorting Task (WCST); Digit span task and the Rey Auditory Verbal Learning Test; e, Rosenberg Self Esteem Scale; f, The Time Use Survey – total hours in structured activity.

Recruitment by trial arm and site

Women comprised 27.1% (n = 102) of participants and the mean age of the entire sample was 25.7 years (SD = 6.05). Forty-nine per cent (n = 185) of participants were white, 105 (27.9%) and the remaining were black, Asian or of mixed heritage (Report Supplementary Material 8). Two hundred and fifty (66.3%) were unemployed, 52 (13.8%) in full-time education and 35 (9.3%) in full-time employment. Living arrangements consisted of 116 (54.4%) at home with parents, 116 (30.8%) in their own home and the remaining 55 (14.5%) lived in hostels, group homes or temporary accommodation. The mean duration of untreated psychosis was 8.64 months (SD = 18.5).

Hours of therapy attended were as follows: Group arm 14.6 hours (SD = 12.72), Independent arm 8.8 hours (SD = 9.9) and Intensive arm 19.4 hours (SD = 12.8).

Results for primary outcome (Goal Attainment Scale)

There was no evidence of a difference in GAS (weighted T-score) scores between the Group and Intensive arms, or in the Independent arm compared with TAU. There was a statistically significant difference between Group and Intensive arms combined versus TAU at post therapy. All the sensitivity analyses gave results consistent with the primary analysis.

At post therapy, the GAS mean difference between Group versus Intensive was 0.737 (SE = 1.652; p = 0.655; lower CI = –2.50 to upper CI = 3.975); Independent versus TAU was 0.695 (SE = 2.448; p = 0.777; lower CI = –4.104 to upper CI = 5.493); Group + Intensive versus TAU was 5.734 (SE = 1.958; p = 0.003; lower CI = 1.898 to upper CI = 9.571).

At 6 months post therapy, the mean difference between Group versus Intensive was 1.975 (SE = 1.984; p = 0.319; lower CI = –1.913 to upper CI = 5.863); Independent versus TAU was –1.353 (SE = 2.938; p = 0.645; lower CI = –7.112 to upper CI = 4.407); Group + Intensive versus TAU was 2.665 (SE = 2.374; p = 0.262; lower CI = –1.988 to upper CI = 7.319).

Results for secondary outcomes

Please see Report Supplementary Material 9 for a list of outcome measures. For all outcomes, higher scores are interpreted as better except for the CAINS where lower scores are better. These results show no strong evidence of a difference in outcomes for any of the contrasts, except for the global cognition scores shown in bold, for which there appears to be a difference in the Group + Intensive versus TAU contrast of 1.479 (95% CI 0.395 to 2.564; p = 0.008) at post therapy but not at the 6-month follow-up. At post therapy, the mean difference between Group versus Intensive was 0.192 (SE = 0.499; p = 0.699; lower CI = –0.785 to upper CI = 1.170); Independent versus TAU was 1.348 (SE = 0.701; p = 0.054; lower CI = –0.026 to upper CI = 2.722); Group + Intensive versus TAU was 1.479 (SE = 0.553; p = 0.008; lower CI = –0.395 to upper CI = 2.564).

Impact of the COVID-19 pandemic

The pandemic occurred before we were able to complete our assessments. This meant that we were not able to see people face to face as easily, but through creative discussions with our patient and public involvement advisers (PPI), we developed both remote and no-contact methods of collecting data while still being supported by a researcher, either online or on the phone. We also moved our supervised training workshops at the end of online therapist training to online. We were also able to help people complete therapy by switching to some online support. These adaptations were agreed by the sponsor NHS Trust and complied with all the guidelines at each stage of the pandemic. So, despite the pandemic, we were able to keep in contact with most participants and complete therapy, assessments and our therapist training. The main negative effect was that we were unable to repeat focus groups with staff at the end of therapy to assess their views of the CR implementation. The pandemic increased staff workloads and so we decided not to increase them even more by asking for participation in this study.

The importance of PPI

Service users were involved at each stage of the project. They helped us to define our primary outcome for the trial and the design of individual studies and were very important in providing suggestions for encouraging recruitment in all studies as well as helping us to consider ways to keep their engagement. For instance, we sent cards at Christmas to remind people about projects and sent newsletters to show our progress to both service users and staff. Although there are often challenges of working with service users and service user researchers, we experienced none in this study and are very grateful to all our PPI advisers. Some members of our Patient Advisory Group (PAG) also agreed to be authors of our papers after critical revisions.

Challenges of National Health Service research

Working in busy NHS services has several challenges which we have mentioned in other chapters, but we will briefly recap. NHS staff have little time, and despite their enthusiasm for different projects, the lack of time does affect their ability to take part in studies, so this did lead to fewer staff completing the satisfaction measure and completing the online survey for assessing organisational climate. This problem also affected the time set aside for our new therapy training. Some trainees took a long time to complete or did not manage to complete all the modules, and we had anecdotal evidence that professional development time that was supposed to be dedicated to this training was often just a pipe dream. These problems were independent of the pandemic. One NHS Trust early in the programme agreed to take part in the trial but only allowed us to collect data from service user participants and not from staff as they felt it would be too much of a burden. We also had some difficulty acquiring Excess Treatment Costs, and although some of those problems have been alleviated by recent changes, some remain. The main difficulty for psychological treatment trials is that a trained therapist is needed, and they will have time when they are not in face-to-face contact with a participant and this time is not currently accounted for in the calculation of Excess Treatment Costs.

Changes to studies

Apart from adaptations required because of the pandemic, we did have recruitment problems in several studies. For the trial, we changed our randomisation process to utilise all consenting participants quickly. For the future, we advise against large block allocation (15 people) and suggest smaller blocks (3 or 4) or individual randomisation. But despite this and the other challenges, including the ambitious design of some studies, it was a success.

Future research

We intend to continue to use our data to understand how we might improve CR benefits. For instance, items in the satisfaction measure at baseline may affect the outcomes and we may be able to tease out the specific cognitive change measures that predict success. CIRCuiTS is built to improve metacognition, and we have some assessments that will allow us to test whether metacognition is also a mediator of treatment outcome. We found that the effects of therapy were related to the number of sessions received, and we know from previous research that therapists can influence that. We want to investigate this therapist effect further with other measures, such as Working Alliance.