Training to enhance user and carer involvement in mental health-care planning: the EQUIP research programme including a cluster RCT

Workstream 1

• Develop a co-produced training intervention to improve service user and carer involvement in care planning for mental health professionals.

• Develop and validate a patient-reported outcome measure (PROM) of service user involvement in care planning, develop an audit tool and assess individual preferences for key aspects of care planning involvement.

Workstream 2

• Evaluate the clinical effectiveness and cost-effectiveness of a co-delivered training package to enhance service user and carer involvement in care planning in secondary care mental health services.

Workstream 3

• Understand the contextual, individual and organisational barriers and facilitators, and examine the processes involved in the development and use of service user- and carer-involved care planning.

Workstream 4

• Disseminate our findings, training intervention materials and patient-mediated resources produced during the programme to all relevant stakeholders using multiple methods.

Methods

Focus groups were conducted with a range of stakeholders (service users, carers and mental health professionals). Focus group interview schedules were informed by the systematic review5 and developed by the research team, with input from academics, clinicians and service users and carers from the research programme and the SUCAG. Schedules covered current perspectives on user involvement in care planning, the process and outcomes of care planning, prior experiences of service user and carer involvement and potential training requirements. Focus groups lasted approximately 60–90 minutes and were undertaken at a range of locations to support participation: at university campuses, trust sites and community locations (e.g. at carers’ centres and participants’ homes). Participants received high-street gift vouchers worth £25 for taking part.

Participants were recruited purposively from Manchester Mental Health and Social Care Trust using a range of methods (trust intranet advertisements, press releases, posters in trust premises and information distributed through carer and service user networks and forums). An initial target to conduct six focus groups was exceeded, with a total of nine focus groups taking place, involving 61 participants across the groups. Four focus groups were conducted with service users and carers (n = 34), four were with health professionals (n = 18) and there was one mixed group with users, carers and professionals (n = 9).

Focus groups were recorded, transcribed and anonymised, then analysed using a qualitative framework approach,25 an acknowledged method of analysing primary qualitative data pertaining to health-care practices with policy relevance. 26 Data from the focus groups were synthesised with further data from individual interviews (study 2) and key findings are reported in study 2.

Methods

Individual qualitative interviews were conducted with service users, carers and mental health professionals.

Data from individual interviews were collected with the intention to combine with study 1 data to prevent missing any issues that participants might be reluctant to raise in a focus group situation. 27 Participants were recruited as per study 1, with further recruitment from Nottinghamshire Healthcare NHS Trust via poster displays, trust newsletters and the intranet, university press releases, oral presentations at service user and carer groups and via service user and carer news bulletins and websites. Interviews lasted approximately 60–90 minutes and were undertaken at a range of locations to support participation: at university campuses, trust sites and community locations (e.g. at carers’ centres and participants’ homes). Participants received high-street gift vouchers worth £25 for taking part. A total of 74 interviews across Manchester (n = 43) and Nottingham (n = 31) were completed (22 service users, 21 carers, 3 user/carers and 28 professionals). A number of interview participants also took part in the focus groups in study 1 (n = 22).

Interviews were recorded, transcribed and anonymised, and analysed in combination with data from study 1 using framework analysis. Framework analysis is commonly used within qualitative health research and allows for both inductive and deductive coding to be incorporated into the analysis process, which means that codes emerging from the data can be combined with important codes that were identified prior to the study. The analysis team comprised service user and carer co-applicants working alongside experienced qualitative researchers for independent coding purposes. The team read their transcripts on multiple occasions to familiarise themselves with the data before starting to code the transcripts. The data and analysis were managed using a Microsoft Excel^® (Microsoft Corporation, Redmond, WA, USA) database comprising participant characteristics, along with a Microsoft Word (Microsoft Corporation, Redmond, WA, USA) document containing emerging themes from each transcript, to provide a data trail. The team met regularly to discuss their own emergent codes, to develop a provisional coding framework, to discuss alternative explanations of interpretations and to ensure that the emerging codes remained grounded in the original data for purposes of validity. This approach to analysis meant that during the constant comparison of new data, the provisional framework was amended and re-shaped to enable the introduction of new codes, and allowed for the removal of other codes that became superfluous over the course of the analysis. The resultant framework contained only those codes agreed on by the whole analysis team. Previous iterations of the coding framework were stored for purposes of transparency and the research team agreed as a whole when data saturation had occurred and no new themes were emerging from the data. In order to further strengthen the validity of the qualitative findings, the final coding framework was presented to the wider study team, which was asked to comment on whether the framework seemed grounded in the data, on any omissions in the framework or any ambiguities.

Health professionals

A clear training need was identified by health professionals, with strong support for the idea of user and carer involvement in that training, and for whole-team training for greater impact. A range of barriers to service user involvement were identified, including individual barriers, such as skills deficits and staff understanding of user-involved care planning, and organisational barriers, including workload/resource pressures, the current key performance indicator/target culture of the NHS and difficulties balancing involvement with risk management.

Service users and carers

In accordance with the professional data, users and carers identified a need for training for all mental health staff, but there was a feeling that senior clinicians might benefit most. There were suggestions that training should prioritise skills in active listening and communication, assertiveness and time for reflection. Participants believed that the training should be mandatory, accredited and updated regularly, and should be co-delivered in order to value the expertise of service users and carers. Potential barriers to effective training were also raised, including staff workload and attitude, lack of accountability and reluctance among service users and carers to be involved as trainers. Issues around care plans also emerged, where care plans were seen as meaningless, not tailored to individuals and not taking into account service users’ and carers’ wishes, experiences or needs. Participants felt that good involvement is facilitated by good relationships with and between staff, effective communication, partnership working and allowing sufficient time during care planning. Barriers to involvement were highlighted as frequent staff changes, workload, lack of knowledge about services by all parties, unhelpful staff attitudes and episodes of severe illness.

Data synthesis

A synthesis day was held with all study applicants and researchers in November 2013.

The two key aims of the synthesis day were to:

1. synthesise the evidence from workstream 1 to develop the training intervention for health professionals

2. develop a ‘train the trainers’ course for service users and carers to co-deliver the training intervention.

Structured summaries of key findings from the systematic review, focus groups and interviews with service users, carers and health professionals, and from the scoping review of training interventions, were distributed by study leads to the group (see Appendix 2). Our synthesis (Figure 2) followed a similar format to formats of previous studies, where we successfully synthesised a variety of data sources. 31–33 A group discussion involved tabulating key evidence statements within a matrix where each row referred to the results from each dataset and each column represented one of the core training components for both the training intervention and the ‘train the trainers’ course (see Appendix 2).

FIGURE 2.

Synthesis of multiple data sources.

Methods

Individual semistructured interviews to explore participants’ views on training acceptability were used. Letters of invitation to take part in the evaluation were sent to all nine trainees approximately 7 months after the completion of training. Participants had to opt in (by e-mailing or phoning). All nine trainees agreed to take part and provided written informed consent. Interviews were digitally recorded and transcribed verbatim. Of the nine participants, there were six service users (four female, two male) and three carers (two female, one male). Transcripts were analysed independently by two members of the trial team using thematic inductive coding of themes emerging to uncover meaning in participants’ accounts of their involvement in the training process. 35 The team met to review themes and reach agreement on the coding of the data and the overarching themes.

Key findings

Course content was rated highly but may benefit from review and/or extension to allow the range of topics and resulting professional training programme to be covered in more depth. Trainees who delivered the training intervention to health professionals were positive about their support experiences, preparedness and personal impacts. Service users and carers wanted to gain new skills and confidence in presentation/facilitation as well as to make a difference to health-care practice. We also found that service users desired different levels of involvement in training facilitation – some wanted to take a more active role than others.

Outputs

A full description of the methods, analysis and results has been published in Fraser et al. 36 (see Appendix 2).

Patient-reported outcome measure

We undertook a systematic review reporting the use, development and/or validation of user- and/or carer-reported outcome measures of involvement in mental health-care planning. 37 The review revealed a lack of care planning measures that are able to meet service user nominated acceptability criteria alongside published standards for psychometric quality. Our aim was to co-develop a PROM to assess user and carer involvement in care planning and the SUCAG suggested that the measure should include the following attributes: suitable for use in the UK; developed via service user and carer collaboration; available in a self-report format for both service users and carers; rated by both service users and carers; based on a social and recovery model; continuous rather than a dichotomous scale; and between 12 and 15 items long.

Audit tool

We developed an audit tool to inform clinicians, services, auditors and researchers who want to quantify levels of user and carer involvement in care planning.

Methods

Potential PROM items were generated from data collected in workstream 1 (studies 1 and 2); 70 candidate items were developed. Face validity was examined with a mixed sample of 16 members of the SUCAG using cognitive interviewing. Nine items were removed because the SUCAG found their language or wording unclear or hard to understand.

The remaining 61 items constituted the nascent scale. Members of the SUCAG were also asked to comment on potential response formats. Consensus was reached for a five-point Likert scale with named anchors of ‘strongly disagree’ and ‘strongly agree’ and a middle neutral value with the label ‘neither agree nor disagree’.

We recruited self-identified service users with severe and enduring mental health problems and carers to complete the emerging PROM, using multiple recruitment strategies (including advertising on NHS trust intranets, newsletter and press releases; posters displayed on trust premises, local trust-based and third-sector study advocates, via Twitter and re-tweeted by local and national mental health charities across the UK and local and national service user and carer forums). Data were collected using online (SelectSurvey version 4.5; ClassApps, Kansas City, MO, USA), postal and face-to-face modalities.

In total, 402 participants completed the 61-item PROM. A randomly selected sample of the 402 were approached to undertake a second completion 4 weeks after baseline, to assess test–retest validity, and 59 test–retest PROMs were completed.

For Rasch analysis, a minimum sample size of 250 allows for > 99% confidence that item calibrations are stable to within ± 0.5 logits, irrespective of scale targeting. This minimum sample size was also deemed sufficient. Prior to statistical analysis, data were double entered and a 5% accuracy check was made. Less than 0.1% errors were detected during the double-entry procedure.

Psychometric and statistical analysis of the data were conducted, involving exploratory factor analysis, Mokken analysis,38 Rasch analysis,39 category threshold analysis, differential item functioning, local dependency, scale reliability, unidimensionality and test–retest reliability. An iterative process of item removal reduced the remaining 61 items to a final 14-item scale (Table 1). The final scale has acceptable scalability (H₀ = 0.69), reliability (α = 0.92), fit to the Rasch model [χ²(70) = 97.25, p = 0.02], and no differential item functioning or locally dependent items. Scores remained stable over the 4-week follow-up period, indicating good test–retest reliability.

For the audit tool (Table 2), we completed a three-round consensus exercise with our SUCAG (n = 16) and reduced the 61 candidate PROM items to form a shorter six-item audit tool. In round 1, items were presented to the SUCAG members (n = 16), who were each asked to select the top 10 PROM items that they felt were most important to include in an audit tool. A total of 27 items were identified by the group. In round 2, these 27 items were discussed, with individuals providing verbal reasoning for their choices. The 27 items were then re-rated for importance based on the group discussion, reducing the pool down to 10 items. In round 3, these 10 items were identified and discussed further until consensus was reached on six audit tool priorities. Psychometric assessment assessed the performance of the six items identified by the SUCAG using a combination of classical test, Mokken and Rasch analyses. Test–retest reliability was calculated using t-tests of interval level scores between baseline and 2- and 4-week follow-up.

Key findings

The 14-item PROM ‘Enhancing the Quality of User and Carer Involvement in Care Planning (EQUIP)’ scale displays excellent psychometric properties and is capable of unidimensional linear measurement. The scale is short, user- and carer-centred and will be of direct benefit to clinicians, services, auditors and researchers wishing to quantify levels of user and carer involvement in care planning (see Table 1). A six-item audit tool was also developed for NHS trusts (see Table 2).

Strengths and limitations

We undertook a stringent, methodological process leading to an initial PROM measure of 61 items, developed in conjunction with service users and carers, and reduced to a 14-item psychometrically validated PROM. Measure length and ease of completion were identified as key user-nominated attributes for PROM acceptability. However, the utility of any measure depends on its validity, reliability, sensitivity and feasibility of completion and a trade-off between these criteria is often necessary. It is possible that some concepts that were originally conceived as important to service users during item generation were not adequately represented by the items retained in the final measure. This accepted, the final measure encompassed a breadth of items that represented a multiplicity of user responses. The attributes selected as most important for the audit tool were chosen by our SUCAG and may lack a diverse or representative sample of individuals using secondary care mental health services.

What this study adds

Current measures, such as those used by the UK CQC, focus on objective indicators of care planning administration rather than those aspects of care planning that service users value most. Our 14-item PROM ‘Enhancing the Quality of User and Carer Involvement in Care Planning (EQUIP)’ addresses this gap. The scale is short, user- and carer-centred and will be of direct benefit to clinicians, services, auditors and researchers wishing to quantify levels of user and carer involvement in care planning.

Outputs

The systematic review reporting the outcome measures of involvement in mental health-care planning has been published in Gibbons et al. 37 (see Appendix 2).

The description of analysis, methods and results of the PROM has been published in Bee et al. 40 (see Appendix 2).

Methods

We used a binary discrete choice experiment with five attributes (i.e. whether or not preferences for care are included in the care plan; whether or not the care plan helps me manage risk; completeness of the information in the care plan; whether or not important decisions are explained to me; and whether or not all important aspects of my life are catered for) and an additional attribute describing the time per person spent on care planning related activities. Each attribute had five levels. Each level described how the service might be described by service users, from completely disagree (the reference level) to completely agree. Table 3 shows an example choice question; 13 choice questions were designed, with each choice question describing two alternative care planning approaches for participants to choose their preferred option.

Service user and carer participants were recruited from adult secondary care mental health services in two NHS trusts, and online via social media. Participant characteristics were summarised with descriptive statistics. The analyses included all participants who completed one or more choice set. For the analysis, the completely disagree level of each attribute was used as the reference level for each of the audit tool attributes. This gives an indication of the value to participants of moving from the worst to an improved level of involvement in the care planning process.

This first step was to summarise and describe the data and inform further analysis. Accordingly, it was hypothesised that higher levels for the audit tool attributes (e.g. agree or completely agree) would be preferred to lower levels (e.g. disagree or completely disagree). In contrast, it was thought that participants would prefer to spend less rather than more time per month on care planning activities. A conditional logit regression using maximum likelihood estimation was used as the starting point for the analysis. 41,42 However, participants’ preferences may vary because of factors that are observed, such as sociodemographic characteristics, type of respondent (service user or carer), design factors (e.g. postal vs. online survey), or because of unobserved factors. To account for this, a random-effects, mixed logistic regression model was used for the main analysis. 43 The analysis treated all attributes as random.

A marginal rate of substitution was calculated using the time spent per month on care planning related activities to estimate the amount of time each person is willing to trade to gain their preferred levels of each audit tool attribute.

Stata^® version 15 (Stata Corp LP, College Station, TX, USA) was used for all the analyses reported here. The mixed logistic regression used the published mixlogit and wtp (delta) user commands developed for Stata. 44,45

Key findings

We recruited 232 participants, of whom 89% completed all choice questions. Most responses were from service users (n = 132/215, 61%), carers (n = 49/215, 23%) and people identifying themselves as both service users and carers (n = 34/215, 16%). Seventeen participants did not report if they were services users or carers. The mixed logit regression results are summarised in Table 4.

The data suggested that preferences were strongest for the attribute ‘my preferences for care are included in the care plan’, with service users prepared to spend 5 hours [95% confidence interval (CI) 1 to 8 hours] and 7 hours (95% CI 2 to 12 hours) per month for improvements compared with the reference level. The least preferred attribute was whether or not the information included in the care plan was complete, with participants willing to spend 1.5 hours (95% CI 0.1 to 3.0 hours) for improvements, compared with the reference level.

Strengths and limitations

Service users and carers were involved in the design of the discrete choice experiment, survey materials and recruitment. Our recruitment methods increased the chance of self-selection into the study and may reduce the representativeness of participants and we used analysis methods to help account for variation in participants’ preferences because of unobserved factors or individual characteristics. In addition, survey respondents were based in the UK and were predominantly female and white British. Thus, the results may not be generalisable within and outside the UK. We used a main effects design that means that important interactions between the study attributes were not accounted for, reducing the robustness of the results.

What this study adds

To our knowledge, this is one of the first large, full-profile, discrete choice experiments conducted with people with serious mental illness. The study results demonstrated that participants preferred care plans that emphasised their involvement by including their preferences, helping them to manage risk, catering for all of the important aspects of their life and by having important decisions explained to them. The completeness of information included in the care plan was the least preferred attribute. The marginal rates of substitution suggested that service users are willing to spend time for improvements to the way in which they are involved in their care planning. Our findings could be used to help services target improvements in care planning to the aspects most important to service users.

Outputs

The full results of the stated preference survey are reported in Appendix 2.

Methods

We conducted a pragmatic cluster trial of the clinical effectiveness and cost-effectiveness of the training intervention to enhance service user- and carer-involved care planning compared with controls in UK NHS community mental health services. The trial used cohort and cross-sectional samples to reduce risks to recruitment and retention. The cluster cohort was recruited at baseline and followed over the 6 months of the trial, whereas the cluster cross-section was recruited at the end of the trial. Consenting service users cared for by each community mental health team (CMHT) were recruited and carers were recruited from consenting service users. Each CMHT was randomised to either intervention (training in care planning) or control (usual-care planning). The CMHTs randomised to intervention received the training package.

We recruited service users and carers from CMHTs between July 2014 and December 2015 from 10 NHS trusts across the UK. Service users were aged ≥ 18 years with a severe mental illness under the care of participating CMHTs. CMHTs screened lists and excluded patients who were not deemed to have capacity to provide fully informed consent or who were too unwell at the time of recruitment. The primary outcome was patient self-reported ‘autonomy support’ measured using the Health Care Climate Questionnaire (HCCQ-10). 46 The HCCQ-10 is a self-report scale based on self-determination theory and measures ‘autonomy support’, defined as patient perceptions of the degree to which they experience their health professionals as supporting choice and ensuring that their behaviour (and behaviour change) is congruent with their values. The scale has 10 items, examples of which include ‘I feel that my mental health-care provider team has provided me with choices and options’ and ‘My mental health-care provider team has worked with me to develop a mental health-care plan’. Items are scored on a seven-point scale from ‘strongly disagree’ to ‘strongly agree’. An overall score is calculated as the mean of the items (expressed out of 100), with a higher score indicating greater ‘autonomy support’.

Secondary outcomes included patient self-reported involvement in decisions (EQUIP PROM);40 satisfaction with services [Verona Service Satisfaction Scale (VSS54)];47 side effects of antipsychotic medication [Glasgow Antipsychotic Side Effects Scale (GASS)];48,49 well-being [Warwick–Edinburgh Mental Wellbeing Scale (WEMWBS)];48 recovery and hope [Developing Recovery Enhancing Environment Measure (DREEM)];50 anxiety and depression [Hospital Anxiety and Depression Scale (HADS)];51 alliance and engagement [California Psychotherapy Alliance Scale (CALPAS)];52 quality of life [World Health Organization Quality of Life (WHOQOL) questionnaire];53 carer satisfaction [Carer and User Expectations of Services (CUES)];54 quality-adjusted life-years (QALYs); and use of services. Measures were completed at baseline (pre training) and at 6 months post training (cohort sample), and at 6 months post training only (cross-sectional sample).

Outcomes for the cross-sectional sample included the HCCQ-1046 and the PROM. 40 Carer measures included the EQUIP PROM40 and WHOQOL53 and carer satisfaction was measured using the Carers’ and Users’ Expectations of Services – Carer version (CUES-C). 54 A summary of outcome measures, including scoring ranges, can be found in Table 5.

To recruit service users into the cluster cohort, clinical studies officers (CSOs) sent out an introductory letter, participant information sheet and consent to contact form. On receipt of the consent to contact form, service users were invited to interview to complete baseline measures. Consenting service users were asked to nominate a carer to be included in the study who, if nominated, was provided with a questionnaire pack (including introductory letter, information sheet, questionnaire, prepaid envelope and consent to contact at 6-month follow-up form).

Service users were recruited to the cluster cross-section by a postal survey, distributed by CSOs to all service users in CMHTs (excluding those already in the cluster trial) recruited to the cluster cohort 6 months following randomisation.

Following recruitment of service users and carers, clusters were allocated randomly to either intervention or usual-care by the clinical trials unit of the Manchester Academic Health Science Centre.

The trial protocol was published55 (the original trial protocol and a summary of amendments can be found in Appendix 3).

Intervention

All consenting CMHTs allocated to the intervention received the training intervention. We asked that at least 80% of staff designated as ‘care co-ordinators’ (i.e. those with a caseload) committed to attending the training. Training was delivered within 6 weeks of service users being recruited into the trial. Clusters allocated to the control condition of ‘usual practice in care planning’ did not have access to the training intervention training.

Sample size and statistical methods

The original sample size calculation was for 24 clusters and 480 patients. Recruitment issues identified early in the trial meant that a decision was made to increase the number of clusters to 36 to ensure sufficient power. The recruitment methods used in EQUIP meant that the proportions of patients responding to the study was variable within sites (from 5% to 30%) and difficult to predict. Therefore, increasing the numbers of clusters to 18 meant that recruitment of more patients than the planned 480 was likely, unless specific measures were taken to reduce the numbers of patients per site (e.g. sampling patients within clusters, or limiting the numbers of patients per cluster). Such additional measures would have proven difficult in practice. Therefore, the decision to increase the number of clusters to 36 led to recruitment of 609 patients. The low-risk and non-invasive nature of the intervention and because the cluster design meant that all patients were exposed whether or not they formally participated in the trial means that it is unlikely that any participants had faced additional risk of harm because of the decision to increase cluster numbers and the subsequent increase in the total sample size.

We wrote to the Research Ethics Committee outlining the recruitment numbers at the end of the trial and the committee raised no issues, but we accept that this issue should have been stated explicitly to the Research Ethics Committee when we raised the cluster numbers.

The Research Ethics Committee’s response was as follows:

The Committee has reviewed your letter regarding the over recruitment into this study. Although they acknowledged that nothing could be done since the study has been declared closed, they pointed out that it was not clear from your letter why the cluster was increased. They stated that whilst the design meant that the numbers in each cluster were difficult to control, the CI [chief investigator] had overall responsibility of the study and should have been aware of the overall recruitment and when a possibility of over recruitment became apparent, the REC [Research Ethics Committee] should have been notified immediately. They concluded that although no actual harm had occurred at this time, they strongly advised that if this happens in the future, you must notify the REC immediately the participant numbers are likely above what had been approved by the REC and submit an amendment.

National Research Ethics Service Committee Northwest Preston (Health Research Authority), 2019, personal communication.

Reproduced with permission from the Research Ethics Committee (25 September 2019)

The primary outcome was the HCCQ-10,46 but data on the use of this scale by people with severe mental illness were limited, so we used a standardised effect to calculate sample size for the cluster trial. Twelve clusters per arm and a mean of 20 service users per cluster (total sample size of 480 participants) would have > 80% power to detect a standardised effect size of 0.4. This assumes an intracluster correlation coefficient (ICC) of 0.05 and an 80% follow-up rate, providing 384 participants with complete data in the analysis. For the cross-sectional study, we required the same number of patients in each cluster.

Analysis was completed using Stata version 13 and followed a statistical analysis plan prepared prior to analysis and approved by the independent Programme Steering Committee. The plan identified the cluster trial as the primary analysis, with the cluster cross-section and combined analyses to be presented as secondary analyses. For the cluster trial, intervention effects were estimated using a linear mixed model with a random intercept for teams. Analysis of outcomes followed intention-to-treat principles with outcome data included for all patients irrespective of receipt of the intervention or completion of care planning during the time scale of the trial. The pattern of missing data was assessed in terms of baseline characteristics of service users to check for differential non-response. Predictors of non-response were included as covariates in each model to satisfy the missing at random assumption of maximum likelihood used in estimating linear mixed models. Missing baseline data for the cohort sample were cluster mean imputed.

Participant flow

Participant flow for the cluster trial and cross-sectional sample is shown in Figure 3. During recruitment, the number of service users per cluster was smaller than estimated in the sample size calculation. We increased the number of clusters from 12 to 18 per arm to ensure sufficient power, and 36 teams were randomised to either the intervention (n = 18) or the usual-care (n = 18) group. There was appreciable uncertainty regarding the effect size for the outcome, which is indicated by the use of a standardised rather than absolute effect size and also the adoption of an ICC of 0.05. Given that there was minimal cost to continuation, it seemed appropriate to continue recruitment of centres rather than termination to protect the power of the trial against the ICC being larger than expected. The Programme Steering Committee, Research Ethics Committee and NIHR approved continued recruitment beyond the target size. In total, 604 service users and 90 carers were recruited to the cluster cohort. Ten out of the 18 CMHTs demonstrated ≥ 80% attendance of care co-ordinators at the training (range 48–100%). Retention at the 6-month follow-up for service users in each CMHT ranged from 76% to 93%, with an overall mean of 82% (n = 497). Retention of carers was limited, ranging from 0% to 100% between clusters, with an overall mean of 56% (n = 50). For the cross-sectional study, 682 service users were recruited [mean number per CMHT was 19.5 service users, standard deviation (SD) 14.0 service users].

FIGURE 3.

The Consolidated Standards of Reporting Trials (CONSORT) flow diagram for cluster randomised controlled trial and cluster cross-section.

Demographics

Combining the cluster trial and cross-sectional samples (n = 1286), 58% of service users were female, 48% were aged between 45 and 64 years, 38% were aged between 25 and 44 years, 87% described themselves as white and only 13% were employed (Table 6). Demographics were broadly similar between the cluster cohort and cross-sectional samples and between intervention and usual care (Tables 6 and 7). Of the 90 carers in the cluster trial, just over half were female and most were white (Table 8).

Cluster trial (service users)

Secondary outcomes (cluster trial)

The results of the secondary outcomes (Table 10) in the cluster trial showed no significant difference between the intervention and usual care at 6 months, except for VSSS-54. The adjusted mean difference indicates that the intervention group had higher (meaning more satisfied) VSSS-54 scores than the usual-care group, which showed a small, statistically significant difference at the 5% level. The 95% CIs are wide and, therefore, the true effect is potentially negligible. With all secondary outcomes, the ICC demonstrates very little variation between CMHTs.

Cost-effectiveness

An economic evaluation was integrated into the clinical trial to assess whether or not the EQUIP training intervention was cost-effective at each of the different levels that decision-makers may be willing to pay in order to gain 1 unit of health benefit.

Methods

Service use data were collected at baseline and the 6-month follow-up for all service users who participated in the cluster trial, as were health status data [EuroQol-5 Dimensions, five-level version (EQ-5D-5L)]. The service use data were collected using a survey adapted for the EQUIP trial. The service use data were combined with published national unit costs to estimate costs. 56,57 The costs of delivering the intervention were estimated from the costs incurred in the trial and the number of people trained. They included the costs of health-care professionals’ time to attend the training and the costs of trainers’ time to deliver the training. The costs of consumables and room hire were also included. The QALYs were estimated by combining the EQ-5D-5L data with UK-specific utility weights using the crosswalk methodology recommended by NICE at the time of the evaluation. 58,59 The analysis used the perspective of the NHS and social care (costs) and service users (QALYs). The time horizon for the primary analysis was the 6-month follow-up point of the trial. Analysis of the economic data was based on intention-to-treat principles, and missing data (owing to incomplete observations and missing follow-up) were imputed using multiple imputation. Regression analyses were used to estimate the net costs and outcomes of the EQUIP training intervention, adjusting for participant sociodemographic characteristics and team cluster (baseline covariates) that may influence costs and QALYs. A generalised linear regression model, with a gamma distribution and log-link, was used to account for the skewed nature of the cost data. An ordinary least squares regression model was used to estimate net QALYs. This required the assumption that the QALY data were normally distributed, which was tested in the sensitivity analysis. The net cost and QALY were bootstrapped to estimate the probability that the EQUIP training intervention was cost-effective at different hypothetical amounts decision-makers may be willing to pay to gain an additional QALY. Sensitivity analysis explored the relative cost-effectiveness of the training intervention if different choices were made about the study methods. These included using different measures of benefit, alternative estimates of the unit cost of the intervention, complete-case analysis and the use of a beta distribution to estimate net QALYs.

Key findings

Eighty-two per cent of service users completed follow-up (497/604). A total of 581 service users (581/604; 96%) completed the EQ-5D-5L at baseline and 537 (89%) had sufficient service use data to estimate baseline costs. At the 6-month follow-up, 442 service users (73%) had complete EQ-5D-5L data and 334 (55%) had complete service use data. Overall, 322 service users (53%) had complete baseline and 6-month follow-up cost, utility and QALY data (see Appendix 3 for further detail). There were no differences between participants with complete and missing data on all but two of the sociodemographic variables. These were whether or not the service user had one or more than one diagnosis (chi-squared test; p = 0.033) and whether or not the participants lived in accommodation that they owned (chi-squared test; p < 0.001). The rate of missing data did not appear to differ between the intervention and usual-care groups. Tables 13 and 14 summarise the EuroQol-5 Dimensions (EQ-5D) data and costs of services used.

There was no indication that the number of people using a service and the average cost per person differed between groups at baseline or follow-up. Wide 95% CIs indicated a high level of variation in the average costs per person.

The net costs and QALYs are shown in Table 15 (primary analysis and sensitivity analyses using the QALY as the measure of health benefit). There was no evidence that the costs or health benefit differed between the training intervention and usual care. The primary analysis, using a willingness-to-pay threshold of £15,000 per QALY, indicated that there was a net saving of £3600 per QALY lost associated with training. Whether or not the savings offset the QALY loss depends on the decision-makers’ willingness to pay to gain 1 QALY. If decision-makers are willing to pay £5000–15,000 to gain 1 QALY then the probability that the training intervention was cost-effective was between 0.29 and 0.49. The probability ranged between 0.09 and 0.75 in the sensitivity analyses using the QALY.

Outputs

A full description of the protocol has been published in Bower et al. 55 (see Appendix 3).

A full description of the methods, analysis and results of the cluster randomised controlled trial (RCT) has been published in Lovell et al. 60 (see Appendix 3).

A full description of the methods, analysis and results of the cost-effectiveness has been submitted.

Delivery and acceptability of the training intervention

Training was delivered to 18 CMHTs in 10 NHS trusts. Owing to service need and the logistics of taking an entire team out of usual work for 2 days, most teams were divided and the training was delivered twice. In total, 350 health professionals (249 female, 101 male) attended the training, ranging from 4 to 39 trainees (mean 19.44 trainees) at each 2-day session. CMHTs teams consisted of 304 care co-ordinators (mainly nurses, occupational therapists and social workers); the remaining 46 were students, support workers and clinical managers. Although psychiatrists were invited, none attended.

We used the Training Acceptability Rating Scale (TARS-161 and TARS-262) to evaluate the acceptability of the training, with six self-report items assessing general acceptability, perceived effectiveness, negative side effects, appropriateness, consistency and social validity. Items were rated on a six-point Likert scale, from ‘strongly disagree’ (score 1) to ‘strongly agree’ (score 6). TARS-2 assesses attendees’ overall impressions of the impact of the teaching process and its outcomes and consists of nine items, rated on a four-point scale from ‘not at all’ (score 0) to ‘a great deal’ (score 3). TARS-2 also includes three open-ended questions asking about the ‘most helpful’ part of the training, any ‘recommended changes’ and ‘any other comments’. Questions 1–6 were summed to calculate an overall acceptability score (range 6–36), and questions 7–15 were summed to calculate a perceived impact score (range 0–27). The overall TARS score was calculated by summing the responses to questions 1–15 (possible range 6–63). 63,64

Analysis

Descriptive statistics were used for the TARS results and open-ended comments were analysed using content analysis. 65

Findings

The results demonstrated high levels of satisfaction/acceptability of the training, with the median overall TARS score as 56 out of 63, median ‘acceptability’ score as 34 out of 36 and median ‘perceived impact’ score as 22 out of 27 (Table 16). Open-ended comments reflected six qualitative themes: the value of the co-production model, time to reflect on practice, delivery preferences, comprehensiveness of content, need to consider organisational context and emotional response (with six people commenting that they did not find the training helpful). The co-production model was key to the training intervention, and 102 participants commented on the value of the service user and carer contribution to the training, in terms of the value of their shared experiences, perspectives and insights, and appreciating them as facilitators. Participants appreciated the opportunity to take ‘time out’ to reflect on practice (n = 50).

All teams were provided with supervision: 2 half-days of 3 hours in the 6 months following the training. Overall, supervision was poorly attended, with < 50% of the original professionals attending. Supervision for those who did attend focused on practice changes that they had made (e.g. including carers in the care plan and writing the service user’s perspective into the care plan) and difficulties encountered when they tried to include some service users in their care plan. Although all of the health professionals were sent the 8-hour optional self-directed learning package, only one individual reported looking at this. Reasons for not looking at the self-directed learning included ‘being too busy’, ‘wanted to look at it but dropped off to do list’, ‘forgot’ and ‘don’t like e-learning’.

Outputs

A full description of methods, analysis and results has been published in Grundy et al. 23 (see Appendix 3).

Strengths and limitations

The study achieved the required sample size in both the cluster trial and the cross-sectional sample and we achieved a high follow-up rate (82%). Our comprehensive outcome assessment increases confidence that our observed lack of effect is robust. Our pragmatic design was intended to improve service user and carer involvement in routine mental health services. Our intervention was delivered in 18 CMHTs across 10 NHS trusts with over 300 care co-ordinators. However, our volunteer sites may not be fully representative of the wider population, and we lacked data by which to compare our participants with the eligible population. A known risk to cluster trial validity is professionals recruiting differently depending on allocation. To reduce risk, we selected patients via existing registers and invited patients before revealing allocation to teams. Practitioners could potentially exclude patients after invitation, although this involved a small proportion of service users and did not differ between arms. Measures of service use and contact with professionals were based on self-report, and such measures may not always agree with other sources, such as service records. Limitations that are specific to the economic evaluation suggest additional uncertainty about the relative cost-effectiveness of the training intervention. Changes in care planning reflect a change in process, which may in turn lead to changes in health and other resource use. However, the 6-month follow-up of the study may be too short for the training to feed through to changing either use of services and costs or overall health and QALYs. There was insufficient evidence to support modelling work to extrapolate from the trial to longer time periods.

The measure of health benefit for the primary analysis was the QALY, which is widely used in mental health trials comparing different types of treatment or patient management. However, it may not be an appropriate measure to detect changes in care planning processes and any indirect impact on subsequent health.

What this study adds

Despite a high level of attendance and satisfaction with the training intervention, no significant effect was found on the primary outcome (perceptions of autonomy support) or other outcomes. There was no evidence that the costs or health benefit differed between the training intervention and usual care. The primary and sensitivity analyses suggested that the probability that the intervention was cost-effective was between 9% and 75% for those analyses using the QALY as the measure of health benefit. Embedding service user and carer involvement in care planning may require considerably greater investment of resources. Benefits for service users may be apparent only over the longer term.

Pre-implementation exploration of the environmental influences

To explore the distal political, economic, policy and clinical environments, we conducted semistructured interviews with key informants involved locally and nationally in mental health policy, practice and research. This was designed to elicit key informants’ perspectives on the state and progress of user involvement in mental health planning and practice. We also carried out a mapping exercise of the contemporary organisational structures and care planning policies. This broader picture was intended to complement analysis undertaken in workstream 1, which was designed to explore current care planning with service users, carers and health professionals with direct involvement in care planning. This was predicated on the notion that key informants would have access to insider policy and practice information, relationships and opinions likely to be relevant in the development and evaluation of the trial being undertaken in workstream 2 to test the effectiveness of a training intervention for mental health professionals (see Brooks et al. 66). We referred to the NPT toolkit framed as a set of propositions to guide the stakeholder interviews (www.normalizationprocess.org/npt-toolkit.aspx; accessed 15 June 2015). We then used NPT as a coding frame67 adapted from the RESTORE71 and EUWISE72 projects to conduct analysis.

Process evaluation

A process evaluation of the training programme delivered as part of the trial was considered appropriate because successful implementation of the training intervention implicated a range of explanatory factors, including the integrity of the intervention and the acceptability of the intervention to both clinicians and service users, which is absent from the traditional outcomes of trials. 73

The process evaluation was designed to explore how far the training intervention had been taken up and implemented into the daily work of the health professionals who attended training and what the consequences of this uptake had been. It aimed to complement and supplement the evidence provided by the cluster trial as recommended by the Medical Research Council (MRC) framework for evaluation of complex interventions. 74

The objectives were to examine how:

• the training for service user- and carer-involved care planning and its principles had an impact on, and was incorporated into, existing routine clinical practices

• care planning training influenced the way in which professionals related to, communicated with and negotiated therapeutic options with users

• care planning training and arrangements had an impact on existing methods of coping, self-care and the development of service user expertise and how it shaped and transformed relationships between service users and professionals

• the impact of training on service users’ perceptions of networks, a sense of control, security and identity compared with previous care planning practices

• service users perceived preparation and support arrangements for engaging with the form and content of the new system of care planning.

Methods and analysis

Semistructured interviews were undertaken with service users (n = 29), carers (n = 4) and mental health professionals (n = 21) from both the usual-care and intervention arms of the RCT at three time points (baseline, 6 months and 12 months post intervention). Participants were eligible for the study if they were service users, carers or professionals from the CMHTs included in the RCT sites. Service users were invited to take part via written invitation including an information sheet and consent to contact form. Eligible staff members were approached by e-mail. Included service users were asked to identify eligible carer participants. Interviews aimed to obtain in-depth data on the experience of utilising and receiving the EQUIP intervention and identify changes to practice over time. Service users involved in the study also completed ‘ego’ network mapping at the beginning of baseline interviews to develop an understanding of the impact of social networks in the management of mental health and to examine the role of care planning over time within those networks (Figure 5).

FIGURE 5.

Example ‘ego’ net. 75

Service users included in the process evaluation were invited to have a care planning meeting observed and to complete a diary over the 12-month follow-up period. Two service users returned completed diaries and one consented to have a care planning meeting observed. The main reason for these low levels of uptake was that most participants reported that insufficient care planning activity took place over the 12-month period to warrant participation in such activities.

Interviews and observations were digitally audio-recorded and transcribed verbatim before being analysed thematically using NVivo software (QSR International, Warrington, UK). Iterative modifications were made to the initial framework through the removal of duplicate codes, recategorisation and the addition of new codes.

Participants who took part in the study were:

• 29 service users from five trial sites – 55% (n = 16) were female and 62% (n = 18) were from the intervention arm of the trial

• four carers from three trial sites – 50% were female (n = 2) and 75% (n = 3) were from the intervention arm of the trial

• 21 professionals from four trial sites – 86% (n = 18) were female and 90% (n = 19) were from the intervention arm of the trial.

Key findings

Outputs

A full discussion of findings from baseline data has been published in Brooks et al. 75 (see Appendix 4) and Brooks et al. 76 (see Appendix 4).

Strengths and limitations

The workstream draws its strengths from the in-depth qualitative approach combined with the longitudinal design and theoretical underpinnings. Such methods allowed for anticipated implementation barriers to be explored in real time. However, there were a number of limitations. Participants included in all of the studies within the workstream self-selected themselves to take part in the study. Therefore, any data produced may not be generalisable to other mental health service users, carers or professionals. Furthermore, in study 7, only those service users who were involved in the trial were eligible to take part. As a result, those service users who were acutely unwell or were considered by professionals to not be suitable for the trial were ineligible. Despite recruitment endeavours to recruit carers, only four were recruited to study 7 (initial target was up to 10) and although carers’ views coalesced with those of service users in the most part, their views may be under-represented in current reporting.

What this study adds

The training was a success in so far as it gained ideological buy-in from those charged with delivering user-centred care planning. However, there was a failure of training to become embedded and normalised in practice. This relates to a lack of organisational readiness to accept change together with an underestimation and lack of investment in the amount of relational work required to make it work. Even the changes that were observed over the 12-month period were too subtle to challenge the existing discontent with care planning as identified at baseline.

Care planning based on shared decision-making that is compatible with user and carer engagement and values is more likely to be achieved if delivered by peer support workers and occupational therapists whose values align more readily than those of existing care co-ordinators or nurses.

Care planning focused on managing mental health based on principles of connecting to others, and activities that are valued in people’s everyday life, is likely to be a more effective and acceptable replacement for traditional care planning than modifying through training professional attitudes to user participation.

User and carer definition of care planning

A definition of care planning using a structured exercise was developed by the SUCAG:

A good care plan is one over which service users have a sense of ownership. It is interactive, person-centred and accessible in an understandable form. It provides recovery-focused individually relevant information, addresses current and future needs and sets out objectives that are achievable, time-bounded and regularly reviewed. All decisions are justifiable and health professionals are accountable for any identified support that is not received.

EQUIP Advisory Group, 2015.

Reproduced with permission from the Advisory Group (2015)

This definition has been incorporated into the ‘Willing Adopters’ training and into the EQUIP cards.

Animations

Findings from the published work undertaken in workstream 1 were used to develop two animations to convey key messages to a lay audience. The two animations were created by a professional animator detailing the 10 Cs of care planning (http://research.bmh.manchester.ac.uk/equip/10Cs;16 accessed 31 August 2019) and carers’ perspectives on care planning (http://research.bmh.manchester.ac.uk/equip/mentalhealthcareplanning;17 accessed 31 August 2019).

Enhancing the Quality of User and Carer Involvement in Care Planning cards

Using a structured session with our SUCAG, we defined five key questions that service users and carers could ask their care team about their care plan. The SUCAG felt that the most effective way of presenting this material was to produce a small pocket-sized fold-out card that contained the five questions and the definition of care planning. The cards are intended to be handy tools for service users and carers to refer to before and during a care planning meeting. Cards were designed utilising the same illustrations as in the animations for continuity. Details of how cards were disseminated are given in Dissemination of materials. Images of the cards can be found in Appendix 5.

Coventry and Warwickshire Partnership NHS Trust

There has been commitment from all levels including senior management since the training intervention was first delivered. The ‘Willing Adopters’ workshop was delivered to the trust in October 2016. Day 1 focused on orientation and implementation of the EQUIP programme for senior managers, health professionals and service user and carers. Days 2 and 3 delivered a hybrid of the training intervention and ‘train the trainers’ involving six service users and carers and six health professionals. However, neither health professionals nor service users and carers felt confident to deliver the training alone and we jointly delivered the 2-day training to two CMHTs. The trust training team has since delivered five training sessions to 82 staff working in CMHTs. The trust is using the EQUIP care planning audit tool to measure service user involvement in care planning. The trust planned to roll out the training to all older adults and learning disability teams in early 2018. To ensure further embedding, a session on enhanced user and carer involvement in care planning is to be incorporated into the trust staff induction programme. Ongoing support by the EQUIP team will continue to be provided via 2- to 3-monthly teleconferences.

South West Yorkshire Partnership Foundation Trust

A 2-day ‘Willing Adopters’ package was delivered to seven senior staff at the trust in March 2017. The trust was keen to embed the training via the Recovery College to maximise service user and carer input. The Recovery College involving nine health professionals and six service users received the 2-day training package in September 2017. A further implementation half-day was delivered in October 2017; plans are now afoot to roll out the training to both community and inpatient staff and intend to use the audit tool. The EQUIP team will provide continued support to the Recovery College.

Pennine Care NHS Foundation Trust

The ‘Willing Adopters’ package was delivered to the trust in July 2017. The trust requested that the training package be condensed to a 1-day version to minimise service disruption. Although committed, it was keen that the EQUIP team deliver the training, which has been delivered seven times to health professionals (with in excess of 250 staff attending) from a wide range of services, including community mental health, early intervention, inpatient, older people’s, learning disability and children and young people’s services. The trust has made significant progress with embedding service user- and carer-involved planning with considerable buy-in from senior management with away days with service users and carers focused on improving care planning. The trust developed and implemented a new care planning template.

Lincolnshire Partnership NHS Foundation Trust

The ‘Willing Adopters’ package was delivered to seven mental health staff, two service users and two carers at Lincolnshire Partnership NHS Foundation Trust. The trust adapted the training to include a local perspective including results from a community survey, and developed examples of good care plans and risk plans, which evidenced service user involvement. The trust has plans to roll out training to the CMHTs during 2018 and use peer workers to support care/safety planning within the teams.

Leicestershire Partnership NHS Trust

The ‘Willing Adopters’ package was delivered on 14 and 15 December 2017. The learning from EQUIP has been fed into a 1-year project on care planning and recovery, and this will result in a new approach for the trust, including training, care planning and tools for recovery.

Greater Manchester Mental Health NHS Foundation Trust

Manchester Mental Health Trust and Greater Manchester West NHS Foundation Trust merged, and are now the Greater Manchester Mental Health NHS Foundation Trust. We delivered the training to the usual-care group and two further CMHT teams, with a further training session delivered in February 2018. The trust distributed the EQUIP cards via the NIHR CRN CSOs and for a 1-month period in 2017 the trust included a card with every care planning appointment letter (n = 2100). Cards were also placed within clinic waiting rooms, handed out during appointments with service users by CSOs and made available to CMHT staff to hand out to their clients.

The trust has included ‘user-involved care planning’ in its quality account, incorporated it into its service user engagement plan, developed a new template for care planning, and are funding a band 4 role specifically designed for an individual with lived experience and to deliver and embed care planning training. We will deliver the training to this newly appointed person and clinical leads in October and will offer further bespoke training to the ‘trainer’.

2Gether NHS Foundation Trust

The 2Gether NHS Foundation Trust was not part of the trial but contacted us following our dissemination and intends to use our materials and training.

Digital dissemination

Considerable effort was focused on digitally disseminating our findings. The digital campaign was launched on 13 July 2017 following a live Reddit Ask Me Anything (AMA) session, hosted by the EQUIP team.

Reddit Ask Me Anything

The live session was focused on the findings from workstream 1 (Grundy et al. ,16 Cree et al. 17 and Bee et al. 40). The session generated a lot of interest, with an audience of 38,500 people and 7283 positive votes (i.e. people liked this work). The total reach of the AMA was 8.88 million people (data provided by Reddit). 78

The National Institute for Health Research ‘Let’s Get Digital’ competition

Both animations were submitted to the NIHR Let’s Get Digital competition, with the carers’ perspectives animation shortlisted and subsequently named as the winner of the video category. Following this success, the animation was presented to Google (Google Inc., Mountain View, CA, USA) at an event in London in November 2017.

Twitter and Facebook

A Twitter account (@Care_Plan) was created in 2013 and was used on an ad hoc basis to share work throughout the programme. Following the Reddit AMA and shortlisting for the NIHR award, a systematic approach to sharing the animations was employed to generate interest and increase the number of views.

Data from a search of keywords tweeted including ‘user involvement’, ‘carer involvement’, ‘mental health, ‘care planning’ and ‘service delivery’ were provided by the University of Manchester library team, which allowed key Twitter accounts to be identified. Accounts were systematically selected based on relevance and number of followers. Direct contact with identified accounts involved tweets from the EQUIP account containing the EQUIP definition and links to the animations.

With support from the University of Manchester Digital Marketing Team in August 2017, we advertised via Twitter and Facebook to increase exposure to the two animations. Viewing metrics were monitored throughout the digital campaign to gauge reach and success of activity. Subsequent adjustments were made to the advertising strategy in real time based on these metrics. The results showed that the combined number of views of both animations had exceeded 335,000, with viewer retention of 6.9%, which exceeds the current average duration of video viewing hosted on Facebook (5.5%). Final analytics revealed worldwide views in Europe, North America, Africa, Asia and Australasia, an audience age range between 13 and ≥ 65 years and a gender split, with female viewers accounting for > 75% of the total audience.

Key messages video

We consulted our SUCAG to determine how it would like to disseminate the key messages from the programme, and, following discussion, it was decided that a short animation would be the preferred method to communicate findings; it is available on YouTube (YouTube, LLC, San Bruno, CA, USA; www.youtube.com) and the EQUIP website: http://research.bmh.manchester.ac.uk/equip/mainfindings (accessed 31 August 2019).

Developing and evaluating our training intervention

Using evidence gathered from literature reviews and our own extensive work with stakeholders, we developed our training intervention in collaboration with service users, carers and NHS representatives. Our training embraced a bottom-up approach to professional behaviour change, and offered 18 hours of facilitated training to multidisciplinary teams (12 hours of direct learning, 6 hours of follow-up supervision), plus an additional 8 hours of optional self-directed learning. Our training was co-delivered, by service users and carers, to 350 mental health professionals (304 care co-ordinators) working in CMHTs in 10 NHS trusts in England. We evaluated the impact of the training on patient outcomes by recruiting 604 service users and 90 carers to a cluster trial, and 682 service users to a cross-sectional study. Our training was well attended and positively evaluated by staff but had no significant effect on our patient-rated primary outcome measure (the HCCQ-10) 6 months after training.

Our inability to demonstrate benefits from our training intervention demands exploration. We evaluated an intervention that was derived from published evidence and designed specifically to overcome known barriers to collaborative care planning. The recent Making Good Decisions in Collaboration (MAGIC) programme79 has identified many front-line challenges to embedding shared decision-making into UK NHS secondary health-care settings, including the misplaced confidence of staff, a perceived lack of support for shared decision-making and inaccurate assumptions about patients’ preferences. Importantly, the MAGIC study excludes mental health settings, but its findings resonate closely with those of our own systematic review, which has similarly identified barriers relating to the readiness, skills and confidence of mental health professionals to engage service users and carers in discussions about their care. 5 Extensive qualitative work undertaken in phase 1 of the EQUIP programme subsequently confirmed the need to train front-line staff in partnership working and identified this as a training priority upheld by all stakeholder groups.

Our training intervention was designed and delivered in line with stakeholder preferences and took account of current best evidence emphasising the importance of interprofessional learning, interactive group work and educational outreach. We set out to train whole teams and included exercises and role plays that challenged embedded attitudes, reviewed current practice, built a shared understanding of service user involvement and improved skills in sharing decision-making with patients. Training attendance was high, but there were important limits. We stipulated an attendance rate of 80% per team, and 10 out of 18 teams achieved this goal (range 48–100%). However, no psychiatrists attended, which may ultimately have limited the impact of our intervention.

Our training was evaluated extremely positively by attending professionals, with co-delivery by service users and carers being the most valued aspect of the intervention. The co-production and co-delivery of training in partnership with service users and carers is becoming increasingly common in mental health and presents a unique opportunity to promote the ethos of partnership working at all levels of a health-care system.

Although we sought qualitative feedback on the relevance and impact of our training from staff, we did not assess the ongoing enactment of enhanced beliefs and skills. Anecdotal feedback collected during supervisory sessions and data collected during the nested process evaluation suggested that pockets of practice improvement occurred, but direct observations and/or analysis of real-time care planning discussions are needed to independently verify this behaviour change. Professionals involved in the process evaluation also felt that such changes may be too subtle for services users to notice within the follow-up period. Without such data, we are unable to determine definitively whether our training failed to translate into practice change, or whether practice changes did occur but failed to demonstrably affect trial outcomes.

Summary Training Acceptability Rating Scale analysis