The effectiveness and acceptability of multimedia information when recruiting children and young people to trials: the TRECA meta-analysis of SWATs

Host trials

Host trials used individual or cluster randomisation as deemed practical and appropriate.

Overall

The sample size for each SWAT was ultimately determined, and constrained, by the number of people approached to take part in the host trial.

An example sample size calculation based on the expected baseline recruitment rate of the host trials (i.e. their recruitment rate without the intervention) is provided. Baseline recruitment rates in trials vary greatly, often ranging from 20% to 80%. A baseline rate of 80% was assumed.

We estimated the sample size based on the relative effect of MMI-only (when compared to PIS-only). Further, we assumed 80% power at standard 5% type I error (α rate) to detect the specified effect and we characterised the effect size as the relative increase in recruitment rate (i.e. a relative risk increase).

Assuming the baseline recruitment rate (in the PIS-only arm) was 80%, to detect an increase to 88% in the MMI arm in a single RCT (with 1 : 1 randomisation between MMI and PIS arms), a sample size of 329 per group was needed. This figure was multiplied × 3 to account for the three-arm randomisation in the SWATs (n = 987).

Results from each SWAT were ultimately combined in a meta-analysis. Given that the Phase 2 host trials had variation in the age and health condition of participants, the baseline recruitment rates and the evaluated interventions, it was thought plausible that MMI effectiveness would vary; that is, there would be heterogeneity in the observed effect across trials. Adjusting for this was approximate (particularly as the heterogeneity was unknown before the host trials were undertaken). However, a rule of thumb for the I² statistic of 50% in the meta-analysis was applied, having the effect of doubling the sample size, deriving an overall sample of 1974 across the six SWATs.

Therefore, the six SWATs in TRECA should (on average) each be approaching 329 people, assuming a baseline trial recruitment rate of 80% of those approached.

This calculation has been updated from that included in the initial published protocol, due to reproducibility issues. However, this does not impact on the validity of the results as the sample size of the overall project was driven by the individual SWATs that were undertaken and could not be pre-determined.

Bone Anchored Maxillary Protraction trial hypothetical setting substudy

The primary outcome of this hypothetical setting substudy was the mean Decision-Making Questionnaire (DMQ) total score. We estimated that a sample size of 109 would give 90% power (alpha 0.05) to detect a statistically significant difference between the groups. A sample of 109 participants (54 and 55 in the two trial arms), allowed for 10% of those randomised not being able to complete the questionnaires. We assumed that a mean between-groups difference of 4.5 on the total DMQ score would be meaningful and estimated that the standard deviation (SD) of the pooled scores would be 6.75 (assuming that 95% scores would fall between 4.5 and 31.5). 70

Primary outcome

Secondary outcomes

Quality of decision-making

We also measured the quality of decision-making by potential host trial participants. Children and adolescents were asked to complete a brief decisional scale, adapted from one used within the REFORM trial71 and drawing conceptually on the SURE72,73 and DelibeRATE scales. 74 When a parent/guardian was involved in the participation decision, we also asked them to complete the scale. The scale was adapted to facilitate completion by young children (see Web Documents 1 and 2). We aimed to obtain decision quality scores both from individuals who decided to participate in the host trial and those who declined. In patients who decided to take part, the CYP and/or parent/guardian were asked to complete the decisional scale once the host trial participation documentation had been completed. In patients who declined participation, they were asked to complete the measure in the clinic or have them posted at home or e-mailed, as appropriate.

The older child and parent/family versions of the questionnaire contained the same number of questions, with slight changes in the phrasing of questions. The decisional scale contained nine Likert questions (with five possible answer options) and there were a further three free text questions, which gave participants the opportunity to give further opinions. The younger patient version of the questionnaire was of a similar format. The decisional scale contained three Likert questions (each with five possible answer options) and the further three ‘free text’ questions (see Web Document 1).

The decision-making scales were scored for both the three- and nine-question versions. Answers to each Likert question were allocated a value of 0–4. The values for each question were summed to create an overall score, out of 12 and 36, for the two scale versions. Up to three missing responses were allowed on the nine-question scale. One missing value was allowed on the three-question version of the scale. A total score was calculated by replacing missing values with the mean score from the completed responses given by the participant. Any questionnaires with more than three (older child/parent/family version) or more than one (younger patient version) missing values, were not scored.

Other important information

To assess any potential moderating influences of other variables on the effectiveness of the MMIs, we aimed to obtain data within each host trial of CYP age, gender and deprivation score, according to allocation in the embedded trial and to host trial participation decision.

Baseline data

All participant baseline data have been summarised descriptively by TRECA trial arm. No formal statistical comparisons were undertaken. Continuous measures have been reported as means and standard deviations (normality was checked and if non-normal, medians and interquartile ranges were reported) and categorical data were reported as counts and percentages. Baseline data for the host trials varied due to different trial data collection; all data that were collected have been reported. Baseline data were only available for participants who were randomised into each host trial.

Primary analysis

Secondary analyses

Quality of decision-making

The responses to each question (including the number of missing responses) and the calculated total scores of the decisional questionnaire were summarised descriptively overall and broken down by host trial, TRECA allocation and type of questionnaire (younger patient, adolescent or parent/family).

As a patient and their parent/carer may have both completed a questionnaire, all the data from all three questionnaires were not combined, due to lack of independence. Hence, scores for patients (younger or older) and parents/family questionnaires were analysed separately using a linear regression, with TRECA allocation and host trial status (whether the participant entered the trial) included as covariates. Clustering was accounted for by using the cluster variable as a random effect. Mean difference has been presented with 95% confidence intervals.

If both the younger and older patient questionnaires were used in a host trial, the questionnaire data from the younger and older patient questionnaires were combined. The scores for these were standardised within the trial by subtracting the sample mean score from the observed score and dividing the result by the sample standard deviation, where the ‘sample’ refers to the subset of participants that used that questionnaire. Then the standard scores for all participants were analysed as described above.

The results were compared using a regression model, adjusted for SWAT allocation, and whether the patient consented to participate in the trial. To assess the robustness of the method used to replace the missing values, a sensitivity analysis was conducted, where the analysis was repeated using only the questionnaires in which all nine questions were answered.

The analysis including missing values was repeated but for only those participants who went on to consent to the host trial.

Recruitment

Two random-effects meta-analyses were conducted using the odds ratio for each of the host trials which were calculated in the recruitment primary analysis (PIS-only vs. MMI-only) and the recruitment secondary analysis (MMI and PIS vs. PIS-only). FORCE could not be included in the second meta-analysis as only two arms were used in the trial. The I² statistic was used to assess the heterogeneity between the trials.

Retention

Two random-effects meta-analyses were conducted using the odds ratio for each of the host trials, which were calculated in the retention secondary analyses. The I² statistic was used to assess the heterogeneity between the trials.

Quality of decision-making

The decision-making data from all six SWATs (and not the BAMP substudy) were also combined in four meta-analyses. Mean difference scores from each of the trials were combined in a random-effects meta-analysis.

That you can leave a trial

This piece of information was particularly important to CYP. CYP with trial participation experience emphasised the importance of knowing that leaving the trial was ‘okay’. For example, knowing that they would be free to leave if the treatment caused significant side effects, helped them to say ‘yes’ to a trial. One young person said this information was her only priority. Professionals also ranked this as an important aspect for families; they reported emphasising it during initial discussions with families about trials.

Knowing that you can stop at any time is also good because say you just didn’t want it anymore and you were getting bad side effects and you didn’t like it, then they could stop it just then and there.

(CYP/16, male, 10 years old, experience of being approached about a trial)

If I don’t feel comfortable in the trial anymore or if I didn’t want to take part because I didn’t think it was working for me, could I just stop it – and they were really good at explaining that you could just stop at any time that you wanted and you just have to let them know … so that was one of the only questions that I had really.

(CYP/23, female, 16 years, experience of being approached about a trial)

Therefore, information that participants could leave the trial at any time and without a reason was given prominence within the MMIs.

What is involved if I participate?

Children, young people and parents wanted to understand what trial participation involved. Several with prior experience of being approached about a trial commented that they would have liked to have known more about what was going to be required. In particular, young people wanted detailed information about the frequency of hospital or clinic visits, whether they would need to take time off school, and what the treatments involved, including what they might experience during treatment and beyond. One parent said that participant information sheets never gave details on how participation would affect the day-to-day life of their child.

Some CYP had a pronounced fear of needles and wanted to avoid pictorial or animated portrayals of injections in the MMIs that could be ‘aggressive’ or ‘scary’. This fear was not only expressed by younger participants. CYP also wanted information about the taste of oral medicines, although professionals noted it can be difficult to provide this information in advance for new drug treatments. Parents often spoke of the need to know how participation would affect their child emotionally and whether it would cause ‘burden’ or ‘undue stress’. Some young people and parents talked about the importance of stress, and that it was not often covered in written trial information. Professionals echoed this, agreeing that most families wanted to know about the potential impact of trial participation.

For young people what’s often overlooked is missing out on things like school and social, that’s often not mentioned.

(CYP/18, female, 18 years, experience of being approached about a trial)

How much time is this going to take? That’s by far the biggest anxiety that our families have that they’re going to have to make extra visits to the hospital or extra tests to do, extra diaries to fill in at home. The burden to them is really important and then safety.

(Professional/15, male)

Detailed information about what trial participation would involve was included in the MMIs. The potential for MMIs to include diagrams, photos and animations also make it much easier to convey this detail.

What is the trial testing?

Children and young people with trial experience often described the science behind their condition and treatment and indicated an interest in knowing how the treatment being tested was thought to work. Parents did not always prioritise such information but some wanted to know whether treatments had been tested previously. Professionals talked about the need to explain the science in simple terms, in order to convey the trial rationale. They stressed the need to be clear that professionals not knowing the best treatment for a condition is the rationale for a trial. Therefore, we ensured that this content was prominent in the MMIs.

Risks associated with the trial – and how to inform about them

Most participants talked about the importance of having risks explained in a ‘balanced’ way. Parents specifically wanted to know about possible side effects of treatments. However, they also thought there was sometimes ‘too much information’ about risks, and that some risks were ‘tiny’ but the explanation of them could be ‘scary’ for young people. Young people and some children wanted to know what they were consenting to in terms of risks and side effects but preferred the focus to be on ‘likely’ risks. A list of every ‘potential’ risk could result in being ‘overwhelmed’. Professionals noted that families often wanted to understand risks and safety issues associated with a trial.

Well risks is mostly for, I’d say mostly for the person taking part because it’s going to happen to you if there is any possible risks. Like I know that my drug doesn’t always work because I’ll have a very, very bad flare up and that’s kind of a risk. It can’t always work but I think it’s really important to know that what could happen to you, what you’re getting yourself into. That just makes you feel, I don’t want to do this or I’m willing to do this. I think that’s really important.

(CYP/23, female, 16 years, experience of being approached about a trial)

The risk factor thing is a huge thing, you know, to me some of the information sheets I’ve read I wouldn’t do it because I mean potentially some of those side-effects are tiny but yet the way it’s put across is so scary that you just wouldn’t do it. I know they have to do it, but is there a better way of doing that?

(Parent/24, female, experience about being approached about a paediatric trial)

Therefore, information about the risks were provided within the key section ‘taking part in the trial’.

Possible benefits of trial participation

Parents and young people wanted to know how trial participation might benefit others, as well as any possible personal benefit for themselves or their child. One parent stressed that clinicians should convey this information, because it is ‘important’ to future patients.

But one thing I do like about trials is that I like to know that the trial is going to benefit somebody in the future.

(Parent/36, female, no experience about being approached about a paediatric trial)

That it’s just going to help other people if they’ve got problems.

(CYP/16, male, 10 years, experience of being approached about a trial)

Sometimes families thought that their child may get better treatment in a trial, which led to a concern for some that non-participation, or quitting the trial, could lead to reduced quality of care. This indicated the importance of emphasising that children will receive good care regardless of their decision on trial participation. The issue of possible differences in care quality between trial participants and non-participants, was difficult to address in the MMI template, given the trial specificity of the issue. However, the MMI section on potential benefits of the trial was made prominent, including its potential to inform future treatments to the benefit of other patients.

Confidentiality

Confidentiality was interpreted differently by different participants. Some parents expected that their child’s data would be shared amongst care providers and NHS organisations, including the General Medical Practitioner, to enable continuity of care. However, others thought it was not important for the general practitioner (GP) to be informed of their child’s trial participation. One parent said GPs are ‘copied into everything’, implying an automatic and sometimes unnecessary sharing of information. Young people and parents said that confidentiality was important but they ‘assumed’ it would be observed; therefore, there was not a need to emphasise this information on the MMIs. Professionals did not see information about confidentiality as a priority for families, particularly for CYP, when considering trial participation.

Well you kind of assume your information is confidential anyway so it’s not an issue that you would encounter getting normal health care so you’d assumed the rules wouldn’t be different in a research study (CYP/18) … But it’s nice to be reassured about it.

(CYP/20, male, 20 years, no experience of being approached about a trial)

In the light of these differing interpretations, we decided to include information on confidentiality in the MMIs, but as a topic within the ‘Questions and Answers’ section, rather than giving it prominence.

Who has reviewed or funded the trial

Most children, young people and parents similarly assumed that quality control of trials was undertaken as a matter of course, and that they trusted the organisations running studies to provide scrutiny. However, some young people and parents talked about preferring hospital-led over commerciallyfunded trials, implying that funding sources was of interest to them. Indeed, some professionals thought that families were more willing to join studies being led by doctors they knew. Therefore, there were mixed views on its prominence within the MMIs. Parents also thought that the inclusion of a relevant logo on the MMIs was important:

If I saw that NHS logo, for me, it gives it more credibility I think.

(Parent/60, male, no experience of being approached about a paediatric trial)

Parents further emphasised the need to know that a website their child was accessing was credible, and that a credible, reputable logo would reassure them. Young people were generally less concerned about website credibility but liked the inclusion of a logo. Therefore, given the range of views, we included information on trial scrutiny and funding but did not make this prominent, and we included logos within the MMIs.

Information about payments

Children and young people did not think that information about incentive payments would influence the decision to take part in a trial. However, they worried that such payments could lead others to overlook the risks, and that payment to trial participants could be ‘dodgy’. Professionals noted that incentive payments to participants in UK paediatric trials were rare. In contrast, professionals and young people agreed that for studies involving healthy volunteers undertaking multiple blood tests with no personal benefit, information about payments would be necessary to achieve recruitment. Most participants did not think that providing information about payments for patient participation was advisable. As a compromise, information on payment was included but in the less prominent ‘Questions and Answers’ section.

Colour

Young people thought that colour in multimedia websites was important for engagement but noted a need for balance to ensure a ‘professional’ appearance. Young children (aged 6–11 years) and their parents preferred brighter colours whereas young people preferred more muted colours. Some parents and professionals advised avoiding a particular shade of green, due to associations with illness. Participants liked that the animation characters in the prototype websites represented a range of ethnicities, but there was concern among some parents to ensure that skin tones were realistic. In response to these comments, we changed the skin tone for several of the characters after the second interviews. The colours of the MMIs were also selected to vary according to target age, using bright colours (predominantly orange) for children and more muted colours (predominantly teal) for young people.

Layout

The need for the MMIs to be well-structured and easily navigable was stressed by young people, with main points ‘staring right in your face’ and ‘simple’ headings. They gave examples of websites that they liked, which were clearly laid out and easy to use, despite having a lot of content. They liked the relative ‘formality’ and ‘professionalism’ of the prototype MMIs and wanted the inclusion of characters and other images to balance out the text; this would make them more interesting. We ensured these aspects were included in the MMIs.

Font, characters, quirkiness and details

Young people had preferences for the font to be easy to read. Indeed, they had little tolerance for decorative fonts that were difficult to read and said this could stop them from using a website. Large font size was particularly important for young children; following their comments we further increased the font size after the second round of interviews.

Something else that is really important is the type and size of font because it’s obvious that people don’t think about. If you can’t read the font then there is no point to making a website.

(CYP/18, female, 18 years, experience of being approached about a trial)

In the first round of interviews participants stated that they wanted the drawn characters to resemble people rather than more abstract ‘blobs’. Both CYP preferred characters that they could ‘relate to’ and easily recognise the role a character was representing:

Maybe you need a nurse that suits a teenager because they [nurse characters in prototype website] seem like they’re a children’s one with the bear. You need one that you can relate to.

(CYP/35, female, 16 years, no experience of being approached about a trial)

Some of the characters in the websites were revised to ensure that portrayed roles were easily identifiable.

Overall, CYP were drawn to characters with brighter clothing and details (e.g. red dress with white spots) and we incorporated these details into the final character set. Quirky or slightly unexpected details often received favourable responses. For example, the initial ‘why do we do trials’ animation often prompted families of young children to begin talking together about the animation:

It was funny wasn’t it the guy with the pan on his head, it’s quite funny.

(Parent/41, female, experience of being approached about a paediatric trial)

Funny. That was funny with the pan on his head [laugh].

(CYP/43, male, 9 years, experience of being approached about a trial)

Consequently, we included characters in the MMIs with interesting details and quirky elements within the animations, to increase engagement.

Animations

Participants stressed the need for animations to be concise, with most saying they would watch animations for up to 90 seconds, implying that longer animations would not sustain their attention. Young people were particularly critical of repetition and some parents also found this distracting. Several animations were shown to participants in the first round of data collection. One animation about a health condition, depicted using Lego characters and narrated by a child, was popular particularly with children (9–11 years) and their parents. Hearing the child’s voice engaged children. However, young people often said that this animation style and narration was too ‘young’ for them. Another animation, ‘Making the right decision for me’, was narrated by a young person and used sounds to accompany actions and was popular with most children, young people and parents.

It was so intriguing just like the video and what she was saying all went together really nicely. You could put yourself physically in it and just imagine it (CYP/20, male, 18 years, no experience of being approached about a trial).

However, one group of professionals responded very negatively to this particular animation commenting that it was ‘absolutely horrendous’ (Professional/3). The animation depicted a young person arguing with her parents when deciding whether to participate in research. These professionals felt the tone was negative and the content condescending as it implied that assent was less important than consent.

Despite differing views on this animation, all participants agreed that animations were a suitable means of providing information, regardless of patients’ age. Parents often talked about the benefits of seeing and hearing an animation to convey complex information. Consequently, we developed four trial-generic animations and a fifth trial-specific ‘explainer’ animation, which would succinctly summarise the key features of a particular trial.

I was thinking just then of watching it as an adult that even though it was an animation, you think animations are for children but actually when you look at it, sometimes when you just read something on a page or you’ve just got somebody speaking at you, it doesn’t necessarily go in easily and I would say it doesn’t matter that it’s an animation like that for an adult watching it even, because you tend to think of it as just for children don’t you? But they’ve made it [Nuffield animation] nice and simple (Parent/32, female, no experience of being approached about a paediatric trial).

Narration

Young people were sensitive to the tone and sound of the narrator’s voice. As previously noted, children and parents liked the example animation narrated by a child:

I really like it because of the Lego and the child’s voice. That catches your attention doesn’t it? (Parent/27, male, experience of being approached about a paediatric trial).

However, young people and parents thought that a child’s voiceover would make relating to the animation harder, and several found imperfect pronunciation distracting. Therefore, we decided to use an adult’s voice to narrate the prototype animations. However, when the prototype was shown, most young people reacted negatively to it, commenting that they found it ‘boring’, ‘monotonous’ or ‘robotic’ and that this could stop them from focusing on the animation. Parents did not generally comment on the voiceovers. In order to address young people’s concerns, we asked the narrator to rerecord the narrations with greater tonal variation.

Hearing from other families about their experience of trials

We showed participants a short example video in the round one data collection of a parent and young child talking about their trial participation. 87 Most parents, young people and children responded favourably, saying that they found it ‘reassuring’ to hear from those actually involved in the trial or treatment. Professionals echoed this, saying that children would want to hear from and see other children in trials, rather than parents or other adults.

It needs to be people who are actually in trials so then it will put people of my age and younger under reassurance that other people have had a trial before (CYP/51, female, 15 years, experience of being approached about a trial).

Therefore, we created and included short video clips of trial participants and parents talking about being in a trial for the MMIs.

Interactivity

Participants differed in the value they placed on interactivity in MMIs, and they varied in how they defined interactivity (which could explain the differences in opinion). For some, interactivity meant including games, while others felt it referred to being able to ask questions and receive a response, and others thought it could mean following a pathway through the website. One young patient interpreted websites with multimedia to be interactive because you can ‘hear and see things’ rather than just read them.

Some parents of the youngest children wanted games, interactive characters and music included in the MMIs and thought that without this their child would not engage. Some professionals agreed that children would like interactivity and this could include ‘quizzes’, ‘games’ or ‘being able to follow a character along a journey’. However, most young people felt that interactivity could distract from the purpose of the MMI unless the interactive features were relevant. Furthermore, some parents thought that interactive elements such as blogs and forums could be ‘negative’ and ‘unhelpful’ unless managed carefully.

It goes one way or the other I think, either people just ignore it [interactivity] entirely or it’s the only thing they look at, neither of which are particularly useful (CYP/20, male, 18 years, no experience of being approached about a trial).

On reflection we decided against interactivity in the MMIs, in part due to the demand it would place on host trials to provide responses, and due to the mixed views on its value.

The importance of wording

The careful choice of words was important to prevent scaring children and families. For example, terms such as ‘risk’ and ‘dangerous’ were seen by parents, some young people and professionals as possible sources of worry. Therefore, in the MMIs for younger children we used an ‘Is taking part safe?’ heading rather than the intended ‘Is taking part dangerous?’ heading.

Children and young people also indicated the importance of familiar terms. For example, ‘FAQs’ (i.e. frequently asked questions, included in the prototype MMIs) meant little to CYP, who preferred ‘Q&A’ or ‘Questions and Answers’. Young people often spoke positively about the text in the MMIs and liked its simplicity.

I like how there’s no big fancy medical words [in the website] because I kind of feel like that’s kind of really difficult to understand when you’re trying to read about what’s going to happen and they’ve got these massive words. You kind of feel like, for people my age and younger, it needs to be easy for us to understand so you actually know what’s going to happen to us (CYP/51).

Therefore, we revised the MMI wording after the second round of data collection to ensure that wording was meaningful to CYP.

Participating in the qualitative study

Most CYP were engaged and interested in talking about their experiences of research during the more traditional part of the interview when the researcher asked questions. However, the energy within the room changed when the ranking exercise was used in the first round of interviews. Some children jumped up in excitement at the opportunity to do an activity; often keen to be able to choose the colour of the laminated topic cards. When the researcher explained how the exercise would work, some children were very animated.

For example,

interviewer: ‘ … but what I want to find out from you is whether you would be able to put them in terms of what’s important to you. So if you think something is really important put it in the middle ….’

Yeah target, where you want the hit! (CYP/16, male aged 10 years, experience of being approached about a trial).

However, two members of a young person’s focus group initially looked hesitant about the ranking exercise. They seemed concerned about having to proactively contribute rather than respond to questions. Any concern about the exercise seemed to cease once the young people began ranking cards and they were actively chatting as each card was picked out. The process was described as ‘fun’ and ‘interesting’ and they challenged each other’s initial response in a friendly way, often with laughter or enthusiasm. This enabled group discussion and a joint decision on final rankings. For example, one focus group of young people was particularly interactive:

‘Why have I been invited?’ [reading card] [pause, laugh] (CYP/18, female, aged 18 years, experience of being approached about a trial).

I assume that’s to the research as in why they’ve selected you to actually participate, I suppose (CYP/20, male, 18 years, no experience of being approached about a trial).

Isn’t it kind of obvious … explanatory? (CYP/18 female, aged 18 years).

I guess sometimes it wouldn’t be … especially in randomised trials, it’s not (CYP/20, male, 18 years).

So I think, there’s like, there’s a lot of leeway in that question (CYP/18, female, aged 18 years).

I don’t think it might be that important (CYP/21, female, 15 years).

It’s not particularly important [laugh] (CYP/20, male, 18 years).

Yeah it’s like if I saw that on a website as I was like scrolling through, I think I’d skim it but I wouldn’t like, it wouldn’t be the one that I’m there like, okay this is the most important question (CYP/19, female, 16 years).

I think like sometimes, like especially in specific conditions really of trials, it’s kind of self-explanatory (CYP/18, female, aged 18 years).

‘Red!’ ‘Yes, I’d say red’ (CYP/18, female, aged 18 years).

Yeah (All).

In joint interviews, participants could choose how they wanted to undertake the activity. Some parents watched and then added their thoughts after their child had completed it, which enabled parents’ views to be captured separately and contrasted with children’s responses, without either party’s views being dominant. Parents sometimes talked to their child about the meaning of the topic on the card, helping them decide its placement. The activity also allowed the researcher to ask questions about that topic as the card was placed. In some joint interviews with parents/guardians and children one participant would rank the card and then the other would say where they would rank it and why. This enabled a discussion on any disagreements about a topic’s ranking. A joint interview between a young person (who was in hospital for treatment of a long-term health condition) and accompanied by a grandfather, showed how the young person could challenge the grandfather’s initial ranking:

Do I have to take part? [reading card] (CYP/29, female, 12 years, no experience of being approached about a trial).

I don’t think that’s important (Grandfather, Parent/29, male).

It’s up to you [taking part]. That’s very important, it’s your decision. And then this one, ‘what are the possible benefits of taking part? [reading card] (CYP/29, female, 12 years, no experience of being approached about a trial).

Do you think that goes in the red or would you put that in the blue? (Grandfather, Parent/29, male).

Benefits are the good things about taking part. I don’t know…I think the red, you want to get something out of it don’t you (CYP/29, female, 12 years).

Participants talked to each other more during the ranking exercise, than they did for the initial part of the data collection. This is illustrated by this conversation in the young people’s focus group:

What will happen to the results of the study? [reading card] (CYP/20, male, 18 years, no experience of being approached about a trial).

We like that (All speaking).

We do (CYP/20, male, 18 years).

That goes in the yellow. Everyone agreed? (CYP/18, female, aged 18 years, experience of being approached about a trial).

Do I have to take part? [reading card] (CYP/19, female, 16 years, no experience of being approached about a trial).

‘Yeah that’s important. Yeah’ (All)

I’m going to put that right in the middle. Does everyone agree? (CYP/19, female, 16 years, no experience of being approached about a trial).

Yeah. Yes. (All).

I’m really enjoying this! [laughs] (CYP/18, female, aged 18 years).

The exercise also worked when interviewing individuals, as the discussion continued between the participant and researcher. Most CYP were very animated with clapping and spontaneous chatting about the importance of each card. Children particularly enjoyed reading aloud and then describing where each card should be placed.

Some young people and parents ranked within each category. For example, the ‘somewhat important’ category had two layers or circles of red and participants would sometimes say something was ‘inner red’ or ‘outer red’, which had not been anticipated.

Professionals also enjoyed and engaged with the activity. There was often laughter within groups about who would take on the role of reading out the cards and who would place them. At the end of the exercise, professional participants compared their rankings with other groups; with one professional remarking:

They [the rankings] are quite similar, it’s surprising (Professional/1, female).

One young person commented that the card topics were ‘all valid’ but individually they would ‘not have been able to recall and discuss all of these topics’ without the cards. It also ensured that all topics were consistently covered with each participant so that we could get an overall view on the importance of each topic ahead of developing the MMIs. It is possible that the use of cards constrained the discussion to only cover the information on the cards. However, allowing participants the chance to offer any topics not covered on the cards at the end of the activity also elicited a few additional topics and rankings.

Testing of the original MMIs

Participants were allowed up to 15 minutes (timed by the researcher) to interact with the MMI before testing; participants informed the researcher when they had finished (if sooner). The researcher then asked each participant 10 questions in a fixed order (Table 4) based on key information items in the MMIs; the question order deliberately did not reflect the order of information presentation in the MMIs.

For each question, participants located the required information in the MMI, then explained what it meant using their own words. If participants struggled to find or understand information, the researcher repeated the question and prompted gently for an answer. A 2-minute cut-off per question ensured that participants did not become unnecessarily distressed if they could not find or understand information and ensured consistency across questions and participants. If participants could not locate the information, the researcher directed them to the relevant content and asked them to explain the information.

Finally, participants were asked for their general opinions of the MMI to determine preferences for style, layout and overall design. Interviews were audio-recorded and transcribed verbatim.

Revision and testing of the revised MMIs

Revisions made between user testing rounds were based on scores and participants’ observed behaviour and feedback. After revisions were made, a second user testing round (as above) was completed.

Participants

The prototype MMIs were tested by 26 participants, comprising seven children aged 7–11 years (mean 8.4 years) tested prototype MMI-1 alongside a parent (n = 6; one parent took part with two of their children); six adolescents aged 12–17 years (mean 14.2 years) and seven parents of adolescents tested prototype MMI-2. As such 20 user testing sessions were completed. Fourteen participants were male (53%). Of the parent participants, 10 (76%) had higher education qualifications. All participants spoke English as their first language.

Findings

Participants spent an average of 11.5 minutes (range 5.0–15.0) looking at the MMI before testing and 17.9 minutes (range 6.9–26.5) completing the user testing questions and providing feedback. Child and parent pairs spent longer looking at the MMI (mean 14.1 minutes, range 12.0–14.1) than adolescents (mean 8.6 minutes, range 5.0–13.5) or parents of adolescents (mean 11.5 minutes, range 7.1–14.5).

Ninety-two per cent of requested information was found (see Table 4). However, 7.5% of this was found with difficulty and only 3 of 20 participants (15%) scored a ‘clear round’ in locating all answers without difficulty. In particular, participants had difficulty locating information about trial length (Q1) and the insulin pump (Q4). Comprehension was high: 17 of 20 participants (85%) scored a ‘clear round’ for understanding.

MMI revisions

In response to user testing scores, observed behaviour and feedback, some changes were made. Data suggested the MMI layout was suboptimal; participants felt they had to scroll too much, causing frustration and resulting in participants missing information at the bottom of pages:

I scrolled down too fast I think, there’s lots of this, scroll, scroll, scroll sort of thing [P11, parent of adolescent (14 years)].

You’re having to scroll up and down to see everything and a lot of kids, people won’t bother they’ll just look at the front page [P23, parent of adolescent (12 years)].

Some participants did not notice the two subtabs under ‘About the trial’ (‘Why is it happening?’, ‘What is being tested?’), possibly due to the first subtab opening automatically. The tabs and subtabs also disappeared from the top of the page when scrolling:

There is a problem with the tabs that first one you read and then, even the first time, [child’s name] and I didn’t spot there was a second tab–[P27, parent of two children (7 and 9 years)].

I didn’t see those [tabs], I think maybe because they’re not quite as bright as the tabs at the bottom … You go down don’t you in order to read the bottom, so you lose the top–[P25, parent of adolescent (17 years)].

This produced specific difficulties in locating information about the insulin pen and pump, exacerbated by the video layout on the page; a line of three videos taking up the width of the screen was located underneath information about the pen. Participants assumed this was the end of the web page and did not scroll down for information about the pump:

Because you said about the difference between the pump and the pen, I’d have done a separate page … because it’s confusing because you had videos so you think it stops there [P34, parent of two adolescents (14 and 17 years)].

Following this feedback, edits included changing subtab colour and size, fixing the tabs and subtabs panel to the top of the web page, separating information about the pen and pump onto different subtabs and rearranging the video layout.

Participants highlighted that important information should not be solely within video as connectivity issues may mean they do not play, or participants may prefer text. This issue caused difficulties in finding out the trial length (Q1) as it was only located in video. Furthermore, this video was located at the foot of a web page, but participants considered this information important and expected it to be more prominent. Finally, participants reported being confused and discouraged by some video content, due to language complexity and regional accents:

There was one thing which I found quite hard, it was one she [the clinician] just said stuff a bit fast [P14, child (8 years)].

Some of the videos were quite hard to understand, certainly the professional, the doctor talking [P17, parent of two children (7 and 8 years)].

Consequently, additional text was included to ensure that important information was always available as text; for example, the length of the trial was written within the ‘Taking part’ tab. While it was not possible to refilm video content, comments on complexity were noted for developing the MMIs for the SWATs in Phase 2.

Participants

The revised prototype MMIs were tested with 26 new participants: seven children aged 7–11 years (mean 8.9 years) tested prototype MMI-1 alongside a parent (n = 6; one parent took part with two children); six adolescents aged 12–17 years (mean 15.0 years), and seven parents of adolescents tested MMI-2. As such, a total of 20 user testing sessions were completed. Twelve participants (46%) were male. Of the parent participants, 12 (92%) had higher education qualifications. All participants spoke English as their first language.

Findings

Participants spent an average of 9.2 minutes (range 4.4–14.5) looking at the MMI and 15.4 minutes (range 6.8–31.2) completing user testing questions and providing feedback. Once again, younger children and their parents spent longer looking at the MMI (mean 12.4 minutes, range 8.4–14.5), compared to adolescents (mean 7.3 minutes, range 5.5–11.1) and parents of adolescents (mean 9.1 minutes, range 4.4–12.6). Second round participants spent less time looking at the MMI and answering questions than first round participants.

The MMI revisions appeared successful; 93.0% of information was found without difficulty compared to 84.5% in round one (Table 5). Overall, 14 of 20 participants (70%) scored a ‘clear round’ for finding information. Persistent difficulties with locating information were largely due to participants reaching the 2-minute limit allocated per question. Finally, 99% of the information was understood, and 19 of 20 participants (95%) scored a ‘clear round’ for understanding.

MMI revisions

The revised MMIs in round two performed well and required no further revisions. Previously identified problems with the subtab layout were resolved with round two participants noting the ease of navigation:

I think the website is very good, you know, it flows really well and you can clearly see where everything is [P47, adolescent (15 years)].

The fact that [child’s name] can navigate his way round it, it’s pitched at the right level [P56, parent of child (9 years)].

Further recommendations were identified for consideration when developing the Phase 2 SWAT MMIs, including adding information to the ‘Questions’ page about (1) potential risks and (2) how to contact trial staff with unanswered questions. It was suggested that hyperlinks could be used to provide links to relevant MMI pages.

Feedback from both testing rounds was combined to determine the overall acceptability of the approach. Feedback was largely positive; most participants liked the appearance of the MMIs and found them reassuring and child-friendly:

I like how there’s all the little pictures and, there’s on the videos, they look very easy to look at, there’s nothing complicated [P57, child (9 years)].

I think it’s really good, like the website looks a lot better than even usual websites. It’s very clean, it’s clear navigation as well [P20, adolescent (16 years)].

The videos were generally received as informative, engaging and reassuring, although not all were found relevant, such as the SCIPI trial participant’s brief video description of the pen and pump, which was also written in text:

There are real people speaking, like, which is a good thing ‘cause it kind of shows evidence that people are actually safe [P12, child (10 years)].

The sound bites are short enough in length so you don’t lose concentration [P59, parent of adolescent (15 years)].

There’s a lot of information, and I’m not sure to be honest whether some of these videos were relevant [P11, parent of adolescent (14 years)].

Participants generally liked the different media, although individual preferences varied:

Some of the cartoony videos were kind of fun to watch and they were also very simple and easy to understand and I think, well I do know a lot more now [P57, child (10 years)].

It’s quite well balanced in terms of pictures and I suppose it’s those kind of learning styles stuff, some people are quite visual so pictures work and then you’ve got your detail of the wording [P25, parent of adolescent (17 years)].

Videos is not for me, so I’d just prefer to see it as text so more easily found, so what will happen and it’s there rather than having to click [P32, parent of two adolescents (14 and 17 years)].

Participants liked the diagrams, noting their role as visual cues to information. For example, they remembered the calendar alongside information about clinic visit frequency on the SCIPI trial. However, it is important that the appropriate format is selected; the insulin pen and pump were noted as abstract and that photos would be preferred. However, participants did not have the condition-specific knowledge that most patients with diabetes would have.

Participants found most of the information understandable (with the exception of some video content), although they appreciated that the topic necessitated some technical language. Views on the amount of included information differed: some found it succinct, while others noted the large quantity:

It was the right balance of not patronising but understandable. You feel you’ve come away, well I felt I came away understanding what the thing was [P10, parent of child (9 years)].

There were certainly bits that I thought were really quite complicated but that she remembered, you know like whether it’s safe to take part [P17, parent of two children (7 and 8 years)].

There’s quite a lot of information but it doesn’t feel unmanageable [P43, parent of adolescent (13 years)].

They’re all simple [words] or if they are big words they’ve been explained to what they mean, which is good [P54, adolescent (17 years)].

Researchers’ attitude to patient and parent involvement

The PPAG thought the study team appreciated the importance of PPI, which enabled them to express opinions openly and honestly. They also noted the importance of being treated with the same courtesies as the academic team members to ensure they felt valued. For example, those members of the PPAG who sat on the SAG appreciated the use of alphabetical ordering of meeting attendees on agendas as they felt this was more inclusive than including all academic members followed by PPAG members.

We really felt like part of a team, as we began our work on the study. To me [group co-ordinator]’s leadership of the group was an example of good PPI [Patient and Public Involvement] as she made us feel valued and an equal with a common aim (Parent 01).

PPI group felt very enthusiastic and cohesive, our contributions felt valued, so we were able to input into design and content, as well as the on the SAG [Study Advisory Group]. There was a feeling of egalitarianism so that even on the SAG, our opinions were listened to and taken on board (Parent 02).

Having a ‘PPI champion’ [PPI chairperson], who is a coapplicant of the project, who then champions and supports a wider group of PPI members [is important] (Young person 03).

Motivation of members

While PPAG members were not formally involved in the development of the research question or protocol, all members felt the topic was important and were therefore committed to being involved.

From the outset I was excited to be part of the study. … I knew as soon as [child’s name] was diagnosed that I wanted to try and change things for others in a similar position to us. I wanted to make a direct impact, help to make a change for the better and it was important that this was not just tokenistic.

(Parent 01)

Without doubt, my motivation for joining the group was inspired by the aim of the TRECA project, as I had been complaining quite vocally about how much I disliked how we provide information to patients about research – particularly for children, young people and their families.

(Young person 03)

Members also felt that it was positive that all members of the group were motivated to be involved for the ‘right reasons’, meaning that they were there to contribute to the project. While members acknowledged the personal benefits of being involved, they thought this was secondary to their desire to influence the TRECA study.

Communication

The inclusion of a PPAG co-ordinator who took time to get to know members was highlighted as a strength. The role of the co-ordinator was particularly important in communicating to the group when their feedback had been incorporated, but also when it had not, and why:

I think that the TRECA team in York listened to us and made our opinions and suggestions real and valid. Interestingly, we did not always agree but I still felt that my views were being respected.

(Parent 01)

Communicating roles at the beginning of the project was also viewed as an important step in enabling people to assess whether they have the capacity and capability to get involved:

Receiving all of this information upfront [e.g. Terms of Reference] was really important in helping me to decide whether or not to join the group. Given that I was about to commence the final few months of my undergraduate degree, without this information notably on the frequency of meetings and time commitment, I would have probably said no – and I would have missed out on so many wonderful opportunities to influence the project.

(Young person 03)

The co-ordinator aimed to maintain communication throughout the study, with updates provided between meetings where possible. Members felt that continued contact was important to keep momentum going, particularly during quieter periods in the study.

Varied opportunities

Members appreciated the variety of ways they could be involved, including more traditional PPI (e.g. reviewing participant information) and wider opportunities (e.g. contributing to presentations and publications). In addition, the level at which members could be involved was flexible around individual needs and commitments. Everyone could be involved in different activities, but there was no pressure to do so.

By getting involved in TRECA, I have been able to learn new approaches, methods and opinions which have been an invaluable learning experience. I have also been privileged to be able to contribute to writing and editing different abstracts and papers, which have been incredibly useful, both from the preparation aspect of things, and also from having publications.

(Young person 03)

Expenses

It was important for inconveniences and out-of-pocket expenses to be minimised, for example, by printing meeting documentation for members, booking travel in advance and providing return stamped addressed envelopes when information was requested via post:

The PPI lead [coordinator] was wonderful in always thinking ahead so we never had to worry about booking trains, getting expense forms – everything was anticipated.

(Parent 02)

Logistics

Difficulties encountered during the process were largely logistical. Despite the flexibility of the PPAG with regards to scheduling, it was sometimes not possible to arrange a meeting due to other commitments including work, education, care or health-related factors. Occasionally, teleconferencing was used at meetings to reduce travel demands, but face-to-face meetings were preferred:

Being able to meet face-to-face was a huge benefit as you can react to people’s body language and pick up on thoughts and feelings of different people in the room.

(Parent 01)

Members would also have liked to meet the wider SAG but due to competing work commitments this was difficult to achieve, especially as most members of the SAG joined meetings by phone.

While this is a logistical nightmare, it would have been beneficial for the entire TRECA PPI group to meet in person with the entire TRECA study team for a kick-off meeting at the start of the project.

(Young person 03)

Payment

While transparency regarding expenses and payment was highlighted as a strength, there were some difficulties. This was largely due to the requirement for members to be officially employed on a casual basis by the University of York, to allow cash payment, which members preferred to vouchers. This had not been anticipated, and therefore not communicated to members at the outset. This had some negative impact upon members, as all payments were taxed. This may be particularly problematic for members who may be receiving carer or health-related state benefits, as well as members in full-time employment.

The way we were compensated for our time was complicated and affected my salary from my job. This is something that would in future make me say I’d rather not accept any money for my time.

(Young person 02)

Appropriate membership

As the focus of the TRECA study was improving information for children, young people and their families, it was important to include members from each of these groups. While a younger child representative was invited to the group, there was no capacity to adapt materials and meetings to be suitable for them. It was discussed that when younger children’s input is needed, it may be preferable to involve a separate group of children (supported by their parents if necessary) to appropriately engage them. Members of the group also highlighted the need to be mindful of the inclusion of members with differing levels of experience of PPI and acknowledged that there is a need to approach hard-to-reach groups to ensure diversity.

Researcher reflections on the public involvement process

The TRECA research team thought the involvement of the PPAG was a highly positive addition to the study. The PPAG co-ordinator felt that being involved with the group also allowed them to develop as a researcher by improving their communication with patients and families and generating an understanding of what factors are important to them and why. Nevertheless, engagement was not always straightforward and study-specific difficulties were identified. For example, due to the extensive use of participatory design in Phase 1 of the TRECA study, researchers occasionally found that incorporating feedback was difficult when Phase 1 study participants and PPAG members had differing opinions. Further, due to the embedded SWAT design of TRECA, MMIs for use in the ‘host’ trials were developed based on the existing participant information in the ‘host’ trials, many of which had already been approved by RECs and may have had prior PPI. Consequently, valid feedback from the PPAG could not always be incorporated into the MMIs. Sensitivity and honest dialogue were required to explain to members why it was not always possible to incorporate feedback. In turn, the researchers appreciated PPAG members’ attention to detail and the confidence with which they articulated their feedback. PPAG members were often able to highlight aspects that the research team may not have noticed or considered, especially on visual aspects or subtle but beneficial wording changes. For example, ‘once daily’ was changed to ‘once a day’.

From a practical perspective time and resource constraints and ongoing study developments meant that some of the researchers’ objectives for PPI could not be met. For example, it was not possible to include younger children on the PPAG and consequently the majority of PPAG members were older than the TRECA study target demographic (CYP aged 6–18 years). The research team had also planned to involve members as cofacilitators in focus groups within Phase 1. However, this was not possible mainly because study participants’ preferences were to take part in individual or joint interviews instead of focus groups. There were also some research governance issues related to whether PPAG members would need special permissions (such as an enhanced Disclosure and Barring Service check and a research passport) to assist with interviews. When focus groups did take place, their location or time was not convenient to PPAG members. It was hoped that members would be able to assist with presenting the work at conferences, but while they were invited to attend it was not possible due to conflicting schedules. These resource constraints were experienced despite allocating a dedicated group co-ordinator to liaise with PPAG members (though the co-ordinator also worked on other projects alongside this role).

Primary analysis (MMI-only, PIS-only: intention to treat)

Of the 728 participants in the PIS arm, 484 consented to the trial (66.4%), compared to 475 of the 681 participants in the MMI arm (69.7%). A mixed-effects logistic regression model gave an OR = 1.35 (95% CI 0.76 to 2.40; p = 0.30), suggesting a small increase in recruitment in the MMI arm, but which is not statistically significant.

The baseline characteristics for participants who consented to the FORCE trial are reported in Table 16.

Primary analysis (MMI-only, PIS-only: modified ITT)

In the MMI arm 38 out of 68 participants consented to CHAMP-UK (55.9%), compared to 28 out of 65 (43.1%) in the PIS arm. A logistic regression model produced an OR = 1.67 (95% CI 0.84 to 3.32; p = 0.14), suggesting that those in the MMI arm were more likely to be recruited, but this result was not statistically significant.

The baseline characteristics for participants who consented to the CHAMP-UK trial are reported in Table 17.

Secondary analysis (MMI and PIS, PIS-only: modified ITT)

There were similar recruitment rates for the combined MMI and PIS (30/68; 44.1%) and the PIS-only arms (28/65; 43.1%). A logistic regression model produced an OR = 1.04 (95% CI 0.53 to 2.07; p = 0.90), suggesting a very small increase in the odds of being recruited for those receiving both MMI and PIS, but the result was not statistically significant.

Primary analysis (MMI-only, PIS-only: modified intention to treat)

The recruitment rate was 51.1% in the PIS arm (24/47), compared to 65.3% in the MMI group (32/49). The modified ITT primary analysis gave an OR of 1.80 (95% CI 0.79 to 4.10; p = 0.16), showing that the MMI produced an increase in the recruitment rate, but this did not reach statistical significance.

Baseline characteristics for those participants who consented to the Thermic-3 trial are reported in Table 18.

Secondary analysis (MMI-only, PIS-only: per protocol)

The recruitment rates when looking at per-protocol analysis was 57.1% in the PIS arm (24/42), and 79.4% in the MMI arm (27/34). The per-protocol analysis gave an OR of 2.89 (95 CI% 1.03 to 8.11; p = 0.04), showing that the MMI produced an increase in the rate of recruitment, which reached statistical significance.

Secondary analysis (MMI and PIS, PIS-only: modified ITT)

In the combined MMI and PIS arm the recruitment rate was 43.1% (22/51), and the rate in the PIS-only arm was 51.1% (24/47). The ITT secondary analysis yielded an OR of 0.73 (95% CI 0.33 to 1.61; p = 0.43), showing that the combination of MMI and PIS did not produce any increase in the rate of recruitment.

Secondary analysis (MMI and PIS, PIS-only: per protocol)

When considering only those who received their allocated intervention, the recruitment rates were 53.9% in the MMI and PIS group (21/39) and 57.1% in the PIS-only arm (24/42). The per-protocol analysis had an OR of 0.88 (95% CI 0.36 to 2.10; p = 0.77), showing that the combination of MMI and PIS did not produce any increase in the rate of recruitment.

Primary analysis (MMI-only, PIS-only: modified ITT)

All of the MMI patients were randomised into the BAMP host trial (100.0%), compared to only one out of three in the PIS arm (33.3%).

Secondary analysis (MMI-only, PIS-only: per protocol)

All three MMI patients received their allocated intervention, and were randomised into BAMP (100.0%). Two of the three PIS patients received only PIS, and one of the two was randomised into BAMP (50.0%).

Secondary analysis (MMI and PIS, PIS-only: modified ITT)

Similarly to the primary analysis, all those in the MMI and PIS group were randomised (100.0%), but only one of in the PIS-only group (33.3%).

Secondary analysis (MMI and PIS, PIS-only: per protocol)

Only two of the three PIS-only patients received this intervention, and one was randomised into BAMP (50.0%). Of the four patients allocated to receive PIS + MMI, they all received that intervention, and were all randomised into BAMP (100.0%).

Primary analysis (MMI-only, PIS-only: modified ITT)

There were no patients in the MMI-only arm. Both of those in the PIS-only group were randomised into UKALL-2011 (100.0%).

Secondary analysis (MMI-only, PIS-only: per protocol)

There were no patients in the MMI-only arm. Both of those in the PIS-only group received the correct intervention and randomised into UKALL-2011 (100.0%).

Secondary analysis (MMI and PIS, PIS-only: modified ITT)

Both of the patients who were allocated to PIS-only were randomised into the trial (100.0%), as was the one patient who received the MMI and PIS (100.0%).