Appropriate design and reporting of superiority, equivalence and non-inferiority clinical trials incorporating a benefit risk assessment: the BRAINS study including expert workshop

Article history

This issue of the Health Technology Assessment journal series contains a project commissioned by the MRC–NIHR Methodology Research Programme (MRP). MRP aims to improve efficiency, quality and impact across the entire spectrum of biomedical and health- related research. In addition to the MRC and NIHR funding partners, MRP takes into account the needs of other stakeholders including the devolved administrations, industry R&D, and regulatory/advisory agencies and other public bodies. MRP supports investigator-led methodology research from across the UK that maximises benefits for researchers, patients and the general population – improving the methods available to ensure health research, decisions and policy are built on the best possible evidence.

To improve availability and uptake of methodological innovation, MRC and NIHR jointly supported a series of workshops to develop guidance in specified areas of methodological controversy or uncertainty (Methodology State-of-the-Art Workshop Programme).

Workshops were commissioned by open calls for applications led by UK-based researchers. Workshop outputs are incorporated into this report, and MRC and NIHR endorse the methodological recommendations as state-of-the-art guidance at time of publication.

The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Copyright statement

Copyright © 2023 Totton et al. This work was produced by Totton et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This is an Open Access publication distributed under the terms of the Creative Commons Attribution CC BY 4.0 licence, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. See: https://creativecommons.org/licenses/by/4.0/. For attribution the title, original author(s), the publication source – NIHR Journals Library, and the DOI of the publication must be cited.

2023 Totton et al.

Background

Randomised controlled trials (RCTs) are an essential methodology for an unbiased assessment of different health technologies in a real-world setting. RCT results are used to inform patient treatment decisions. In National Institute for Health and Care Research (NIHR) Health Technology Assessment (HTA) programme-funded RCTs, the comparison is often between a new health technology and an existing one that is already part of regular practice. In these cases, the benefit (i.e. superiority) of the new health technology may be not a health outcome, but instead an improvement in the safety profile, cost or convenience of delivery. However, it remains important that the RCT demonstrates that it is similar or at least non-inferior to the existing health technology for the desired health outcome to ensure that it would be acceptable for implementation in regular practice (assuming its superiority on another outcome).

Instead of the RCT being designed on a superiority basis, the trial may have an equivalence or non-inferiority design. Equivalence implies that the two health technologies are ‘equal’ to each other with respect to the primary outcome, whereas non-inferiority implies that the ‘new intervention is “not unacceptably worse” than the intervention used as the control’. 1 In this report, key factors that influence the trial design are outlined to aid the decision of which design is most appropriate.

Certain key elements of the trial, such as the sample size, depend on the trial objective chosen (i.e. superiority, equivalence or non-inferiority), which is usually based solely on the primary outcome. However, this singular focus does not reflect the complexity of policy decisions when information on multiple outcomes is of interest. For example, as well as the clinical effectiveness of a new health technology, there is a need to compare the effect of the technology on safety, cost(-effectiveness) and convenience with that of the existing technology when deciding if the former should be used in regular practice in the NHS.

There are existing statistical methodologies to account for the issue of more than one outcome of interest, such as the use of multiple primary outcomes,2 composite outcomes3 or multivariate analysis (i.e. where two or more outcomes are analysed simultaneously). These methods may be appropriate to use when multiple outcomes are expected to be improved by the treatment or when there is a clear ranking of health states in terms of their importance. However, they still require implicit judgement about how the outcomes relate and careful interpretation when the outcomes are in competing directions. For example, if a new health technology is clinically better than current practice, but results in an increase in the number or severity of adverse events, a method to identify the overall best health technology is required.

Benefit–risk (B–R) approaches are a group of methodologies that can be used in such situations to evaluate the net clinical benefit and allow a direct comparison of competing health technologies. B–R approaches are already used in the regulatory setting, where it is important that regulators can evaluate the benefits of a drug against its harms. In addition, the B–R trade-off is an important element of portfolio management decisions. 4 A European consortium B–R methodology project [named Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consortium (PROTECT)], which had a goal to improve the evaluation of benefit and risks related to medicines, is now complete. 5 However, despite their utility, the methods have yet to be fully applied to publicly funded RCTs, such as NIHR HTA trials.

Aims and objectives

The overall aim of this project is to provide consensus-driven recommendations for the inclusion of B–R approaches in the design of NIHR HTA programme-funded RCTs.

The objectives involved in achieving this aim are to:

• describe the factors affecting selection of the appropriate design for a trial (i.e. superiority, non-inferiority or equivalence)

• evaluate when it may be appropriate to apply B–R methods

• explore the different B–R methods that could be included within NIHR HTA RCTs

• define the information related to B–R that should be included in trial proposals, protocols and reports.

Scope of the report

This report discusses trial design in relation to superiority, non-inferiority or equivalence, but does not discuss the differences in other trial design features, such as crossover trials, network analyses or indirect comparisons. The focus of this report is predominantly on the parallel-group design in the first instance; in the future, work could be extended to assess the appropriateness of other designs.

In addition, this report considered clinical trials in which outcomes are assessed at the individual level. Again, this work could be extended in the future to contain population-level outcomes, for example including studies that aim to estimate prevalence effects.

Development of the recommendations

The Benefit–Risk Assessment to Inform Non-Inferiority and Superiority Study Design (BRAINS) project consisted of three key research stages, leading to the development of the recommendations. The stages were as follows.

Definitions

This chapter will further define the different types of trial objective (superiority, non-inferiority and equivalence) mentioned in Chapter 1 (Figure 1 presents a visual representation), before providing a list of factors that affect the decision of which design is the most appropriate for a trial. This was the item in the survey that most researchers said that they wanted guidance on (see Appendix 1).

FIGURE 1.

Demonstration of intervention effects for a health technology, with 95% confidence intervals, in relation to superiority, non-inferiority and equivalence. Adapted from Hahn et al. 8 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution Non-Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt and build upon this work, for non-commercial use, provided the original work is properly cited. See: https://creativecommons.org/licenses/by-nc/3.0/.

Copyright © 2023 Totton et al.

Outcome measure selection

Selecting the outcome measures to be used in a trial is often based on the clinical decision of which aspects of health and well-being the health technology will affect. The primary outcome should be the most important outcome related to the health technology and the most important to patients. In addition, the Core Outcome Measures in Effectiveness Trials Initiative12 collates Core Outcome Sets to inform the decision of which key outcomes to include depending on the disease area and/or health technology. It is important to gather a range of opinions from various stakeholders, including patients, when selecting important outcomes for the trial to ensure that their considerations are included. 13

Factors for selecting a trial design

Through the workshop, a list of factors that inform the selection of the trial design was compiled that aims to assist decision-making when a study is being planned. These 19 items have been summarised in Table 1 and further elaboration is provided in the subsequent section. This list of factors will be illustrated further through worked examples.

There is a degree of overlap between some of the factors mentioned. However, based on the workshop, where the importance of items was voted on, the factors remain separated to reiterate or provide nuance to points when necessary. Those items in italics in the table represent factors that would necessitate choosing the given trial design. Other factors will influence the trial design, but do not dictate it.

This list mirrors both the population, intervention, comparison, outcome (PICO) format, which is used to appropriately frame research questions,14 and the estimand framework, which was an addendum to the International Council for Harmonisation (ICH) E9 guidance. 15 The aim of this framework was to clearly describe the trial objectives and, ultimately, ensure appropriate analysis, reporting and interpretation in relation to these.

Elaboration and explanation

Each of the items in Table 1 is explained further in the subsequent sections and examples are provided (one in Example: superiority and two further examples in Appendix 4).

Example 1: superiority

A RCT is being used to compare the clinical effectiveness and cost-effectiveness of two management strategies for non-acute anterior cruciate ligament (ACL) injury. 21 Box 1 presents the completed list of factors that have an impact on the trial design.

As there are two active health technologies to be assessed, an argument could be made for a non-inferiority trial design. However, there is no basis for suggesting that any of the secondary outcomes (in addition to knee health) would be superior for the intervention over the comparator; therefore, showing superiority on knee health is essential. Furthermore, as the comparison is between two management strategies, the desire is to identify which is more clinically effective. Owing to the lack of evidence regarding the health technologies, the main aim of the study is to define which is more clinically effective for treating knee health (i.e. a superiority design).

Demonstrating superiority on the primary outcome is particularly important when trying to convince clinicians that rehabilitation should be used instead of surgical management, which is currently the most commonly used treatment. Given all of this information, the trial was designed on a superiority basis.

Benefit–risk dependent on trial design

The considerations that take place when selecting the most appropriate trial design can also have an impact on whether or not a B–R method is appropriate. In the case of a superiority trial design, in which the only consideration is the primary outcome (and, potentially, other superiority secondary outcomes, as measured in the trial), trial conclusions could be considered self-evident and so a B–R method would be unnecessary. 22 One workshop participant said:

. . . for others [trials] it probably is quite benign treatments. You’ll just go with the most effective one or the most cost-effective one, other considerations might not be as important.

Workshop participant

However, there are numerous scenarios in which it could be useful to include a B–R method in a clinical trial, especially when an equivalence or a non-inferiority trial design has been selected.

Reason for inclusion

The key reasons for using a B–R method were identified from the survey, literature review and workshop and were collated into a few specific themes that are described in the following sections and summarised in Box 2.

Definition

Benefit–risk assessment is a group of methods to compare or trade off favourable and unfavourable effects of a treatment. 27 This could be a subjective judgement or one achieved through more complex, quantitative, methods, but the overarching idea is that information related to multiple outcomes is taken into account simultaneously. The aim of this may be to assess if a single treatment has a positive B–R balance (i.e. the favourable effects outweigh the unfavourable effects) or to see which treatment has the best overall B–R balance, and thereby inform clinical practice.

Methods

Numerous methods are captured under the umbrella of B–R assessment; other reviews28–30 and the PROTECT group’s website5 provide a useful overview of these, as well as further detail.

The rapid review identified 92 appropriate methods, and these methods, along with information gained from the workshop, were split into seven groups. An overview of these methods is provided in this section and further details, including examples of specific methods, are reported in Appendix 2.

Applications of methods

During the workshop, there was universal agreement, supported by the literature,22,33 that there is not one method that best fits all situations. The scope of this project was not to assess the intricacies of which specific method to use. Rather, this section will suggest which of the group of methods outlined in Chapter 4, Methods, may be appropriate for each reason. Case studies that implement a range of methods are summarised in Appendix 4 for further information.

Implementation of methods within randomised controlled trials

Within an individual trial, multiple B–R methods could be appropriate at different stages. In addition, there is an additive nature to the methods, meaning that multiple methods could also be appropriate at the same stage. For example, when using a quantitative trade-off, it could still be useful to present the results in a summary table and provide a narrative summary of the final decision and reason. As the complexity of the method increases, simpler methods could often be used to support the presentation of information.

There are also many situations when more than one method is required, for example the use of uncertainty estimation when quantitative trade-offs have been completed. A diagram showing how the methods interact is included in Appendix 2.

To assess the use of methods at different stages of the study, the roadmap,5,36 which contains five steps and was created by the PROTECT group, has been mapped onto the stages of a typical, individual NIHR HTA RCT (Figure 2). Potential method groups that could be appropriate at each of these stages, along with their purpose, are described in Table 2. In addition, as previously discussed, a descriptive framework may be useful, spanning all stages of the RCT. Examples of studies that have used B–R methods are provided in Appendix 4.

FIGURE 2.

Mapping of the PROTECT roadmap (reference 5 and 36) to an individual ranomdised controlled trial.

General considerations

Given the nature of B–R methods, there are a few additional considerations that should be taken into account.

Checklist for reporting on trial design and results

The sections below provide checklists for the reporting of B–R assessments. These are split between the information to include when reporting the trial design, for instance in a funding application or protocol, and reporting on the results of a study (e.g. in the final trial report or a journal article containing the results). When reporting the trial design, this information is likely to be found in the methods section of the documents; however, when reporting the results of a trial, the information may need to be split across the methods and results sections of the document.

Example

This example has been created by following the checklist to ensure that all appropriate information is included. Although the example is a real trial, the results presented are fictional and do not represent actual trial results.

The MAGIC (Melatonin for Anxiety prior to General anaesthesia In Children) trial is a multicentre, parallel RCT, aiming to compare the use of melatonin and midazolam as premedication for anxious children attending for elective surgery under general anaesthesia. This trial has a primary outcome to assess children’s anxiety on a non-inferiority basis using the modified Yale Preoperative Anxiety Scale (mYPAS) outcome measure.

Implications for practice

Findings from this research suggest that there are circumstances in which B–R methods would be useful in publicly funded clinical trials to assess the overall effects of treatments that depend on multiple outcomes. If one of the six key reasons that were identified applies to the trial, the team should consider including a B–R method. Funding panels can also use the list to assess the appropriateness of the research plan when reviewing applications.

The available methods vary widely in complexity and processes; appropriate methods should be chosen based on which of the reason(s) are relevant to the trial. Again, trial teams and funding panels can be informed by the results of this work.

Limitations

The limited scope of this report means that there may be additional considerations required if other, more complex, design features are also present. This work would need to be evaluated to ensure that it remains applicable to these situations.

In addition, the sample size used in this work was small, which may have an impact on the representativeness of the results. However, the breadth of expertise was felt to be suitable for the recommendations produced.

Recommendations for future research

This project has focused on assessing when a B–R method, in any form, is applicable; however, detail on the individual methods has not been included, beyond using the methods as examples. Future research should create resources on specific methods and provide detail on how they can be integrated into a publicly funded clinical trial to support future research teams.

Patient and public involvement

Patient and public involvement was not included within this project as the focus of the work was to gain expert opinion to help design clinical trials. The participants included were selected based on their expertise and the methodology selected was trialled with similar experts before use to ensure its appropriateness.

Contributions of authors

Nikki Totton (https://orcid.org/0000-0002-1900-2773) (Research Fellow, Medical Statistics) designed the study, conducted the survey, rapid review and consensus workshop, analysed all collected data, and drafted and prepared the final manuscript.

Steven A Julious (https://orcid.org/0000-0002-9917-7636) (Professor of Medical Statistics) designed the study, provided oversight for the conduct of the study and read, contributed to and approved the final manuscript.

Elizabeth Coates (https://orcid.org/0000-0002-2388-6102) (Research Fellow, Qualitative Research) designed and ran the consensus workshop, prepared the workshop results for publication and read, commented on and approved the final manuscript.

Dyfrig A Hughes (https://orcid.org/0000-0001-8247-7459) (Professor of Pharmacoeconomics, Health Economics) designed the study, provided oversight for the conduct of the study and read, contributed to and approved the final manuscript.

Jonathan A Cook (https://orcid.org/0000-0002-4156-6989) (Professor, Medical Statistics) designed the study, provided oversight for the conduct of the study and read, contributed to and approved the final manuscript.

Katie Biggs (https://orcid.org/0000-0003-4468-7417) (Assistant Director, Literature Reviews) designed and conducted the rapid literature review, contributed to the workshop content, prepared the review results for publication and read, commented on and approved the final manuscript.

Catherine Hewitt (https://orcid.org/0000-0002-0415-3536) (Professor of Trials and Statistics) designed the study, contributed to the workshop content, contributed throughout the study and read, commented on and approved the final manuscript.

Simon Day (https://orcid.org/0000-0002-5672-6818) (Director, Clinical Trials and Statistics) designed the study, contributed throughout the study and read, commented on and approved the final manuscript.

Andrew Cook (https://orcid.org/0000-0002-6680-439X) (Consultant in Public Health Medicine and Fellow in HTA) designed the study, contributed throughout the study and read, commented on and approved the final manuscript.

Publication

Totton N, Julious S, Hughes D, Cook J, Biggs K, Coates L, et al. Utilising benefit-risk assessments within clinical trials – a protocol for the BRAINS project. Trials 2021;22.

Data-sharing statement

All data requests should be submitted to the corresponding author for consideration. Access to anonymised data may be granted following review if appropriate.

Disclaimers

This report presents independent research funded under a MRC–NIHR partnership. The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, the MRC, the HTA programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, the MRC, the HTA programme or the Department of Health and Social Care.

Methods

The survey was hosted on the Qualtrics platform46 between May 2019 and August 2019. The link was sent through the UK Clinical Research Collaboration Clinical Trials Network Lead Statisticians, Health Econ on JISCMail and Medical Research Centre Hubs for Trial Methodology Research (Trials Methodology Research Partnership) distribution lists, as well as through the steering group to known networks or researchers in the area.

Results

The demographics of the survey participants are shown in Table 5.

The data collected showed that most survey participants did not have experience with B–R methods. Those who did had mostly worked on publicly funded superiority trials testing a drug (Table 6); however, this finding reflects the demographics of the participants and is, therefore, not necessarily generalisable.

The reasons provided for carrying out a B–R assessment were that showing efficacy is not enough, to consider patient preferences in treatment decisions and because health economics were required in the trial, prompting the use of B–R methods. These were combined with the transcripts from the workshop to assess the key reasons for completing these designs. Out of eight respondents, only one had considered the B–R assessment in the design of the trial, although some respondents suggested that it was an important element to consider.

When asked, 44% of respondents stated that they would like recommendations on how to select the appropriate trial design, and 41% stated that they would like recommendations on how to select the end points, both of which have, therefore, been incorporated into the report (Selecting a trial design and Outcome measure selection, respectively). Other items were included where possible, but further work is needed in this area to provide further details and recommendations (Figure 3).

FIGURE 3.

Items requested within the survey for advice and recommendations.

Limitations

A key limitation of the survey is that we were not able to estimate a response rate because of the use of mailing lists and personal networks to promote the survey. Therefore, we are unable to assess the potential bias in the selection of respondents; the focus on publicly funded trials suggests that the responses are not representative of all researchers using B–R methods.

Methods

Full details of the rapid review methods are given on the PROSPERO web page. 47 The rapid review has been registered with PROSPERO (PROSPERO reference CRD42019144882).

Results

In total, 1196 articles were identified, with 76 extracted (Figure 4). The 97 methods that were identified were categorised into seven groups; common examples are presented in Table 7, along with information about whether or not additional data would be required on top of usual data collection to include these in a NIHR HTA RCT. Figure 5 shows how these groups of methods interact and indicates the split when additional data would be required.

FIGURE 4.

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow chart from article identification to inclusion for the rapid review.

FIGURE 5.

Benefit–risk method groups from the rapid review and their interactions.

Key recommendations from regulatory agencies, which commonly use B–R methods to assess the balance of drugs, are summarised in Table 8 and have been used to suggest which methods may be appropriate in a range of circumstances.

Limitations

The use of a rapid literature review was selected because of the limit on time and resources in this project. This is not as thorough as a full systematic review,29 which could have resulted in less representative results. However, comparing the results of the rapid review with another published systematic review showed that more methods were identified in our review and none were omitted, providing some reassurance.

Methods

Experts were invited to join the consensus workshop through several avenues:

• Respondents to the survey who expressed interest in the workshop were provided with further information.

• All co-applicants of the BRAINS project were invited.

• Experts from NIHR funding panels were provided with information about the workshop.

• Prominent experts in the area were invited directly.

The workshop contained presentations over the 2 days to support the discussions and, ultimately, the recommendations produced. The presentations were based on the findings from the two previous work packages (WPs) and included:

1. background to the project and its focus, highlighting the perspective of the funder who commissioned the work

2. results from the survey

3. background to the different trial designs (i.e. superiority, equivalence and non-inferiority), with examples

4. results from the rapid literature review

5. an overview of available B–R methods

6. background to the differences between the regulatory and publicly funded setting

7. an introduction to other checklists, also highlighting the purpose of improving reporting.

Two different techniques were used in the workshop to answer the research objectives. NGT was used as a consensus methodology to enable the generation of item lists, and open discussions were used to produce themes.

The NGT is an interactive multistage methodology through which group consensus can be sought during a face-to-face meeting. 6 This structured approach has several benefits, as it encourages equal and wide-ranging contributions from all participants and helps to reduce conflict and the potential for the discussion to be dominated by some participants. In addition, it can be completed in a relatively short time.

The potential use of B–R in clinical trials has additional complexities because of the number of methods, reasons for use and perspectives to consider. Therefore, an open discussion was held to ensure that no restrictions on points of discussion were in place. In addition, other relevant discussions ensued during the workshop following the presentations. Each of the workshop sessions was recorded and the recordings from the open discussion regarding the appropriateness of when to use B–R methods, as well as other relevant discussions, were transcribed verbatim. These transcripts were analysed thematically to identify key themes. 7

Results

The key participant characteristics are shown in Table 9. Some participants spanned multiple sectors, but their primary purpose for being at the workshop was recorded. The workshop attendees provided a good mix of those early in their career, developing in their career and in senior positions. This provided an overview of experiences within the group.

Limitations

One limitation relates to the relatively small number of people who took part in this stage of the research. Owing to the limited use of B–R methods in the publicly funded setting, it was always going to be challenging to achieve a large sample size. Unfortunately, this means that the workshop is unlikely to have been widely representative; however, participants in the consensus workshop represented a range of specialties and perspectives, which helped to make up for the small number. As the aim was to achieve expert consensus, the representation achieved was suitable to reach the required conclusions. In addition, the draft results were shared with other researchers in the area, who were unable to attend the workshop, for input and confirmation.

Example 2: equivalence

The Individualised Screening for Diabetic Retinopathy (ISDR) study67 is funded by NIHR to evaluate standard annual screening against individualised, risk-based, variable-interval screening for people with diabetes. Box 3 presents the completed trial selection considerations.

Given the lack of evidence around the timing of screening and the similarity between the groups, there is limited evidence that superiority would be seen on any of the health or screening attendance outcomes. These are the two key outcomes to assess in the trial, alongside the cost-effectiveness, meaning that consideration of equivalence or non-inferiority should be considered.

There is a resource constraint in the health-care system around the number of screening visits, which suggests that equivalence would be required on this outcome, whereas non-inferiority would be required for detection of sight-threatening diabetic retinopathy.

Although an equivalence design would require a larger sample size, it was felt that it was feasible to achieve such a sample in the trial population. In addition, the equivalence restriction is essential to answering the research question and, therefore, the increased sample size (and, thus, cost) was felt to be good value for money.

Example 3: non-inferiority

PRaCTICED (Pragmatic Randomised Controlled Trial assessing the non-Inferiority of Counselling and its Effectiveness for Depression)68 aimed to assess counselling for depression (CfD) against cognitive–behavioural therapy (CBT) in primary care patients who had suffered with depression. Box 4 considers the factors that influenced the selection of the trial design.

The primary outcome for this trial must be the patient’s depression, to ensure that this is sufficiently treated by the interventions. Owing to the similarity of the two interventions, there is no evidence base to suggest that the intervention would be superior to the comparator; however, it may be cheaper and/or preferred by patients. As patient preference is often related to the efficacy of a treatment and not cost-effectiveness, this suggests that it would be important to demonstrate non-inferiority on the key outcome (i.e. depression). To change practice, the delta value chosen for the non-inferiority margin may need to be small in order for patients, clinicians and decision-makers to see this as a worthwhile treatment.

There is no restriction on an improvement in the outcome of depression for the intervention and, therefore, the added complexity of an equivalence design is not required. This trial was designed, using depression as the primary outcome, on a non-inferiority basis.

Checklist for reporting the trial design

Checklist for reporting the trial results