The assessment and appraisal of regenerative medicines and cell therapy products: an exploration of methods for review, economic evaluation and appraisal

The ethics, feasibility and reliability of small randomised controlled trials

Although the randomised controlled trial (RCT) is the expected level of evidence needed for regulatory assessments, it is recognised that for some indications such expectations are unrealistic. Conducting RCTs in populations with severe or advanced disease may be problematic for a variety of reasons. Such populations may be very small and, consequently, recruitment into an adequately sized trial would require a large number of centres and would be very expensive and take a very long time. In addition, when no alternative treatments exist, patients with life-threatening diseases or severe morbidity typically need, and desire, accelerated access to innovative new therapies. Patients with more severe or advanced disease may be more willing to accept the risks of an experimental therapy. In such situations randomisation to a control treatment may be ethically problematic (because of an absence of clinical equipoise).

A health technology assessment (HTA) review of ethics issues in the design and conduct of RCTs described numerous situations in which alternative non-randomised designs are morally or practicably preferable. These included when large differences between treatments are expected; when a disease, if left untreated, is lethal and for which there is no known effective treatment (i.e. unmet need); and when a disease is rare and recruitment is slow. 5 Additionally, when trial populations are small, it may be difficult to differentiate a true treatment effect from a chance effect. Important chance imbalances in relevant prognostic factors between groups at baseline are more likely in small trials. The HTA review highlighted the problem of underpowered RCTs, which were described as ‘necessarily unethical’ as they were unlikely to produce clear-cut answers. 5 This argument was supported by 15 articles that stipulated the statistical necessity for random errors in measured effects of treatments to be small in comparison with the size of the therapeutic effect sought. Other articles in the review discussed the ethics of stopping RCTs early when there is some evidence of efficacy and the subsequent problems that this may cause, with reduced statistical precision, clinicians not being persuaded by results and secondary trial aims being compromised being some of the key problems.

A related HTA review discussed further the ethics issues that may arise when early-phase (e.g. single-arm) trials produce very encouraging results: it may be unethical to conduct a further trial if the intervention is apparently effective in a small number of patients. 6 In such a situation, the argument for a trial rests on demonstrating a grey area between a reasonable hope that the intervention is effective in a few patients and a rational and justified belief that it is effective for the studied patient population more generally (i.e. the evidence to date has sufficient external validity).

Possible alternatives to the randomised controlled trial

More recently, a framework for using unfamiliar trial designs when rare diseases are studied has outlined several possible alternative approaches. 7 The framework aims to facilitate research when populations are small. Two of the ‘adaptive’ designs outlined may be particularly relevant for regenerative medicines, for which treatment intentions may be curative. The first is responsive–adaptive randomisation, which maximises allocation to the most effective treatment and minimises the required sample size. Outcomes for previous participants affect the subsequent treatment allocation probabilities. This ‘play the winner’ rule has the potential to reduce the number of patients who are allocated to less effective treatments and can therefore reduce the ethical concerns associated with randomisation. However, this design is limited to studies that assess rapidly available outcomes (as results from previous patients are needed to influence future allocations). Modified designs have also been outlined to counter the criticism that comparisons may be obtained in which only one patient has received conventional treatment. The second adaptive design that may also be useful for studying regenerative medicines is the internal pilot design. This design eliminates the loss of scarce eligible participants because of participation in a prior pilot study. Once the pilot phase is finished, a sample size is recalculated with the study continuing until this number is recruited; patients from the pilot phase are included in the final analysis.

A EMA reflection paper on methodological issues associated with adaptive designs suggested that such designs ‘would be best utilised as a tool for planning clinical trials in areas where it is necessary to cope with difficult experimental situations’ (p. 3). 8 Cited examples of such situations included ‘small populations or orphan diseases with constraints to the maximum amount of evidence that can be provided’ and when there are ‘ethical constraints to experimentation’ (p. 10). However, the US Food and Drug Administration (FDA)9 raised two principal issues with regard to adaptive design methods more broadly:

1. whether the adaptation process has led to design, analysis or conduct flaws that have introduced bias which increases the chance of a false conclusion that the treatment is effective (a type I error)

2. whether the adaptation process has led to positive study results that are difficult to interpret, irrespective of having control of type I error.

This draft FDA guidance document also noted that, for some of the more recently developed adaptive methods (including adaptive randomisation methods), the magnitude of the risk of bias and the size of the potential bias, and how to eliminate these effects, are not yet well understood.

Although adaptive designs may be useful in some situations, it is still likely that single-arm trials will form the basis of many submissions for the regulatory approval of regenerative medicines (because of the nature of the target populations). Nevertheless, a study that reviewed 31 oncology drugs or biologics approved by the FDA (between 1973 and 2006) without a randomised trial that incorporated a comparator treatment, supportive care or placebo arm concluded that such drugs have a reassuring record of long-term safety and efficacy despite the fact that nearly all of the evidence studies were single-arm Phase II trials. 10 The median number of patients studied per approval was 79 (range 40–413); response rate was the primary end point for most drugs, and the median objective response rate was 33%. At the time of publication (2009) all but one of the drugs were still approved; marketing authorisation for gefitinib (Iressa^®; AstraZeneca, Cambridge, UK) was rescinded after a RCT showed no survival improvement. Nineteen drugs have additional uses, with formal FDA approvals obtained for 11.

Evolving regulatory pathways

Since the late 1980s and early 1990s, regulators and HTA bodies/payers around the world have produced new approaches to provide patients with timely access to new medicines. 11 These new regulatory pathways can also improve competitiveness; shortened product development times prior to licensing can be very beneficial and more appealing to emerging small and medium-sized enterprises.

An overview of the relevant EMA regulatory accelerated access pathways is presented in Table 1. The main mechanism for accelerated access of these pathways is the reduced development time.

The EMA’s most recent development in this area is the adaptive licensing pilot programme, which was launched in 2014. The programme utilises the regulatory processes within the existing EU legal framework and is defined as being a prospectively planned adaptive approach to bringing drugs to market. It is more of a staggered iterative system than previous approval pathways. Such a ‘life-cycle approach’ to acquiring and (re)assessing evidence will consider the basis of decision-making in the following stages of a product’s life cycle: development, licensing, reimbursement, monitoring/post-licence evidence and drug utilisation. 11 Importantly, the approach encompasses both the authorised indication and the potential further therapeutic uses of the medicine. The EMA changed the name of the pilot project from ‘adaptive licensing’ to ‘adaptive pathways’ to better reflect the idea of a lifespan approach.

The pilot project aims to examine whether or not this kind of approach to medicine development and authorisation will offer advantages in terms of achieving the best balance between the need for timely patient access and the importance of providing adequate, evolving information on benefits and risks. In so doing it is expected to develop thinking in the following areas:12

• encourage developers of medicines to consider all regulatory tools and flexibilities within the existing EU legal framework when planning the life cycle of medicine development

• explore the extent to which regulatory demands for the generation of evidence around efficacy and safety are compatible with demands around evidence generation from other stakeholders (e.g. HTA bodies, payers, patient organisations)

• investigate in a timely manner the hurdles that exist in realising the most efficient medicine development pathways, including the role and limitations of real-world data.

Ideas for refining and improving this lifespan approach are developing at pace. For example, with MAPPs (Medicines Adaptive Pathways to Patients) the development plan across target populations and indications will be agreed upfront with the EMA, which distinguishes the MAPPs process from the conventional indication expansion approach. The MAPPs plan may include a range of studies, such as RCTs, single-arm studies, pragmatic trials and other forms of real-world study. 13 A newly formed public–private project called ADAPT SMART (Accelerated Development of Appropriate Patient Therapies: a Sustainable, Multi-stakeholder Approach from Research to Treatment-outcomes), which is funded by the EU Innovative Medicines Initiative, aims to facilitate and accelerate the availability of MAPPs. 14 NICE is one of the 32 international partners that together represent regulators, patients, academia and industry. The challenge for ADAPT SMART is to develop a MAPPs model that aligns the needs of all stakeholders, including patients, member state payers, regulators, medical practitioners and industry. A major task will be the identification of opportunities and obstacles, and providing a framework for MAPPs that will overcome the latter and seize the former. ADAPT SMART will address the challenges to the broad implementation of MAPPs by exploring new concepts to align the various stakeholders and create a consensus on what evidence will be required, how multiple sources of available data can be best used to facilitate MAPPs and which scientific challenges related to MAPPs need to be addressed. 14

In the UK there is another initiative that may facilitate the pathway to market: the Medicines and Healthcare products Regulatory Agency (MHRA) operates an early access to medicines scheme (EAMS), which was launched in April 2014. This voluntary scheme (which does not replace the normal licensing procedures) is aimed at unlicensed or off-label treatments deemed by the MHRA to be ‘promising innovative medicines’ (PIMs) for treating life-threatening or seriously debilitating conditions for which there is unmet need. Once a PIM designation is obtained (stage 1 of the process), the MHRA can then provide benefit and risk information (stage 2 – scientific opinion) to doctors who may wish to prescribe the unlicensed medicine under their own responsibility. However, it appears somewhat unclear how the EAMS assessment output may impact on ongoing or forthcoming EMA assessments of the same therapy (e.g. in terms of speeding up processes or reducing repetition of information). Further uncertainty around EAMS exists regarding how therapies with this regulatory status can be funded. As EAMS is not accompanied by any funding arrangements, meeting the costs of the therapy is currently the responsibility of the manufacturer; this can act as a barrier to adoption, especially for high-cost therapies produced by small enterprises.

Regenerative medicines in the new regulatory environment

The experience gleaned from the EMA adaptive licensing pilot so far appears to be quite limited with respect to regenerative medicines: as of May 2015 one of only three candidate ATMPs had been selected for a ‘stage II’ proposal. 15 By far the most accommodating regulatory environment for developing regenerative medicines is currently Japan, where, under the new 2014 legislation, regenerative medicines can receive accelerated conditional approval after a single clinical study, provided the trial has demonstrated the therapy to be safe, with evidence of a probable therapeutic benefit. This approach aims to dramatically accelerate patient access and meaningfully shorten clinical development times, thus promoting investment (as faster, cheaper development, coupled with accelerated commercialisation, would shift the risk–reward ratio favourably from an investment perspective). 16 However, there is concern that this approach may leave Japan with regenerative medicines that are unrecognised by other countries because of efficacy concerns: the lack of an explicit plan for determining efficacy during the conditional approval period points to a strong underlying assumption that regenerative medicines will ultimately prove efficacious, whereas experience from other areas of clinical research suggests that such optimism may be misplaced. 17 The initial demonstration of safety based on only Phase I trial data is an additional major concern.

The concern raised about the limited evidence that will probably be presented when a product is submitted for regulatory approval is by no means limited to the Japanese regenerative medicine experience. As many regenerative medicines will be developed with the initial aim of treating small patient populations in which there is unmet need, it is likely that they will be evaluated via a regulatory pathway that offers patients accelerated access to the new treatment. A consequence of this is that many of the studies submitted will be early-phase, small single-arm trials. Nevertheless, the Japanese regulations excepted, the newer regulatory pathways being developed across the world do not focus specifically on facilitating the licensing of regenerative medicines. The newer pathways are primarily aimed at addressing unmet need in serious conditions for which no alternatives exist, regardless of the type of technology. However, much of the focus and expectation for success in this area seems to have been directed at regenerative medicines, possibly because they may evolve over time and may therefore, ultimately, not be restricted and limited by having single modes of action. The submission of evidence that is based on single-arm studies appears to be less to do with regenerative medicines being a ‘special case’ category of interventions, but rather a consequence of the seriously ill, very small populations of patients with unmet medical needs who are often the initial target of new regenerative medicines.

Methods

From the regenerative medicine literature and experts in the field we sought to identify regenerative medicines that have been granted marketing authorisation in the EU. In addition to EMA assessment documents, we also sought any NICE or FDA documents. We extracted key details from these reports, with a primary focus on identifying issues that might be unique, or particular, to regenerative medicines.

Results

We identified six regenerative medicines that are (or have been) licensed in the EU: ChondroCelect^® (TiGenix, Leuven, Belgium), matrix-applied characterised autologous cultured chondrocyte implant (MACI) (Sanofi, Gentilly, France), Glybera^® (alipogene tiparvovec; uniQure, Amsterdam, the Netherlands), Holoclar^® (Holostem Advanced Therapies, Modena, Italy), PROVENGE^® (sipuleucel-T; Dendreon Corporation, Seattle, WA, USA) and ReCell^® (Avita Medical, London, UK). No allogeneic therapies were identified – all were autologous. Summary details are presented in Table 2; more comprehensive details can be found in Appendix 1 (see Table 42).

ChondroCelect and MACI are both therapies for treating knee cartilage defects. ChondroCelect was the first ATMP to receive marketing authorisation, in 2009. The marketing authorisation for MACI was suspended in September 2014 as an authorised manufacturing site no longer existed. Holoclar is a therapy used for treating corneal lesions resulting from burns to the eye. In 2014 it became the first stem cell-based ATMP to gain regulatory approval (conditional marketing authorisation was granted). PROVENGE is an active cellular immunotherapy for asymptomatic or minimally symptomatic metastatic hormone-relapsed prostate cancer when chemotherapy is not yet indicated; this therapy purportedly helps the immune system to selectively attack cancer cells (rather than directly attacking tumour cells, as happens with CAR T-cell therapies). EMA marketing authorisation was granted in June 2013 but withdrawn in May 2015 at the request of the manufacturer, for commercial reasons. Glybera is used to treat familial lipoprotein lipase deficiency (a rare genetic disorder) with associated pancreatitis. Its mechanism of action is viral vector delivery of a therapeutic gene to muscle cells. In 2012 it became the first gene therapy to be approved in Europe or the USA. The ReCell spray-on skin system is a regenerative medicine device. It harvests a small amount of a patient’s skin cells, which are then processed to produce a mixed cell population for immediate delivery onto burn wound surfaces. ReCell can be given rapidly as there is no need for proliferation of the harvested skin cells. A Conformité Européere (CE) mark was granted in 2005 (under Medical Devices Directive 93/42/EEC28).

Summary

The key issues arising from the reports of licensed regenerative medicines, that is, the issues that may be beneficial to consider when appraising future regenerative medicines, were:

• the importance of considering the likelihood of cure or improvement without experimental treatment when evaluating the results of single-arm studies

• the positive and negative implications of long-term persistence of therapies within patients

• the use of reliable surrogate outcomes (i.e. the need for validation of the relationship between surrogates and real clinical outcomes)

• the problems of long-term evaluations when therapies evolve over time

• none of the six regenerative medicines approved for use in the EU to date was an allogeneic therapy.

Methods

Pragmatic surveys of the literature were carried out to address these research questions. One review addressed the reliability of obtaining treatment effectiveness estimates in comparative NRSs. Two further separate reviews were carried out with respect to the second research question: one focusing on methods to adjust for bias in the evidence synthesis process and a second on methods of analysing individual patient data (IPD) from NRSs. A final review explored the literature relating specifically to single-arm studies. For each review a number of key articles were identified using unstructured searches of MEDLINE and studies known to the team. Based on these key studies snowballing techniques were then applied in which citation searches were carried out and references checked for relevant studies. Citations and references of any additional studies identified were then also checked until no further relevant studies were identified.

Records identified in both the searches of MEDLINE and citation searches were screened by a single reviewer and the full texts of those deemed potentially relevant were obtained and also screened by a single reviewer.

Results

Relevance to future regenerative medicine submissions

Although RCTs continue to be the dominant method for evaluating treatment effectiveness, a large number of studies has been conducted devoted to establishing the relatability of evidence from NRSs. This sizable literature demonstrates both the value and the challenges of using observational data. Although the evidence is mixed regarding the reliability of observational data for evaluating treatment effectiveness, the existing studies do seem to indicate that, in some cases at least, confounding is a potential issue and will impact on treatment effectiveness estimates. Furthermore, the current evidence suggests that retrospective studies and, in particular, historical control studies are more likely to result in biased estimates of effect. As observed in Clinical efficacy and safety issues arising from European Medicines Agency, National Institute for Health and Care Excellence and Food and Drug Administration assessments of licensed regenerative medicines, many recent regenerative medicine submissions have been based on data from single-arm studies, which have been compared with historical controls. The findings of this review therefore suggest that a degree of caution is necessary in interpreting estimates from these comparisons, as bias in estimates of effectiveness from historical comparisons will add additional uncertainty not accounted for in the CIs/credible intervals presented. A key factor in the reliability of estimates of effectiveness based on observational data is the statistical analysis used, and a large number of studies have similarly sought to develop and evaluate methods for adjusting and eliminating bias resulting from confounding. Despite this, it is unclear which methods are most appropriate in certain circumstances and further research is needed. Consequently, results generated from NRSs will be subject to an unknown degree of uncertainty, even after adjustment for confounding. Single-arm trials are reliable indicators of treatment benefit only when the natural history of the disease is very well known, the patient population is homogeneous and the control treatment has little impact on outcomes. It is interesting that increasing the size of single-arm trials is not always helpful.

If regenerative medicines continue to be targeted at tightly defined conditions, with a narrow population to minimise heterogeneity, when patients have little or no chance of recovery/improvement otherwise, the use of NRSs and, in particular, single-arm studies may be adequate. To complement the data from such trials, robust accurate evidence of the outcomes achieved with standard care must be provided. When appropriate, methods to adjust for confounding should be employed, with the selection of the method used being explicit and based on sound reasoning. Confidence in estimates of effect may also increase by utilising multiple methods of adjustment, although care should be taken to ensure that methods are appropriate to the decision problem in question. However, many regenerative medicines may require highly skilled and specialised facilities for optimum delivery. Consequently, the evidence on their efficacy and safety may be derived from only small, single-centre studies, which (more often than not) might overestimate effect estimates or which might lack the external validity needed to support more widespread uptake of the intervention.

In terms of NICE methods and processes, methods research may be considered to inform guidance both for manufacturers (e.g. minimum reporting requirements for analysis methods for comparing single-arm trial data with historical control data) and for ERGs (e.g. checklists for appraising how historical control data were identified and analysed by manufacturers).

Introduction

As discussed earlier (see Clinical efficacy and safety issues arising from European Medicines Agency, National Institute for Health and Care Excellence and Food and Drug Administration assessments of licensed regenerative medicines), it can be anticipated that almost all of the pivotal trials of regenerative medicines submitted for assessment for marketing authorisation will utilise a surrogate or intermediate outcome (or end point). A surrogate may be either a laboratory or a physiological measure of the patients’ experience that could be used to predict or provide an early measure of therapeutic effect. This section presents an overview of surrogate outcome measures and their use in clinical research and highlights issues pertinent to the development and appraisal of regenerative medicines.

Methods

To describe the use of surrogate end points as primary outcome measures in trials of new therapeutic agents a review of the most relevant and up-to-date literature was performed. The review was not systematic but was designed more as a pragmatic rapid review to assimilate current information and opinion on the use and suitability of surrogates in therapeutic trials. The review began with a search of key guidelines on the use of surrogate end points produced by the FDA, NICE DSU (Decision Support Unit) (University of Sheffield) and European Network for Health Technology Assessment (EUnetHTA) and survey results produced by the National Institute for Health Research (NIHR) HTA programme on the cost-effective use of surrogate outcomes. Citation and reference searches followed, which produced a library of relevant peer-reviewed publications and statistical reports on evidence for the use of surrogate end points in medicine. All relevant studies identified are presented in Appendix 4 (see Table 44).

Definition and examples of surrogate outcomes

Ideally, it is expected that the relative effectiveness of drugs and treatments will be based on final clinical end points,87 that is, an outcome that the patient, the clinician and other stakeholders hope to avoid such as morbidity, impaired quality of life and/or death. 88 RCTs with large sample sizes and extended follow-up periods are often required to capture the statistical significance of a treatment’s or an intervention’s impact on a patient-relevant outcome. 87 However, the requirements of RCTs are often impractical when considered alongside pressures of time for products to go to market and in particular the urgent need for new treatments for patients with chronic but life-threatening diseases. The principal rationale for the use of a surrogate outcome is a more rapid assimilation of data without the need for large and lengthy trials in patients for whom mortality rates are high or treatment options are few. 89

For example, OS is considered the gold standard to measure benefit in many clinical trials as it provides a precise and statistically and clinically meaningful end point. However, mature OS data are difficult to achieve because of the length of time needed and the number of deaths required for appropriate statistical analyses. Furthermore, OS as a measure of therapeutic success becomes less useful as the course and duration of diseases such as cancer move from being acute to more chronic; longitudinal effects of chronic disease such as comorbidities and additional ongoing treatments add further limitations to OS as an outcome. 90,91 As a solution, there has recently been a steady move (by regulatory bodies) away from OS as a clinical end point measure and towards more short-term surrogate measures.

A generally accepted definition of a surrogate has followed that of Temple (p. 4):92 ‘a laboratory measurement or physical sign used as a substitute for a clinically meaningful endpoint that measures directly how a patient feels, functions or survives’. However, chronic disease programmes and patient-reported outcomes have meant that a broader definition is now needed to better fit the HTA perspective. 93,94 Although the term ‘intermediate end point’ is sometimes used synonymously with surrogate end point,95 it is often used to refer to more patient-relevant outcomes than those typically thought of as surrogates. However, for the purposes of this report, the term ‘surrogate outcome’ will be used in its broadest sense.

Examples of approved drugs based on the use of validated surrogate end points include antihypertensives and blood pressure in stroke research, cholesterol-lowering agents and serum cholesterol and treatments for glaucoma and intraocular pressure;96 CD4 count for acquired immunodeficiency syndrome (AIDS) or death in human immunodeficiency virus (HIV) infection;97 and bone density for bone fracture in osteoporosis. 89 However, occasionally such approvals have to be revised when long-term data become available. The drug gefitinib was approved in the USA in 2003 for patients with non-small-cell lung cancer based on tumour response rate, a surrogate end point. When, in 2005, the results from later studies showed no significant benefit in terms of survival, the FDA withdrew approval for its use in new patients. Therefore, although surrogate end points offer the potential of real benefit – in providing patients with faster access to treatments and saving triallists time and resources – they may also have important drawbacks. Most notably (as the gefitinib example demonstrates), there may be uncertainty about the relationship between surrogate and real clinical end points and this may result in treatment efficacies being overestimated. A meta-epidemiological study that compared 84 trials that used surrogate outcomes with 101 trials that used patient-relevant outcomes showed that trials reporting surrogate end points had larger treatment effects: on average, trials using surrogate outcomes reported treatment effects that were 28–48% higher than those of trials using final patient-relevant outcomes and this result was consistent across sensitivity and secondary analyses. 98 The study characteristics of trials using surrogate outcomes and those of trials using patient-relevant outcomes were well balanced except for median sample size (371 vs. 741) and single-centre status (23% vs. 9%). Their risks of bias did not differ. This finding illustrates the importance of surrogate end points being appropriately validated and of quantifying the level of certainty of association of treatment effect between the surrogate and patient-relevant final outcomes. 98

Validation

Surrogate outcomes can be unreliable without sufficient validation; for example, two major antiarrhythmic drugs, encanaide and flecanaide, reduced arrhythmia but caused a more than threefold increase in overall mortality99 and cardiac inotropes improved short-term cardiac haemodynamic function but can increase mortality. 100 Such examples may fuel uncertainty about the validity of surrogates. The results of a questionnaire study of 74 stakeholders in the drug development of cardio-renal disease indicated that, although the use of surrogates is not opposed, most are not considered valid. 101 Out of the four surrogate outcomes suggested as an end point for trials – blood pressure, glycated haemoglobin (HbA_1c), albuminuria or C-reactive protein (CRP) – only use of blood pressure was considered moderately accurate. Questionnaire responders from industry valued the accuracy of surrogates consistently higher than academic and regulatory responders.

Current issues for health technology assessment and cost-effectiveness models

Regulatory bodies find it acceptable for trials to be shorter, to have fewer participants and to use surrogate outcomes when populations are rare and there is a high unmet clinical need. However, a commitment to ongoing research is mandatory to receive longer-term approval; if research is not continued or if it is continued but efforts to validate the surrogate fail, the approval will be withdrawn. 96 From a regulatory and HTA perspective, the absence of data on clinical end points might be acceptable when a clinical end point is difficult or impossible to study. The EUnetHTA summarised its findings into eight recommendations for end points used in the relative effectiveness assessment (REA) of pharmaceuticals:87

1. Efficacy assessments of pharmaceuticals should be based whenever possible on final patient-relevant clinical end points (e.g. morbidity, overall mortality).

2. Biomarkers and intermediate end points will be considered as surrogate end points if they can reliably substitute for a clinical end point and predict its clinical benefit.

3. Surrogate end points should be adequately validated and this must have been demonstrated based on biological plausibility and empirical evidence.

4. Validation of a surrogate is normally undertaken in a specific population and for a specific drug intervention. Demonstration of surrogate validation both within and across drug classes should be thoroughly justified.

5. The availability of a sufficiently large safety database is particularly important and evidence on safety outcomes should always be reported.

6. The absence of data on clinical end points might be acceptable when a clinical end point is difficult or impossible to study (very rare or delayed) or the target population is too small to obtain meaningful results on relevant clinical end points even after very long follow-up (very slowly progressive and/or rare diseases). However, these exceptions need to be carefully argued and agreed in advance.

7. Reassessment requirements for further data should be clearly defined when an assessment has been previously made based on surrogate end points.

8. Further methodological research on the use of surrogate outcomes is needed to inform future REA approaches for the handling of surrogates.

Similarly, Elston and Taylor88 recommended that a HTA or cost-effectiveness model based on a surrogate outcome should be undertaken only when it is not possible to base the assessment of clinical effectiveness and cost-effectiveness on final patient-related outcomes [i.e. mortality, important clinical events and health-related quality of life (HRQoL)]. In such cases, a systematic review of the evidence for the validation of the surrogate/final outcome relationship should be performed and the evidence on surrogate validation should be presented according to an explicit hierarchy.

Given the difficulty in validating surrogate outcomes, which conflicts with the need to use such outcomes in clinical research, Ciani and Taylor93 commented on the requirement to recognise the need for pragmatic high-level evidence, preferably from meta-analyses and regression modelling using both surrogate and final outcomes for HTA. This is demonstrated by a study conducted to illustrate the potential to reduce uncertainty around the clinical outcome by estimating it from a multivariate meta-analysis. 116 Bayesian multivariate meta-analysis was used to synthesise data on correlated outcomes in rheumatoid arthritis. Estimates from the Health Assessment Questionnaire (HAQ) were mapped onto the HRQoL measure, the EuroQol-5 Dimensions (EQ-5D) questionnaire, and the effect was compared with mapping the HAQ obtained from the univariate approach. The results showed that use of multivariate meta-analysis can lead to reduced uncertainty around the effectiveness parameter. By allowing all of the relevant data to be incorporated in estimating clinical effectiveness outcomes, including data from surrogate outcomes, multivariate meta-analysis can improve the estimation of health utilities through mapping methods.

In their review of HTA and cost-effectiveness models, Taylor and Elston89 found that only one of the four reports undertook a systematic review to specifically seek the evidence base for the association between surrogate and final outcomes. Furthermore, this was the only report to provide level 1 surrogate–final outcome validation evidence (i.e. RCT data) showing a strong association between the change in surrogate outcome (biopsy-confirmed acute rejection) and the change in final outcome (graft survival) at an individual patient level. The outcome of the review was to make recommendations for the evaluation of surrogate end points in a HTA (these are listed in Appendix 4, Table 44).

Taylor and Elston’s89 HTA publication has been key to providing insight into the use of surrogates within the HTA and cost-effectiveness models framework and presents the range of approaches. This includes HR calculation, transition probabilities within a model of natural history and predictive risk equations, used by researchers to quantify the relationship between surrogate and clinical end points. 88

In addition to calls for the validation of commonly used surrogate outcomes, there is a need for novel, more appropriate, more valid outcomes. An editorial in the Journal of Clinical Oncology117 commented on the large number of novel anti-tumour agents currently being tested in ever smaller groups of patients with increasingly specific tumour characteristics. Cancer types will continue to be divided into many subentities that differ from each other in terms of genetic make-up, natural course and sensitivity to systemic treatments. This, together with the limited number of patients who are available for clinical studies, means that a new approach to oncology research is needed. The editorial called for more intensive efforts at the preclinical stage to better understand the mode of action of potential new agents and for this information to be used to select more precisely the target population and appropriate and valid surrogate outcomes. By so doing it should be possible to achieve a higher success rate in Phase III studies, with smaller numbers of patients needed.

Summary

• Studies looking at surrogates for OS demonstrate how difficult it is to validate even commonly used surrogates.

• On average, it seems that trials using surrogate outcomes report larger treatment effects (28–48%) than trials using final patient-relevant outcomes.

• However, a desire to get regenerative medicines to market quickly means that manufacturer submissions are likely to be supported by short-term trials reporting primary outcomes that are surrogates.

• Regulatory agencies may accept evidence based on surrogate outcomes; for example, the FDA accepts that MRD is the strongest predictor of long-term EFS in ALL, although there is considerable uncertainty about its value in relapsed populations.

• The choice of surrogate outcomes must be researched, explicit and justified. Ideally, a systematic review of the evidence for the validation of the surrogate/final outcome relationship should be performed and the evidence on surrogate validation should be presented according to an explicit hierarchy.

• Analyses, at whatever stage of development and maturity of data, should include all available outcome data to minimise uncertainty.

Previous regenerative medicine evaluations evaluated within the National Institute for Health and Care Excellence technology appraisal process

Broader consideration of potential conceptual differences and possible methodological challenges for cost-effectiveness analyses

A separate review of known references and key citations of these was undertaken to identify other potential conceptual differences between regenerative medicine and cell therapies and more conventional medicines to identify potential methodological challenges for cost-effectiveness assessments.

During the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) conference held in November 2014, a workshop was held to discuss potential HTA and reimbursement challenges for regenerative medicines and curative treatments. As part of the workshop, Towse127 argued that many of the challenges for curative therapies appear similar to those for disease-modifying therapies for chronic diseases, sharing several associated problems, most notably (1) the use of short-term trials using surrogate outcomes that may not produce relevant clinical outcomes, (2) the use of outcomes that may not be sustained over time and (3) safety problems that may emerge over time.

In considering how these uncertainties might be formally incorporated within policy decisions regarding reimbursement, Towse127 acknowledged that value of information approaches provide a potential analytical framework. This framework formally evaluates the potential trade-off between the net health benefits to current patients from early access to the technology and those to future patients from withholding access to the technology until additional research has been conducted. The framework can then be used to help guide more appropriate policy choices between (1) adopt and reimburse now, (2) delay adoption/reimbursement and undertake further research (i.e. OIR) and (3) adopt/reimburse now and undertake further research [akin to coverage with evidence development (CED) or approve with research (AWR)]. 128 Towse127 also acknowledged the importance of risk-sharing approaches and particularly how these could enhance the value of CED/AWR.

As discussed in Clinical efficacy and safety issues arising from European Medicines Agency, National Institute for Health and Care Excellence and Food and Drug Administration assessments of licensed regenerative medicines, assessments of the clinical effectiveness and cost-effectiveness of regenerative medicines and cell-based therapies, particularly early in their life cycle, may be less extensive and lower in quantity than evidence for more conventional pharmaceuticals. Under these circumstances it may become even more critical to consider conditional reimbursement and possible risk-sharing agreements between the manufacturer and the payer.

Towse127 concluded that the main reimbursement challenges for regenerative medicines relate more to their financing than to the methodology of HTA and cost-effectiveness. In particular, concern was raised by Towse regarding whether health-care systems could cope with the potential high upfront costs of a curative treatment that appeared cost-effective using conventional thresholds, thereby presenting a potential barrier to adoption.

In the same workshop, Faulkner129 explored differences between regenerative medicines/cell therapies and conventional biologics. Specific reference was made to the more limited understanding by physicians and payers, leading to potentially greater requirements being made for longer-term data collection to more robustly demonstrate value. The multiple procedural steps required for some therapies were identified as another potential challenge, as these may be subject to different regulatory and reimbursement processes. Similar concerns were raised by another speaker, Husereau,130 who highlighted that regenerative medicines can be considered as both a biologic and a device/procedure, with many countries also having different reimbursement procedures, often based on cost minimisation for the funding of procedures because of fixed Healthcare Resource Group (HRG) pricing. In common with the arguments made by Towse, both Husereau and Faulkner also highlighted the challenge of applying a single financing approach for reimbursement.

The potential to enable a disease cure or prolonged therapeutic effect was also identified as a relevant characteristic by both Faulkner129 and Husereau. 130 However, Husereau130 identified a specific challenge surrounding the classification of a cure and its distinction from a prolonged therapeutic effect. To be considered curative, a therapy needed to demonstrate ‘no chance of re-entering suboptimal health state from same disease’. In reality, it seems unlikely that a new therapy can be definitively classified as curative prospectively, as many of the required elements cannot be demonstrated until a full lifetime of a cohort of treated patient has passed.

Husereau130 also raised the question of whether there is anything specific regarding curative therapies relative to standard treatments that could be perceived as providing additional benefits to patients beyond the current QALY framework. This was further considered in a subsequent publication. 131 Husereau reported that, although there was limited direct empirical evidence to address this specific question, important insights could be generated from the large literature exploring valuation issues for treatment (which he noted was often labelled as ‘cure’ within these studies) compared with prevention. Husereau reported a potential disconnect between existing literature reporting individual preferences and that reporting societal preferences. When given a choice between prevention and treatment, individuals appear to state a preference for prevention. However, similar preferences are not apparent when (societal) willingness to pay is considered. This disconnect was attributed to separate psychological factors including time, certainty of individual decisions and the valuation of identifiable compared with statistical lives. Husereau concluded that if a similar disconnect exists for curative therapies relative to standard treatments this could lead to considerable public debate and that further research was required.

Importantly, several of these issues and challenges highlighted in the workshop do not appear to be unique to regenerative medicines and cell-based therapies and similar challenges are often present when appraising more conventional biologics, companion diagnostics and devices more generally. However, it appears reasonable to conclude that these issues will be more commonly faced within evaluations concerning regenerative medicines and cell therapies. Furthermore, as many regenerative medicines and cell therapies will be considered as both a biologic and a device/procedure, manufacturers may have to address the specific regulatory and reimbursement challenges faced by both pharmaceutical and device manufacturers internationally.

Because of the personalised nature of regenerative medicines, the manufacturing and production process is typically more complex than that for traditional drug therapies. Current pharmaceutical manufacture is largely based around drugs being prepared, tested and manufactured in bulk at a consistent quality in advance of need, using automated processes. In contrast, many regenerative medicines require significant personalisation at the point of need. The complexity of the process, and the high level of personalisation required, may result in significantly higher marginal costs of production than for conventional pharmaceuticals or biologics. Inevitably, these additional complexities are likely to lead to higher upfront costs to health-care systems. However, high-upfront-cost therapies are not limited to regenerative medicine. A recent example is Sovaldi^® (sofosbuvir; Gilead Sciences, Foster City, CA, USA) for hepatitis C, which was recommended by NICE in some genotypes of the disease despite its high initial cost. 132

This complexity and personalisation is likely to be coupled with a requirement for the provision of additional health-care services within the overall process. The additional demands raise issues around the impact on the wider health-care setting, both at a marginal cost and wider infrastructure level. These demands may differ according to the extent of the services provided by the manufacturer and those requiring separate funding by the health-care system. In the sipuleucel-T appraisal,121 the lack of experience in the UK of using the treatment raised additional uncertainties about whether the NHS would incur additional costs that were not reflected in any of the scenarios evaluated, raising an additional source of uncertainty.

As the provision of regenerative medicines and cell-based therapies often entails multiple procedural steps (e.g. cell extraction, processing and administration) and may be undertaken alongside additional procedures (e.g. leukapheresis in the case of sipuleucel-T and arthrotomy for ACI), additional uncertainties are likely to be raised concerning the generalisability and transferability of evidence between different settings. That is, separating out the specific effect of the regenerative medicine or cell therapy from the effect of broader health provision, which itself may be subject to significant variation across different health-care systems, represents an important challenge to HTAs and cost-effectiveness assessments.

Many regenerative medicines and cell therapies also appear likely to share similar ‘unique’ characteristics to those reported for medical devices. 133 For example, particular parts of the procedural process may change significantly over time, experiencing incremental or step changes as new processes and infrastructure develops. Additionally, the requirement for highly specialised infrastructure and staff indicates the potential for a learning curve over time both for manufacturers and for health-care providers. Although increased automation methods and the ‘scaling out’ of services may subsequently reduce the need for highly specialised staff (and lower the marginal costs of production), the infrastructure requirements and implications for possible learning curve effects are likely to be an important consideration when assessing the cost-effectiveness of regenerative medicines and cell therapies.

In addition to issues related to uncertainty, issues of irrecoverable costs may pose an additional challenge. Irrecoverable costs are those costs that once committed cannot be recouped if changes occur at a later time, most commonly thought of as investment costs associated with the capital expenditure on equipment, new facilities or training and learning costs. These are likely to be most significant when the introduction of a new regenerative medicine or cell therapy imposes additional infrastructure requirements on the health system. Within economic evaluations, these costs are commonly annuitised and allocated as ‘per-patient’ costs by spreading the cost over the number of patients likely to be treated during the lifetime of the equipment. However, if reimbursement decisions about the technology change before the end of the lifetime of the equipment (e.g. approval is withdrawn), then these costs may not be recovered and hence need to be explicitly considered.

The risks of these more conventional types of irrecoverable costs to the health system may be more limited if the manufacturer provides the necessary infrastructure and associated training. However, irrecoverable costs also potentially exist at the patient level. Regenerative and cell therapies are developed, by design, to have a significant (if not permanent) period of effect, during which they may be neither removable nor reversible. The irreversibility of these therapies implies that any uncertainty associated with the long-term efficacy and adverse event profile has a greater potential significance than for treatments that can easily be reversed or switched.

Conclusions

The review has identified a number of common themes and potential challenges in relation to HTAs and assessments of cost-effectiveness for regenerative medicines and cell therapies. Some of the challenges identified do not appear to be unique to these types of therapies and are also faced by manufacturers of more conventional pharmaceuticals, biologics and devices. However, it seems likely that these challenges may be faced more routinely for regenerative medicines and cell therapies.

There is already provision within NICE’s methods guide to accommodate some of these aspects,122 although potential challenges may arise in ensuring that this is consistently applied between committees and understood by manufacturers. NICE will also need to consider whether further amendments to its processes and methods are required. Broader consideration will also need to be given to approaches that may extend beyond NICE’s existing remit, for example alternative funding approaches. Consequently, other bodies and manufacturers themselves may also have an important role in identifying more innovative approaches to seeking reimbursement that recognise the inherent uncertainties and lead to a more efficient sharing of associated risk.

About chimeric antigen receptor T-cell therapies

Chimeric antigen receptor T-cell therapies have been under development for around 20 years. The specific CAR T-cell therapies considered in this appraisal consist of autologous (i.e. the treated individual’s) T-cells that are genetically modified to redirect the target of the T-cell receptors. These receptors target specific proteins found on the surface of leukaemia cells, in this case the protein CD19, which is present on B-cell leukaemias as well as on healthy B cells, but which is not found on haematopoietic stem cells (which are situated in the bone marrow) or on other tissues. 145 The activated T-cells can then attack and destroy the leukaemia B cells. Persistence of a given CAR T-cell therapy within the body is linked to the properties of the T-cell from which the cells were derived as well as the immune environment into which they are infused. CAR T-cell therapies have already begun to evolve, with second-generation therapies currently being trialled in Phase II studies. Research efforts at developing future generations are focused on addressing the key challenges of T-cell target specificity, persistence and ability to exert the desired anti-tumour effects as well as identifying new target antigens. 146 CAR T-cell therapies have recently emerged as regenerative medicines with promising potential to treat haematological cancers. In July 2014 the FDA granted ‘breakthrough therapy’ status to the CAR T-cell therapy CTL019 (manufactured by Novartis, Basel, Switzerland) for the treatment of adult and paediatric relapsed or refractory ALL. 147

Although CAR T-cell therapies may offer relapsed or refractory B-cell acute lymphocytic (lymphoblastic) leukaemia (B-ALL) patients a ‘bridge’ to stem cell transplantation, or possibly even a cure for B-ALL, it is likely that patients will need to be monitored for some key adverse effects that are often reported. These include cytokine release syndrome (CRS) and B-cell aplasia (an absence of B cells). CRS occurs as a result of cytokines being released from the successfully targeted cancer cells and can result in various symptoms such as fever, headache, nausea and a rash. The severity of CRS appears to be proportional to the tumour burden. Although CRS is an adverse effect of CAR T-cell therapy, there may be a correlation between the development of CRS and response to therapy; patients who do not develop CRS may be less likely to benefit from CAR T-cells, whereas those who develop CRS often respond to the therapy. Although there may be some correlation between developing CRS and efficacy, there does not appear to be a strong correlation between the degree of CRS and response to therapy. 148

B-cell aplasia is an expected adverse effect of successful CAR T-cell therapies, which eliminate normal mature and precursor B cells. As long as CAR T-cells persist, B-cell aplasia continues (which provides what appears to be a highly accurate pharmacodynamic marker of CAR function). 148 B-cell aplasia is a manageable disorder; patients may be treated with intravenous immunoglobulin (IVIG), although this is an expensive treatment. Persistent B-cell aplasia could result in an increased risk of infection even with replacement therapy. 149

Overview of disease

B-cell acute lymphoblastic leukaemia is a subtype of ALL. ALL is a cancer that starts from the immature lymphocytes in the bone marrow and then invades the blood fairly quickly, spreading to other parts of the body including the lymph nodes, liver, spleen, central nervous system (brain and spinal cord) and testicles (in males). The term ‘acute’ means that the leukaemia can progress quickly and, if not treated, is probably fatal within a few months. The American Cancer Society has estimated that, including both children and adults, in 2015 there were about 6250 new cases of ALL (3100 in males and 3150 in females) and around 1450 deaths from ALL (800 in males and 650 in females). 150 The risk for developing ALL is highest in children aged < 5 years. Although most cases of ALL occur in children, most deaths from ALL (about four out of five) occur in adults. 150

UK statistics presents a similar picture. Statistics for the incidence of ALL in the UK (2009–11) are provided by Cancer Research UK,151 based on data sourced from the UK Office for National Statistics (ONS), Information Services Division Scotland, the Welsh Cancer Intelligence and Surveillence Unit and the Northern Ireland Cancer Registry. Across all ages in 2011 there were 654 new cases reported in the UK (377 in males and 277 in females), with crude incidence rates of 1.2 for males and 0.9 for females (per 100,000). Incidence is strongly related to age, but ALL is unusual as it does not follow the pattern of increasing incidence with age seen for most cancers; instead, the highest incidence rates are in children, teenagers and young adults. In the UK between 2009 and 2011, an average of 65% of cases were diagnosed in people aged < 25 years and only 6% of cases were diagnosed in those aged ≥ 75 years. Age-specific incidence rates are highest in infants aged 0–4 years and drop sharply through childhood, adolescence and young adulthood, reaching their lowest point at age 35–39 years and increasing slightly thereafter. Incidence rates are similar between males and females in all age groups except for those aged 15–19 years, when age-specific rates are significantly higher in males (male-to-female ratio of around 22 : 10). Averaged across all patients aged < 30 years the mean number of cases of ALL per year is 462.

There are subtypes of ALL based on the type of lymphocyte (B cell or T cell) and how mature these leukaemia cells are. 150 About 80–85% of ALL cases are B-ALL. There are several subtypes of B-ALL: early precursor B-ALL (early pre-B-ALL, also called pro-B-ALL); common ALL; pre-B-ALL; and mature B-ALL (also called Burkitt leukaemia). This last type is rare, accounting for only about 2–3% of childhood ALL; it is essentially the same as Burkitt lymphoma and is treated differently from most leukaemias. T-cell acute lymphocytic (lymphoblastic) leukaemia (T-ALL) constitutes about 15–20% of cases of ALL. This type of leukaemia affects males more than females and generally affects older children more than does B-ALL. It often causes an enlarged thymus (a small organ in front of the windpipe), which can sometimes cause breathing problems. It may also spread to the cerebrospinal fluid (the fluid that surrounds the brain and spinal cord) early in the course of the disease.

Based on an estimate of 82.5% of cases of ALL being B-ALL, there will be 540 cases of B-ALL in the UK per year, of which 381 will be in those aged < 30 years. Approximately 20% of these cases will be refractory to treatment or relapse,152 giving an estimate for young people with relapsed/refractory B-ALL in the UK of 76.

Overview of current practice

Although the management of ALL in adults and children is similar, the prognosis is different, with that in adults (aged > 30 years) being much poorer than that in the younger age group. Hence, they are generally considered as two distinct clinical groups.

With stepwise improvements in risk-adapted chemotherapy and supportive care over the past five decades, current overall cure rates of newly diagnosed ALL are approaching 90% in the developed world in children and around 50% in adults. 153 Treatment involves induction with combination chemotherapy for the attainment of CR (both clinical and haematological) followed by post-remission maintenance therapy with or without haematopoietic stem cell transplantation (HSCT) (which enhances relapse prevention, particularly in patients aged < 35 years). However, because of the morbidity and mortality risks associated with transplant, HSCT is usually reserved for high-risk patients.

US National Comprehensive Cancer Network guidelines for first-line treatment are based on risk stratification and age, as follows:154

• Philadelphia chromosome-positive (Ph+) ALL (adolescents and young adults) – chemotherapy and tyrosine kinase inhibitor (TKI), followed by allogeneic stem cell transplantation (ASCT) if an appropriate donor is available; if transplantation is not feasible, continue multiagent chemotherapy and a TKI

• Ph+ ALL (adults aged < 65 years) – chemotherapy and TKI; consider ASCT if an appropriate donor is available and the patient has a good performance status and no or limited comorbidities; if transplantation is not feasible, continue multiagent chemotherapy and a TKI

• Ph+ ALL (adults aged ≥ 65 years or with substantial comorbidities) – TKI and corticosteroids or TKI and chemotherapy (evaluate end-organ reserve, end-organ dysfunction and performance status)

• Philadelphia chromosome-negative (Ph–) ALL (adolescents and young adults) – paediatric-style multiagent chemotherapy

• Ph– ALL (adults aged < 65 years) – multiagent chemotherapy

• Ph– ALL (adults aged ≥ 65 years or with substantial comorbidities) – multiagent chemotherapy or corticosteroids (evaluate end-organ reserve and end-organ dysfunction).

However, little progress has been made in the treatment of relapsed ALL. Following initial induction and maintenance therapy most adults will relapse and long-term leukaemia-free survival is achieved in only 20–30% of cases; following relapse, response rates to further chemotherapy are low at around 20–30% and long-term OS rates of 3–24% have been reported. 155 From a UK study of 608 adult patients, OS at 5 years in newly diagnosed patients was 38% (95% CI 36% to 41%) but after relapse was only 7% (95% CI 4% to 9%). 156

Relapse is less common in paediatric ALL but accounts for the highest proportion of cancer deaths in children. 157 Studies of Nordic and Austrian data found that, of children with ALL, 25% had a first relapse, 8% had a second relapse158,159 and 2% had a third relapse. 159 Around 50% of relapsed ALL in children does not respond to salvage therapy and for these patients survival rates are < 10%. 159 In children, age and white blood cell count at primary diagnosis of ALL are the most important prognostic factors for relapse: age < 1 year or ≥ 10 years is associated with the worst prognosis. In addition, site of relapse and duration of first remission are the major criteria for the classification of patients after first relapse. 157

Therapy after relapsed ALL consists of re-induction chemotherapy followed by consolidation chemotherapy and/or HSCT. Time to relapse (length of first remission), site of relapse and ALL immunophenotype are established factors that are prognostic at first relapse and can be used to determine further treatment. 157 B-ALL has a better prognosis than T-ALL. Various regimens have been investigated and re-induction remission rates of 71–95% have been reported; the higher rates are generally associated with later first relapse. Patients who are refractory to re-induction therapy or who have a further relapse have a poor prognosis, with survival rates of < 10%. 159 Failure to achieve CR after late re-induction chemotherapy is associated with previous failures to achieve CR or short remission. The proportion of patients achieving CR has been shown to reduce with subsequent relapses: in a study of 225 patients with ALL (including 195 patients with B-ALL), the mean (standard error) CR rates were 83% (4%) for early first marrow relapse, 93% (3%) for late first marrow relapse, 44% (5%) for second marrow relapse and 27% (6%) for third marrow relapse. 152 Five-year DFS rates in second CR and third CR were 27% (4%) and 15% (7%), respectively. Although some therapies with curative intent are capable of inducing a second remission in patients refractory to previous therapy, these are often associated with high treatment-related morbidity, mortality and minimal survival. 160 Such patients are eligible for innovative therapies in Phase I or Phase II trials. Therapies for relapsed B-ALL that have been licensed by the EMA or the FDA are discussed in Review of licensed treatments for relapsed/refractory B-cell acute lymphoblastic leukaemia. In particular, clofarabine (Evoltra^®; Genzyme Europe, Naarden, the Netherlands), a purine nucleoside anti-metabolite, (which affects DNA elongation, synthesis and repair), was granted EMA marketing authorisation in 2006 for use in children and young adults with a second or greater relapse (or refractory patients). The pivotal trial of clofarabine (n = 61 with second or greater relapse) reported an overall remission rate of 20% (12/61 patients), with 16% (10/61) of patients going on to receive HSCT. 161 Clofarabine had been studied only in single-arm trials and marketing authorisation was granted ‘under exceptional circumstances’.

As discussed in Chapter 3 (see Review of the use of surrogate end points as primary outcome measures in definitive effectiveness trials of new therapeutic agents), the use of CR as an outcome is not specific at predicting which patients might subsequently relapse. In recent years, evaluation of response to therapy in B-ALL patients has become more precise with the development of methods to detect MRD. Although the FDA has concluded that the evidence base to indicate that early MRD status is the strongest predictor of long-term EFS in ALL is unequivocal, there is some uncertainty surrounding how MRD correlates with long-term outcomes in relapsed populations. The use of MRD as a surrogate end point is discussed further in Chapter 3.

Decision problem

The key aspects of the decision problem to be addressed were identified and agreed at a meeting between the academic group and a subgroup of the project advisory group (topic experts). These were based on discussion of the Phase I/II CAR T-cell trials in B-ALL,164–166 which had been identified by literature searches performed by Cell Therapy Catapult. The agreed components of the decision problem were as follows:

• Intervention – CD19 CAR T-cell therapies.

• Indication – patients with B-ALL who have relapsed (with no further planned curative chemotherapy or HSCT) or who are refractory to standard chemotherapy. As described in Overview of current practice, the treatment pathways and prognosis for patients aged < 30 years and > 30 years are very different. Consequently, the indication considered in this assessment has focused on those aged < 30 years.

• Subgroups – sources of heterogeneity such as relapsed/refractory status, previous HSCT, CAR design, dose, conditioning chemotherapy, tumour burden at the time of therapy or age of the patients may be explored.

• Comparators – best supportive care (e.g. salvage chemotherapy).

• Efficacy outcomes – response criteria such as CR, partial response/remission (PR) and MRD; OS; progression and/or EFS; persistence of CAR T-cells; HRQoL; and rates of HSCT.

• Adverse event outcomes – CRS, B-cell aplasia, febrile neutropenia and neurological effects.

Methods

Initially, studies of CD19 CAR T-cells in B-ALL were identified by staff at Cell Therapy Catapult (who are part of the project advisory group, being experts in cell-based regenerative medicines). PubMed was searched using the search terms ‘CD19’, ‘chimeric antigen receptor’, ‘CAR’ and ‘CD19 CAR’ with a cut-off date of 21 October 2014. The ClinicalTrials.gov trials registry and relevant published reviews of CAR T-cell therapies were also searched.

To identify any further relevant clinical trials we performed update searches in MEDLINE and EMBASE up to May 2015, using the same search terms as those previously described. One reviewer performed an initial screen of the abstracts and those deemed potentially relevant were then screened by a second reviewer. Google (Google Inc., Mountain View, CA, USA) searches to identify further data on already-identified trials were also undertaken. Our clinical advisor was also contacted regarding any relevant 2015 conferences where new data may have been presented. To further inform the study design details of the hypothetical data sets, the ClinicalTrials.gov trials registry was searched for ongoing trials that had commercial involvement: the focus was on trials designed with the likely aim of acquiring marketing authorisation.

Overview of studies

Three published papers were identified from the Cell Therapy Catapult searches. 164–166 No further studies were identified from the MEDLINE and EMBASE update searches. Two conference abstracts167,168 (relating to two of the published studies) and one conference video169 (relating to one of the published studies) with more up-to-date data were identified from the Google searches.

Of the planned or (other) ongoing CAR T-cell studies identified on ClinicalTrials.gov, seven had commercial involvement: three were Phase II trials, all with estimated enrolments of 67 patients; one was a Phase I/II trial with an estimated enrolment of 80 patients; and three were Phase I trials (Table 4). All were single-arm studies. Two of the Phase II trials were multicentre studies (i.e. they listed more than one centre for recruitment in the contacts and locations field) and three had a primary outcome that assessed response or remission; the time frames stated for these outcomes ranged from 9 weeks to 1 year. Only one trial reported the collection of longer-term survival data, with a stated time frame of 5 years for OS, EFS and RFS.

Efficacy results

Details of the three trials identified from the Cell Therapy Catapult searches are presented in Table 5. Two of the studies were Phase I trials164,166 and one was categorised as a Phase I/IIA trial165 (on ClinicalTrials.gov); safety was the primary outcome in all trials (one study also had maximum tolerated dose as a co-primary outcome164). All trials had recruited < 40 patients, although two were ongoing. 165,166

As can be seen from Table 5, notable clinical heterogeneity was evident both within and across trials. One study was of children and adults165 and one studied children and young adults. 164 The remaining study included only adults166 and so ultimately was not of further use for the assessment. Most patients had had a previous relapse following remission; a small proportion of patients were refractory to previous treatments. In two studies the CAR T-cell treatments were mostly used to enable patients to receive HSCT (i.e. used as a bridging therapy). 164,166 The remaining study appeared to recruit a more difficult-to-treat population, with most patients having two or more relapses and previous HSCT; here, the treatment intention may possibly have been curative. 165 CAR T-cell design also varied, with either the CD28 or the 4-1BB co-stimulatory domains being used, a difference that might explain the more prolonged persistence of circulating CAR T-cells seen in one of the studies. 165 Persistence of CAR T-cell therapies in the body can result in benefit and risk, depending on the duration of persistence.

Two surrogate end points were reported in all three trials: CR and MRD. Rates of CR ranged from around 70% to 90%. As would be expected, the rates of achieving a status of MRD were lower, ranging from around 60% to 80%, although only one trial stated the MRD threshold used, which was 0.01% (i.e. one cancer cell in 10,000 normal cells). 164 All three trials reported OS data. In one trial the probability of OS was 52% at 9.7 months. 164 The other two trials reported probabilities of OS at 6 months, which were 58%168 and 78%, respectively. 165

Adverse events

The key adverse events noted in the trials were CRS, B-cell aplasia, febrile neutropenia and various neurological effects. In two studies most patients had mild to moderate CRS,164,165 although a greater incidence of severe CRS was evident in the trial in adults. 166 Affected patients were treated with steroids or tocilizumab. Two of the three studies reported the incidence of B-cell aplasia. In one study prolonged B-cell aplasia did not occur164 and in the other B-cell aplasia occurred in all patients who had a response and persisted for up to 1 year after CAR T-cells were no longer detectable. 165 Significant proportions of patients had febrile neutropenia or neurological adverse effects such as hallucinations or altered mental status (see Table 5).

Summary issues for the target product profile and hypothetical data sets

• The B-ALL population is narrowly defined with extremely poor prognosis and limited alternative therapy options. This is likely to be typical of regenerative medicines.

• Therapy potentially offers a ‘cure’.

• There are potentially serious adverse effects of therapy.

• Limited data are available (single-arm studies).

• Appropriate comparator and control data need to be identified/generated.

Single-arm B-cell acute lymphoblastic leukaemia trials: identifying appropriate control data

As discussed in Chapter 3 (see Study biases: an overview of their importance and methods to quantify and adjust for their impact), although the results of single-arm trials can be compared with historical control data, the results of such comparisons can be considered as reliable indicators of treatment benefit only when the disease natural history is very well known, the patient population is homogeneous and the standard of care treatment has little impact on outcomes. For Evoltra, Blincyto and Marqibo, different approaches were used to devise a control data set.

For Blincyto a weighted analysis of patient-level data from 694 retrospective controls (1990–2014) was performed. 174 This study, which was of relapsed/refractory adults treated with standard of care therapy, utilised databases in several EU countries and the USA. The manufacturer of Blincyto (Amgen) also conducted a model-based meta-analysis of clinical study data to project the effect of Blincyto relative to existing therapies. For its ongoing trial in children, Amgen cited both a key paper on prognostic factors in B-ALL152 and the CR rates seen with Clolar (clofarabine), stating that the primary efficacy end point would be met if the CR + CRh (complete remission with incomplete haematological recovery) rate was at least 22.5% (suggesting efficacy similar to or greater than that for clofarabine). 172 For the Marqibo FDA submission, literature searches were performed to identify response rates in relevant patients. 173 The study reporting the best historical comparison data still had some key differences with regard to the Marqibo trial population, most notably in terms of line of treatment, eligibility for transplant and site of adjudication. 175 Comparisons were made with the closest matched subgroup of patients in the historical study: patients who received third-line single-agent treatment. 173 Clofarabine was assessed in 2006 and thus the aforementioned O’Brien study175 was not available. Instead, data were obtained from German and Dutch cancer registries; simple comparisons of median survival results were presented to the EMA. 170

Defining the sample size and maturity of evidence in the evidence sets studies

Current evidence on the efficacy of CAR T-cell therapies is limited to early Phase I/II studies with patient numbers of between 16 and 30. 164,165,179 It is anticipated that larger clinical studies will be needed to meet the minimum requirements for positive regulatory approval. This evidence is expected to come from a series of Phase II/III studies. Several previous pharmaceutical treatments for ALL have been granted regulatory approval based on efficacy and safety data from single-arm Phase II studies with sample sizes ranging from 61 to 189. This appears to be reflected in the design of planned and ongoing CAR T-cell studies in ALL. According to the ClinicalTrials.gov trials registry, there are currently three registered Phase II trials investigating the efficacy and safety of CAR T-cell therapies. The planned sample size of these trials is 67 patients (see Table 4). There is also one Phase I/II trial with an estimated enrolment of 80 patients.

The expected minimum data requirement for regulatory approval of CAR T-cell therapy was therefore set in the region of 60–80 patients. This sample size was used in both the minimum and the intermediate evidence sets. The mature evidence set was assumed to be based on trial evidence derived from a larger sample of patients than in the minimum and intermediate evidence sets. This evidence set was designed to reflect a scenario in which the evidence base for CAR T-cell therapy could include data from a more conventional RCT (or alternatively a larger uncontrolled study) with sufficient duration of follow-up to determine the longer-term efficacy for key clinical end points including OS. In practice, the sample size and maturity for such a study would be determined by a number of factors, including conventional statistical power calculations, likely accrual rates, the competitive landscape and overall study costs. In the time available, it was not feasible to formally consider these elements in estimating the anticipated sample size for this study. Instead, the sample size was based on the planned enrolment size of an ongoing Phase III trial of blinatumomab in adult ALL, identified from previous hand-searching of ClinicalTrials.gov. 180 In this study, the planned enrolment was for 400 patients to be randomised to blinatumomab or standard of care at a ratio of 2 : 1. As such, of the total 400 randomised patients, 133 would be randomised to the control arm and 267 would be randomised to blinatumomab. For the mature set, the study sample size was therefore set in the region of 120–140 patients per treated group (240–280 in total).

In using this specific sample size for the mature evidence set, we recognise that there are differences between the patients recruited into the blinatumomab trial and the specific population being considered here both in terms of age and previous history of relapse. However, for the purposes of a hypothetical exemplar this was considered to provide a reasonable basis for investigating the potential impact of increased precision.

The intermediate and mature evidence sets were also assumed to be based on trial evidence, with longer efficacy follow-up durations than the minimum set. For the minimum set, trial follow-up was based on a similar duration to that reported within existing CAR T-cell studies, with a median follow-up of approximately 10 months. For the intermediate and mature sets, trial follow-up was based on the maximum planned study duration for all Phase II CAR T-cell trials registered on the ClinicalTrials.gov trials registry. Across these studies, the longest planned follow-up period was 5 years.

A summary of the targeted sample size and level of evidence maturity considered across each of the evidence sets is provided in Table 6.

Estimating the efficacy of chimeric antigen receptor T-cell and comparator treatments in the evidence sets

For all dichotomous outcomes, including response, remission and use of HSCT, parameter estimates were extracted directly from the existing CAR T-cell and clofarabine publications. The effect of increased sample size on the variance parameter for each dichotomous outcome was modelled using a beta distribution. As these outcomes tend to be measured during the first few months of a study, it is expected that longer follow-up would not directly impact these parameter estimates.

For the OS end point, parameter estimates were derived by digitising the Kaplan–Meier (KM) curves reported in the main study publications and using the algorithm by Guyot et al. 72 to impute the patient-level time-to-event and event-type (censored or event) data. These data were then analysed using conventional semiparametric survival modelling techniques using the statistical programming platform R (version 3.0.2; The R Foundation for Statistical Computing, Vienna, Austria). 181 This included assessments of landmark survival probabilities at 6, 12 and 60 months and derivation of the HR for CAR T-cell therapy compared with standard of care therapies and restricted mean survival times (i.e. in the event of non-proportional hazards, formally tested using the Schoenfeld residual test).

Using the studies by Maude et al. 165 and Lee et al. ,164 it was possible to generate samples of between 21 and 30 patients treated with CAR T-cell therapy. However, the expected minimum sample size for regulatory approval may be in the region of 60–80 patients. Therefore, it was necessary to increase the size of the imputed data sets from between 21 and 30 to between 60 and 80. This was achieved by replicating each of the imputed data sets until the total sample size was between 60 and 80 patients for the minimum/intermediate evidence sets and between 120 and 140 patients for the mature evidence set. By creating the pooled sets in this way, the mean survival probabilities and KM plots for OS remained consistent across evidence sets, whereas the variance around those estimates was allowed to vary in line with the sample size.

For the mature and intermediate data sets it was also necessary to simulate an increase in the duration of study follow-up to account for a scenario in which CAR T-cell studies had a longer follow-up duration, up to a maximum of 5-years. This adjustment was made by adding survival time to the imputed patient records, which were censored after the last recorded event in each study. These patients were subsequently assumed to be re-censored at their new survival time. The same approach was applied in both the CAR T-cell and the clofarabine arms. By extending the study time of patients who were censored after the last recorded event, the following assumptions were made:

• patients who were alive and censored at the end of the studies were likely to be ‘cured’ of ALL, such that they would also be alive and censored at the end of the longer follow-up period

• patients who were censored prior to the last event were assumed to have been lost to follow-up, such that additional trial follow-up would not lead to further information on the timing of death (or re-censoring) in those individual patients.

This approach ensured that the KM curves remained consistent across the evidence sets. The net result of this adjustment is that the more mature evidence sets contain more information on the long-term survival of those alive and censored at the end of the current studies. An illustration of this approach is provided in Figure 1.

FIGURE 1.

Illustration of adjustment to the data sets to account for additional study maturity.

It is important to highlight that, in practice, additional follow-up time points in trials would be likely to result in changes to the KM curves and estimates of survival benefit. To predict these changes would require access to IPD to elicit the characteristics of patients who are censored at shorter follow-up times and to then predict the unobserved event time for the censored patients, conditional on their characteristics. With the imputed data it is not possible to identify the characteristics of those who are censored and there are too few data to develop a prediction equation for the unobserved event times.

Bridge to haematopoietic stem cell transplantation target product profiles

Data on the evidence sets assumed for the clinical efficacy of CAR T-cell therapy as a bridge to HSCT are reported in Table 7 (OS) and Table 8 (dichotomous end points, adverse events). The associated KM plots are reported in Figure 2.

FIGURE 2.

Kaplan–Meier plots of time from randomisation to death for CAR T-cell and standard of care therapies in the different scenarios: (a) Bridge to HSCT – current trial follow-up; (b) curative intent – current trial follow-up; (c) bridge to HSCT – mature trial follow-up; and (d) curative intent – mature trial follow-up.

Curative intent target product profiles

Data on the evidence sets assumed for the clinical efficacy of CAR T-cell therapy as a curative treatment option are reported in Table 9 (OS) and Table 10 (dichotomous end points, adverse events). The associated KM plots are reported in Figure 2.

Implications for model conceptualisation

The systematic and pragmatic searches highlighted a number of potential implications for our evaluation. Within existing studies, it is clear that the main benefit of existing treatments has been related to their ability to provide a ‘bridge’ to HSCT. The primary factor determining cost-effectiveness in the reviewed literature was the increased likelihood of receiving HSCT with a new treatment and the associated assumptions made regarding subsequent health gains associated with transplantation. Only a limited survival gain was attributed to patients who did not subsequently receive HSCT, such that none of the treatments reviewed appeared to be cost-effective as a palliative option.

The key structural assumptions employed within these studies were the potentially curative effect of HSCT and the short life expectancy assumed for the comparator treatments (best supportive care/palliative treatment alone) derived from historical controls. The majority of studies assumed a ‘cure point’ associated with HSCT, although the timing was different across studies. The ‘cure point’ was assumed to represent the time at which patients are assumed to no longer be at risk of disease relapse. The study by Costa et al. 182 assumed that at 5 years post transplantation the patient will be free of any procedural mortality risk or any risk of disease recurrence. In Lis et al. 183 and the AWMSG appraisal of nelarabine185 this cure point was assumed to be 2 years after HSCT, whereas in the AWMSG appraisal of clofarabine171 the cure point was assumed to be 1 year after HSCT.

The studies also differed in the assumptions made concerning subsequent survival after the ‘cure point’. Costa et al. 182 acknowledged that long-term ALL survivors are likely to be subject to significant comorbidities over their remaining lifetime despite being leukaemia free. To account for the impact of comorbidities, an assumption was made that the long-term survival of ALL patients would be 50% less than that in the general population. The authors acknowledge that this was an arbitrary adjustment because of the lack of data on the long-term mortality rate in long-term survivors of ALL reported at the time. In contrast, the study reported by Lis et al. 183 and the clofarabine submission171 effectively assumed no additional comorbidities (i.e. beyond those experienced by the general population) beyond the ‘cure point’. Hence, patients were subsequently assumed to return to the age-adjusted mortality risk and utility of the general population. The AWMSG raised concerns that, not only was this assumption insufficiently justified but, also, the model was very sensitive to changes in the long-term survival probability.

In the absence of RCT data, each model incorporated historical control data as the basis to inform outcomes associated with the comparator (best supportive care, palliative care and clofarabine within a scenario for the submission for nelarabine). 171,185–187 However, insufficient details were reported regarding the source of the historical control data used and whether attempts were made to identify possible biases or to formally account for potential confounding.

The existing cost-effectiveness literature is limited in ALL. No completed NICE appraisals of licensed treatments for ALL were identified. Furthermore, of the studies published, none was reported in sufficient detail to provide a suitable basis for informing the exemplar application. In the absence of previous NICE appraisals or sufficient reporting within existing publications, the development of a de novo model to inform the exemplar application was considered necessary. Full details of this are reported in the next chapter.

Patient population

In this evaluation we assessed the cost-effectiveness of CAR T-cell therapy in the treatment of children and young adults with two or more relapses or refractory ALL. The baseline demographic characteristics of this patient group are summarised in Table 12.

Comparator

The comparator treatment to CAR T-cell therapy was defined as standard of care. In the base case, the standard of care treatment was assumed to be clofarabine. The mean cost for a course of clofarabine treatment is approximately £43,200 per patient. 171

As part of a separate sensitivity analysis, the standard of care treatment was assumed to be FLAG-IDA. The mean cost for a course of FLAG-IDA treatment is approximately £3803 per patient (see Administration and monitoring costs).

Bridge to haematopoietic stem cell transplantation scenario

Key structural assumptions

The bridge to HSCT model consists of a decision tree model (days 0–56) and a series of partitioned survival models (day 56 to lifetime) that, when combined, provide an estimate of the lifetime costs and effectiveness of treatment in ALL. An illustration of the structure of the model is provided in Figure 3.

FIGURE 3.

Illustration of the bridge to HSCT model.

The short-term decision tree component of the model consists of three chance nodes that represent a series of clinically relevant events that may occur during the first 56 days (2 months) of treatment:

1. remission status at day 28 (remission, no remission or death)

2. MRD status at day 28 (negative or positive)

3. transplantation status at day 56 (HSCT or no HSCT).

These events are considered to be prognostic of the duration and quality of life of patients with ALL and were therefore included in the model to link short-term measures of trial efficacy to longer-term health outcomes. The three nodes of the decision tree are sequenced in the order of remission, MRD status and transplantation status.

At the first chance node, the hypothetical cohort are distributed across three states: remission, no remission or death. In the model, remission is defined using the criteria applied in the CAR T-cell and clofarabine clinical trials. 164,178 CR is defined as < 5% marrow blasts by flow cytometry, an absence of circulating blasts and no extramedullary sites of disease with an absolute neutrophil count of ≥ 1000/µl and a platelets count of ≥ 100,000/µl. In accordance with both the Lee et al. 164 study and the Jeha et al. 178 study, remission status is determined at day 28 (month 1) of the simulation.

At the second chance node, patients with remission are reassigned to one of two states: remission and MRD negative or remission and MRD positive. A MRD-negative status is defined as < 0.01% marrow blasts and a MRD-positive status is determined by marrow blasts of between 0.01% and 5% (at > 5% patients are no longer in remission).

At the third and final node (day 56), all patients are assigned to states corresponding to the use of HSCT (HSCT vs. no HSCT). The final determination of health status (remission – MRD – HSCT) was assumed to occur at day 56 (month 2) of the simulation. This time period was chosen based on the mean time from CAR T-cell therapy to HSCT estimated from data reported in the study by Lee et al. 164 (mean 54 days, 95% CI 45 to 77 days).

At the end of the decision tree phase, the cohort is assigned to six mutually exclusive states (presented in order of best prognosis):

1. HSCT – remission and MRD negative

2. HSCT – remission and MRD positive

3. HSCT – no remission

4. no HSCT – remission

5. no HSCT – no remission

6. death.

After day 56, the long-term survival of the cohort is modelled through a series of related partitioned survival models (see Figure 3) that are used to model the long-term outcomes of treatment (day 56 to lifetime). The model includes four distinct partitioned survival models that are used to evaluate survival in the following groups:

1. HSCT – remission and MRD negative

2. HSCT – remission and MRD positive

3. HSCT – no remission

4. no HSCT.

Haematopoietic stem cell transplantation recipients with MRD-negative status prior to transplantation are assumed to have the best prognosis in terms of long-term survival. An increasing level of marrow blasts is assumed to be associated with a lower probability of long-term survival, such that HSCT recipients with MRD-positive status have (on average) a worse survival prognosis than MRD-negative patients. HSCT recipients who fail to achieve remission prior to transplantation are assumed to have the poorest prognosis of all HSCT patients.

For patients who did not receive HSCT, the probability of OS was significantly lower than for HSCT patients. It was assumed that CR was not associated with improved probabilities of survival in non-HSCT patients. This assumption was made on the basis that in the bridging scenario it is through HSCT (and not remission in the absence of HSCT) that meaningful gains in survival can be achieved. The impact of this assumption on the results of the evaluation was tested in the one-way sensitivity analyses, in which it was assumed that non-remission non-HSCT patients had an inferior survival prognosis to remission non-HSCT patients.

At year 5 of the simulation, those who were alive were subsequently assumed to be long-term survivors of ALL. From this point forward, the cohort was considered to be effectively ‘cured’ of ALL and experienced the mortality risk profile consistent with that of a long-term survivor of ALL. The mortality risks after year 5 were therefore modelled based on general population age- and sex-adjusted all-cause risks of mortality adjusted for excess morbidity and mortality reported in cohorts of long-term survivors of ALL. 188 The approach is more formally described in Model development.

The model also included treatment-related adverse events. These include events such as CRS, encephalopathy, hypotension, febrile neutropenia, neutropenia, anaemia, thrombocytopenia, leukopenia, hypokalaemia and hypophosphataemia. The costs and consequences of these events were assumed to occur at the start of the evaluation. As prolonged B-cell aplasia did not occur in the Lee et al. 164 study, the costs and consequences of this were not included within the bridge to HSCT scenario. The key structural assumptions applied in the model are outlined in Table 14.

TABLE 14 - Summary of the key modelling assumptions in the bridge to HSCT responder model

Input	Assumption
Surrogate relationship between MRD status and HSCT	A lower marrow blast status prior to transplantation (as captured through MRD status) is associated with a higher probability of experiencing sustained remission and long-term survival benefits in ALL
HSCT	All HSCT events were assumed to occur at day 56 of the simulated time horizon. No further HSCT events were permitted after this point
Survival during the first 5 years of the evaluation time horizon	Survival post HSCT was modelled based on the constant transition probability. There is no difference in survival between remission non-HSCT patients and non-remission non-HSCT patients
Survival after the first 5 years of the evaluation time horizon	All patients alive at 5 years post HSCT are considered to be long-term survivors of the disease. Long-term survivors of ALL experience excess morbidity and mortality compared with the general population
Treatment/retreatment	In the base case it was assumed that each patient would receive a single full course of therapy. Retreatment with CAR T-cell therapy or standard of care therapy was not permitted in the base case but was considered in the sensitivity analysis
Treatment effect	Compared with standard of care therapy, CAR T-cell therapy improves the probability of remission, the probability of a MRD-negative status and the probability of successful HSCT. The clinical parameter estimates used to inform the models and TPPs can be generalised to the UK NHS
Patient follow-up	After HSCT, patients receive ongoing care and rehabilitation up to 2 years post HSCT. Patients who do not receive HSCT are assumed to require hospitalisation prior to death
Adverse events	Treatment-related adverse events were considered in the evaluation and included events such as CRS, the incidence of which is expected to increase with the use of CAR T-cell therapy. The costs and health consequences of adverse events were accrued at the start of the evaluation
HRQoL	Patients who achieve remission status are assigned a higher utility weight than patients who do not achieve remission. Transplantation is associated with a one-off decrement to HRQoL

Clinical justification for the structure of the model

The conceptual structure of the bridge to HSCT model is based on an assumed relationship between HSCT use and final clinical benefits and the assumption that the effectiveness of HSCT is dependent on MRD status prior to transplantation.

Allogeneic HSCT is a potentially curative treatment option in patients with ALL. However, the long-term benefits of HSCT are uncertain, with some patients experiencing long-term benefits, including the effective cure/suppression of ALL, and other patients experiencing relapse and/or mortality shortly after transplantation. In this evaluation, survival benefits are established through remission and MRD status prior to HSCT.

Several studies have investigated the relationship between remission/MRD status prior to HSCT and the long-term outcomes of HSCT therapy. 110,111,189 These studies have shown, to varying degrees, that MRD status prior to HSCT appears to be an important prognostic determinant of long-term RFS and OS, with MRD-negative (< 0.01% marrow blasts) patients experiencing superior survival compared with MRD-positive (> 0.01% to 5% marrow blasts) patients, including within studies of children with relapsed ALL. These data support the assumption of a continuous relationship between MRD level prior to HSCT and 5-year survival probability.

For patients who do not receive HSCT, the long-term outcomes of treatment are generally poor. In the study by von Stackelberg et al. ,160 the median survival in refractory patients who failed to respond to induction therapy and who went on to receive palliative care was 89 days (equivalent to 3.17 months).

In the model, it was assumed that all patients who did not receive HSCT (including remission and non-remission patients) went on to receive palliative care, having exhausted all treatment strategies that may be curative. Remission status was not considered to be prognostic of survival in the non-HSCT population, such that non-HSCT patients who achieved remission were assumed to be at the same risk of mortality as non-HSCT patients who failed to achieve remission. However, as discussed later in Model inputs: utilities, all patients with remission were assigned an improved health utility compared with those who failed to achieve remission. These benefits were, however, assumed not to extend to improved life expectancy.

In previous economic evaluations in ALL (reviewed in Chapter 6), it had been assumed that survivors of ALL experience the same mortality risk profile as that of the general population. This assumption implies that there is no excess mortality or morbidity risk associated with their previous illness. This assumption is not supported by the published literature, which generally reports excess mortality and morbidity among the long-term ALL survivor population compared with match-adjusted individuals without ALL (i.e. siblings). 190,191 In the model, the risk of mortality assigned to survivors of ALL was set equal to the general population background all-cause mortality risk profile, with an adjustment for an increased mortality risk among survivors of ALL.

The point at which patients were assumed to be long-term survivors of ALL (5 years) was based on the definition used in a number of published studies reporting long-term survival data in ALL. None of these studies provides an explicit rationale for selecting 5 years as the cure point and, to our knowledge, there appears to have been no published attempts to empirically justify the widespread use of the 5-year cure point. However, across a number of studies, the KM curves for post-HSCT survival appear to stabilise within the 5-year time frame, such that the curve becomes flat and the incidence of death reduces to near zero.

Efficacy parameter estimates

In the decision tree component of the model, the data for remission, MRD and HSCT status of the modelled cohort were derived from the separate evidence sets reported in Chapter 5 (see Bridge to haematopoietic stem cell transplantation target product profiles). The key assumptions required to generate the estimates for the evidence sets are outlined in Table 15. A continuity correction was applied to particular estimates to take account of the low numbers (e.g. when 0 events were recorded) in the probabilistic analysis.

TABLE 15 - Parameter estimates for the bridge to HSCT scenario

Node parameter 1	Node parameter 2	Node parameter 3	Estimate (%)	Probabilistic distribution	Minimum data set, n	Intermediate data set, n	Mature data set, n	Key assumptions
CAR T-cell therapy
Remission			66.7	Dirichlet distribution	63	63	126	Remission probability based on 14 of 21 patients achieving remission in Lee et al.164 By day 56, one of 21 patients had died (5%). All remaining patients (28.6%) were assumed to have non-remission
No remission			28.3
Death (day 0–56)			5.0
Remission	MRD negative		85.7	Beta distribution				12 patients in Lee et al.164 had a MRD-negative status. In total, 14 patients were in remission. Thus, 12 of 14 patients were MRD negative and in remission
	MRD positive		14.3
Remission	MRD negative	HSCT	83.3	Beta distribution				12 patients were MRD negative in Lee et al.,164 of whom 10 underwent HSCT
		No HSCT	16.7
Remission	MRD positive	HSCT	0	Beta distribution				In Lee et al.,164 no MRD-positive patients received HSCT
		No HSCT	100
No remission	HSCT		0	Beta distribution				In Lee et al.,164 none of the patients who failed to achieve remission received HSCT
	No HSCT		100
Clofarabine
Remission			11.5	Dirichlet distribution	61	61	122	Remission based on seven of 61 patients achieving CR in Jeha et al.178 By day 56, 32.7% of patients had died (based on digitisation of the published KM curve). All remaining patients (55.8%) were assumed not to be in remission
No remission			55.8
Death (day 0–56)			32.7
Remission	MRD negative		14.3	Beta distribution				One patient with remission had undergone HSCT and was considered to be in long-term remission (> 200 days alive). This patient was assumed to have a MRD-negative status (MRD status not reported in Jeha et al.178). Thus, one of seven remission patients was MRD negative
	MRD positive		85.7
Remission	MRD negative	HSCT	100.0	Beta distribution				Assumption that all patients who were MRD negative went on to receive HSCT
		No HSCT	0.0
Remission	MRD positive	HSCT	16.7	Beta distribution				Seven patients were in remission, of whom one was assumed to be MRD negative and had undergone HSCT. Of the remaining six patients (MRD positive), a further one patient had undergone HSCT. Thus, one of six MRD-positive patients had undergone HSCT178
		No HSCT	83.3
No remission	HSCT		20.6	Beta distribution				In total, nine patients underwent HSCT in Jeha et al.178 Two HSCT patients were in remission, with the remaining seven HSCT patients not in remission. During the initial 56 days, an estimated 34 (55.8%) patients were not in remission. Thus, seven of 34 no-remission patients underwent HSCT
	No HSCT		79.4

Figure 4 presents the proportion of patients occupying each state at the end of the decision tree model. The model predicts that 48% of patients receiving CAR T-cell therapy and 15% of patients receiving standard of care treatment will receive HSCT. All patients who underwent HSCT following CAR T-cell therapy were assumed to have a MRD-negative status. In contrast, most patients who underwent HSCT after receiving clofarabine had not achieved remission (11.5%) prior to transplantation, with only a small proportion of patients receiving HSCT after CR (1.6% MRD negative, 1.6% MRD positive).

FIGURE 4.

Proportion of patients occupying each state at the end of the decision tree model (day 56) by CAR T-cell and comparator group: bridge to HSCT scenario.

With the structural link included within the model, it was necessary to use external data rather than the evidence sets themselves for the purposes of extrapolation and estimating lifetime mortality. This was necessary because the existing survival data for CAR T-cell therapy were not reported in terms of being conditional on remission, MRD or HSCT status. Hence, the parameter estimates for the partitioned survival analyses were sourced from two external studies: Leung et al. 111 for the post-HSCT survival probabilities and von Stackelberg et al. 160 for the non-HSCT survival probabilities. A summary of the survival rates is provided in Table 16.

TABLE 16 - Survival rates in patients who receive HSCT based on MRD and remission status: bridge to HSCT scenario

NA, not applicable.

After 5-years, all patients alive are assumed to be long-term survivors of ALL.

Assumed to be the same standard error as in the MRD-positive analysis.

Treatment status	Status prior to treatment	Exponential rate parameter (standard error)	Sampling distribution used in probabilistic analysis	Proportion alive and considered ‘effectively’ cured at year 5 (%)	Mean time to death following HSCT (years)a	Notes	Source
HSCT	MRD negative (< 0.01% bone marrow blasts)	–	NA	99.0	43.70	Based on all-cause mortality, with adjustment for excess mortality in ALL survivors	Assumption
	MRD positive (> 0.01% to 5% bone marrow blasts)	0.0121 (0.0232)	Log-normal (applied to rate)	48.5	22.43	Leung et al.111 report a 5-year post-HSCT survival probability of 48.5% in patients with a MRD-positive status. The 5-year cumulative probability was converted to a monthly rate using the equation [–(1/60) × log(0.485)]	Leung et al.111
	No remission (> 5% to 25% bone marrow blasts)	0.0175 (0.0232b)	Log-normal (applied to rate)	35.1	16.74	5-year post-HSCT survival probability estimated by fitting a linear regression model to the survival data by MRD status, reported in Leung et al.111 The independent variable in the regression was the cumulative hazard rate at year 5 and the dependent variable was the mid-point of each MRD category (on the log to base 10 scale). To predict the cumulative hazard at year 5 for patients without remission prior to HSCT the mid-point MRD level of 15% was used
No HSCT	All patients	0.2425 (0.2085)	Log-normal (applied to rate)	0	0.35	See main text	Von Stackelberg et al.160

In the base case it was assumed that all transplant recipients in remission and with a MRD-negative status prior to HSCT reverted to the same mortality rates as seen in long-term survivors of ALL, from the time point of HSCT. Employing this assumption, as opposed to using the data reported in Leung et al. 111 for this population, provided a more consistent prediction of survival data from Lee et al. ,164 in which it was reported that all 10 HSCT recipients with a MRD-negative status were leukaemia free and alive at the end of study follow-up.

Transplant recipients in remission and with a MRD-positive status were assumed to have an inferior long-term survival prognosis compared with those who were MRD negative. Similarly, recipients who failed to respond to therapy were assumed to have an inferior long-term prognosis compared with those who responded (including MRD-positive and MRD-negative recipients). Parameter estimates were obtained from Leung et al. 111 and were modelled assuming an exponential distribution for time to death.

The Leung et al. 111 data were used in the base-case analysis as this was the only study identified in the literature review that reported post-HSCT survival in patients who failed to achieve remission (marrow blasts > 5.0%). The parameter estimate for no-remission HSCT patients forms an important part of predicting the long-term survival benefits of standard of care therapy, as approximately 11% of the standard of care population underwent HSCT, despite having failed to achieve CR (vs. 0% of the CAR T-cell trial population).

For patients who do not receive HSCT (including remission and non-remission patients), long-term survival was modelled using data from von Stackelberg et al. 160 A series of parametric survival functions was fitted to estimates of patient-level data generated from the published KM curve. According to goodness-of-fit statistics, the best-fitting distribution was the log-normal. However, when the function was applied in the model, the predicted OS for the total CAR T-cell and standard of care populations became visibly disjointed, with the risk of mortality in the decision tree phase being significantly greater than the risk being applied at the start of the partition survival phase. Consequently, there was an uncharacteristic ‘plateau’ in the modelled survival curve between day 56 (month 2) and day 84 (month 3). This plateau effect was caused by an initially low probability of death predicted from the von Stackelberg et al. 160 data.

Because of the implausible nature of the survival curve, an alternative survival distribution for von Stackelberg et al. 160 was selected in the base case. To be consistent with the approach used in modelling post-HSCT survival,111 the exponential distribution was chosen for the base-case analysis. The mean time to death with the exponential function was 0.35 years, which is consistent with the mean time to death estimated using the log-normal distribution (0.34 years).

The validation of the responder model in predicting the outcomes of the Lee et al. 164 and Jeha et al. 178 studies was assessed by comparing the predicted survival probabilities from the model with the KM data extracted from these studies. As shown in Figure 5, the final model appears to provide an accurate prediction of reported survival for both CAR T-cell therapy and the comparator.

FIGURE 5.

Model prediction (lines) compared with reported KM curves of OS: bridge to HSCT scenario.

The background all-cause mortality risks were obtained from the interim life tables published by the UK ONS. 188 The ONS data report annual all-cause mortality rates by sex and age (yearly increments from 0 to 100 years). A sex-averaged mortality risk was derived based on a cohort that was 33.3% female (n = 7/21). 164 An adjustment factor for excess mortality in ALL survivors was also incorporated and modelled using data from MacArthur et al. 191 [standardised mortality ratio (SMR) 9.1, 95% CI 7.8 to 10.5]. These data were combined using the following equation:

⁽¹⁾ TP ( age ) = 1 − e ( − ( MR ( female , age ) × P ( female ) + MR ( male , age ) × P ( male ) ) × 1 12 × SMR ) ,

where TP(x) is the monthly transition probability for the cohort with average age x, MR(y, x) is the ONS all-cause mortality rate for sex y and average age x, P(y) is the proportion of the cohort with sex y, and SMR is the SMR for long-term ALL survivors compared with the general population. The factor of 1/12 was included to convert the annualised mortality rates from the ONS to monthly rates, and probabilities, for use in the model. The mortality risk was assumed to remain constant within each year of the cohort’s age.

Curative intent model

Structural assumptions

A simple three-state partitioned survival model was developed to assess the cost-effectiveness of CAR T-cell therapy used with curative intent. The three health states included in the model were alive and event free, alive with relapsed disease and death. An illustration of the structure of the model is provided in Figure 6.

FIGURE 6.

Illustration of the partitioned survival model structure. (a) Illustration of the state structure; and (b) illustration of partitioned survival.

The health state of alive and event free included all patients who either had stable disease or had responded to therapy. The health state of alive with relapsed disease included patients who had failed induction therapy, had relapsed after previously responding to treatment or had developed second malignancies. This definition is based on the criteria used in the UK ALL study. 115

State occupancy in the model was derived using the partitioned survival technique. This involved the direct extrapolation of EFS and OS curves, which were then used to estimate the proportion of patients occupying each of the three states using the following equations:

⁽²⁾ alive and event free ( t ) = P ( E F S , t ) ,

⁽³⁾ alive with relapsed disease ( t ) = ( P ( OS , t ) − P ( EFS , t ) ,

⁽⁴⁾ death ( t ) = 1 − P ( O S , t ) ,

where P(event,t) is the cumulative survival probability for the event at time t.

Data on EFS were not available from either the CAR T-cell164 or the clofarabine178 studies. In the absence of data, the EFS curve was derived from the available OS data, through assuming a proportional relationship between EFS and OS. This relationship is justified on the basis that EFS is highly correlated with OS as it includes death prior to recurrence.

In the short term it was assumed that the cumulative hazard function for EFS would be proportional to the cumulative hazard function for OS. This was modelled based on data from the UK ALL study. 115 The proportional relationship between EFS and OS is not expected to continue indefinitely, given the potential for cure of disease and the expectation that after a finite period of time all patients alive in the simulation would also be free of relapsed disease (EFS = OS). This is equivalent to saying that, at some point in time, all patients who are alive are long-term survivors of ALL. Therefore, in the model, the proportional relationship between EFS and OS was assumed to continue up to year 5 of the simulation (the assumed point of ‘effective’ cure in ALL). After year 5, the cumulative survival probabilities for EFS were assumed to be flat up to the point at which EFS is equal to OS. In all cases, EFS was always assumed to be less than or equal to OS to avoid a negative number of patients being assigned to the relapsed disease state.

In common with the bridge to HSCT scenario, at year 5 of the simulation, those who were alive in the curative intent model were also subsequently assumed to be long-term survivors of ALL. From this point forward, the cohort was considered to be effectively ‘cured’ of ALL and experienced the mortality risk profile consistent with a long-term survivor of ALL. The mortality risks after year 5 were also modelled based on general population age- and sex-adjusted all-cause risks of mortality adjusted for excess morbidity and mortality reported in cohorts of long-term survivors of ALL.

The model evaluation also included the costs and consequences of treatment-related adverse events, which included CRS and B-cell aplasia, whose occurrence is specifically associated with CAR T-cell therapy. Other events captured in the model include encephalopathy, hypotension, febrile neutropenia, neutropenia, anaemia, thrombocytopenia, leukopenia, hypokalaemia and hypophosphataemia. All events, with the exception of B-cell aplasia, were assumed to occur at the time of treatment initiation and to resolve within the first year of therapy. The cost–consequences of these events were therefore captured at the start of the evaluation.

The occurrence of B-cell aplasia in patients treated with CAR T-cells is an expected consequence of CAR T-cell therapy and is linked to the proliferation of CAR T-cells and the associated durability of the clinical effect. Consequently, for some patients, treatment of B-cell aplasia is expected to persist beyond the first year post CAR T-cell therapy. To capture this in the model, a series of survival models was fitted to data on the time to CD19 positivity or relapse reported in Maude et al. 165 and used to predict the proportion of patients requiring treatment for B-cell aplasia. The primary assumptions made in the curative intent model scenario are summarised in Table 17.

TABLE 17 - Summary of key modelling assumptions: curative intent scenario

Input	Assumption
Survival during the first 5 years	Survival was modelled based on a weighted average survival distribution
Survival after the first 5 years	All patients alive at 5 years are considered to be long-term survivors of the disease. Long-term survivors of ALL experience excess morbidity and mortality compared with the general population
Treatment/retreatment	In the base case it was assumed that all patients received a single full course of therapy. Retreatment with CAR T-cell therapy or standard of care therapy was not permitted in the base case, but was considered in the sensitivity analysis
Treatment effect	Treatment with CAR T-cell therapy is assumed to lead to an increase in the number of patients achieving a sustained cure for ALL and therefore extend the life expectancy of patients with ALL. The clinical parameter estimates used to inform the models and TPPs can be generalised to the UK NHS
Adverse events	Treatment-related adverse events were considered in the evaluation and included CRS and B-cell aplasia. The costs and health consequences of all adverse events except B-cell aplasia were accrued at the start of the evaluation. The costs of B-cell aplasia were modelled by estimating the probability of patients having B-cell aplasia over time, using data from Maude et al.165
HRQoL	Patients who are event free are assigned a higher utility weight than patients who have relapsed disease. Transplantation is associated with a one-off adjustment to utilities

Efficacy parameter estimates: partitioned survival model

The primary data sources for OS in the curative intent model were the same imputed patient data used to derive the evidence sets reported in Chapter 5 (see Curative intent target product profiles). Each separate evidence set was then analysed using parametric survival modelling to inform the 5-year survival estimates and projections applied within the cost-effectiveness analyses. The parametric analyses were undertaken using the FlexSurv package in the statistical programming platform R (version 3.0.2).

A series of survival distributions was considered in the analysis, including exponential, log-normal, Weibull and Gompertz. Because of the potential curative nature of CAR T-cell therapy (and therefore the potential for an unconventional hazard function), a series of flexible cubic spline models was also considered in the analysis. The cubic spline models were based on those developed by Royston and Parmar. 192 Cubic spline models expressed on the proportional odds scale were used as they appeared to converge to an optimised solution more frequently than the proportional hazards or probit variants of the cubic spline model. A series of one-, two-, three- and four-knot spline models was considered. The knots were evenly distributed across the time scale of the study, as per the default settings for the FlexSurv package in R.

Separate curves were fitted to the hypothetical CAR T-cell data and the comparator data to allow both the shape and the scale of the distribution to vary between these. Alternative options include fitting proportional hazards models to a data set containing both treatments and including a covariate in the regression for treatment assignment. This alternative approach was not considered here, given that an earlier assessment of the validity of the proportional hazards assumption illustrated that this assumption may not consistently hold across all evidence sets.

Within cost-effectiveness studies it is common practice to use a single survival distribution in the base-case analysis. This is chosen based on goodness-of-fit statistics, the fit of each distribution to the KM curves and the clinical plausibility of subsequent model projections over the full time horizon. However, it is unlikely that a single survival distribution can adequately characterise uncertainties over the longer-term extrapolation period. The robustness of the ICER estimates to alternative distributions can be considered within separate sensitivity analyses or scenarios. However, transparency concerns may exist regarding this approach if their weighting is not explicitly specified in subsequent policy decisions.

To more formally account for the uncertainty surrounding choice of survival distribution, a model-averaging approach was adopted using the methods outlined in Jackson et al. 193 This technique involves the parameterisation of uncertainty surrounding the choice of distribution through including all plausible survival functions as part of a weighted distribution and sampling both the parametric uncertainty associated within each distribution and the uncertainty (or weights) surrounding the choice of preferred method. Through the probabilistic analysis, it is therefore possible to estimate the joint distribution of uncertainty around the parameter estimates and the choice of survival function.

Each model is assigned a weight that represents the adequacy of that distribution in predicting the lifetime survival of the modelled cohort in comparison to all other distributions considered in the model. There are a number of measures of model adequacy that can be considered. Examples include statistical adequacy measures such as the Akaike information criterion (AIC) and Bayesian information criterion and expert judgement. The weights considered in this evaluation were based on AIC scores. As outlined in Jackson et al. ,193 the AIC value reported from each survival distribution was converted to a probability weight (w_k) using the following equations:

⁽⁵⁾ A k = e ( −0.5 × AIC ) ,

⁽⁶⁾ w k = A k Σ A k .

The weighted distribution was then applied in the base-case analysis. Different model weights and parameter estimates were considered across the three different data sets, as outlined in the following sections.

Minimum data set

A summary of the goodness-of-fit statistics for each distribution fitted to the imputed survival data across each of the evidence sets is provided in Table 18.

TABLE 18 - Summary of goodness-of-fit statistics and weights for survival distributions: minimum set

Distribution	CAR T-cell therapy		Standard of care
	AIC	AIC-based weight (%)	AIC	AIC-based weight (%)
Exponential	127.91	1.9	302.15	0.1
Weibull	129.88	0.7	303.31	0.0
Gamma	129.91	0.7	304.09	0.0
Gompertz	139.21	0.0	303.01	0.0
Log-normal	128.70	1.3	291.00	13.8
Spline with a single knot	121.02	60.1	288.65	44.6
Spline with two knots	122.97	22.7	290.41	18.5
Spline with three knots	124.93	8.5	291.32	11.7
Spline with four knots	126.44	4.0	291.42	11.2

According to the AIC statistic, the distribution with the best goodness of fit to the CAR T-cell data was the spline model with a single knot (AIC = 121.02), followed by the spline model with two knots (AIC = 122.97). The spline model with a single knot was assigned the highest single weight of 60.1% and was followed by the two-knot (22.7%), three-knot (8.5%) and four-knot (4.0%) spline configurations. A visual comparison of the survival data based on the weighted distribution and several single distributions is reported in Figure 7.

FIGURE 7.

Plot comparing fit of weighted distribution and key single distributions to CAR T-cell data: curative intent scenario.

Because of the limited maturity in the minimum data set, there was considerable variation in the predicted long-term survival of the modelled cohort, as shown by the spread of survival trajectories in Figure 7. Although the ‘best-fitting’ spline models appeared to generate a robust fit to the data over the first 3 months of the study, the functions were not able to accurately predict the tail of the distribution. In this case, the ‘best-fitting’ model underestimated the KM probabilities from month 18 of the simulated time horizon. The weak fit of the model to the tail of the KM curve is partly driven by the limited data available to support the continued flattening of the curve. As shown in the following section, with additional data maturity, the parametric models tend to provide a better prediction of the tail of the KM curve as there are more data to support the long-term flattening of the survival curve.

In the standard of care group, the distribution with the optimal predictive validity as judged using the AIC was also the spline model with a single knot (AIC = 288.65). A weight of 44.6% was assigned to the spline model with a single knot, followed by weights of 18.5% for the spline model with two knots and 13.8% for the log-normal model. A visual comparison of the survival data based on the weighted distribution and several single distributions is reported in Figure 8.

FIGURE 8.

Plot comparing fit of weighted distribution and key single distributions to standard of care data: curative intent scenario.

Intermediate and mature data sets

Summaries of the goodness-of-fit statistics and weights are reported for the intermediate and mature evidence sets in Tables 19 and 20, respectively.

TABLE 19 - Summary of goodness-of-fit statistics and AIC-based weights for survival distributions: intermediate set

Distribution	CAR T-cell therapy		Standard of care
	AIC	AIC-based weight (%)	AIC	AIC-based weight (%)
Exponential	157.43	0.0	345.45	0.0
Weibull	147.51	0.0	329.54	0.0
Gamma	148.64	0.0	337.70	0.0
Gompertz	161.81	0.0	343.38	0.0
Log-normal	143.41	0.0	308.08	0.1
Spline with a single knot	125.18	61.2	296.00	56.0
Spline with two knots	126.95	25.3	298.08	19.7
Spline with three knots	128.84	9.8	299.58	9.4
Spline with four knots	130.84	3.6	298.66	14.8

TABLE 20 - Summary of goodness-of-fit statistics and AIC-based weights for survival distributions: mature set

NA, not applicable.

Distribution	CAR T-cell therapy		Standard of care
	AIC	AIC-based weight (%)	AIC	AIC-based weight (%)
Exponential	312.85	0.0	688.89	0.0
Weibull	291.02	0.0	655.07	0.0
Gamma	293.28	0.0	671.39	0.0
Gompertz	339.75	0.0	644.99	0.0
Log-normal	282.81	0.0	612.15	0.0
Spline with a single knot	244.36	60.3	586.00	34.0
Spline with two knots	245.89	28.0	588.17	11.5
Spline with three knots	247.65	11.7	589.17	7.0
Spline with four knots	NA	NA	585.32	47.6

The additional maturity of the data in these evidence sets and the superior AIC statistics associated with the flexible spline models resulted in none of the standard distributions being assigned a weight > 0.1%. The different levels of precision resulted in small differences in the weights assigned to the spline models across the intermediate and mature evidence sets.

Visual comparisons of the survival data based on the weighted distribution and several single distributions are reported in Figure 9.

FIGURE 9.

Plot comparing fit of weighted distribution and key single distributions to KM curves across the data sets: curative intent scenario. (a) CAR T-cell therapy minimum data set; (b) standard of care minimum data set; (c) CAR T-cell therapy intermediate/mature data set; and (d) standard of care intermediate/mature data set.

In comparing across evidence sets, the survival models fitted to the intermediate and mature evidence sets appear to have a shallower slope than those fitted to the minimum evidence set, resulting in a longer tail to the predicted survival curves. This is driven by the assumption that in the more mature evidence sets there is greater certainty over the ‘curative’ benefit of treatment because of additional evidence on patient survival up to month 60 of the hypothetical evidence set (vs. maximum survival of approximately 24 months in the minimum data set). This is broadly equivalent to saying that, in the intermediate and mature evidence sets, there is greater certainty over the flattening of the KM curve.

When comparing across competing survival models, the intermediate and mature evidence sets are also associated with a more consistent set of survival projections than the minimum data set. This leads to a narrower range of potential survival probabilities being predicted at later time points in both the intermediate data set and the mature data set. Therefore, unlike the bridge to HSCT model, additional evidence maturity in the curative model leads to a different projection of survival benefit, as well as impacting on the parametric uncertainty surrounding model extrapolations.

There are slight differences in the survival curves predicted from the intermediate and mature evidence sets because of differences in the weights applied to different functions. These differences cannot be clearly seen on the plots as the difference in weights is marginal. The key difference between these evidence sets is the additional sample size assigned to the mature data set, which primarily impacts on the uncertainty/precision surrounding survival estimates, which is not shown on these plots.

Adverse events: B-cell aplasia

A series of survival models was fitted to data on the time to CD19 positivity or relapse reported in Maude et al. 165 and used to predict the proportion of patients requiring treatment for B-cell aplasia. The best-fitting distribution was the Weibull distribution.

The accuracy of the partitioned survival model in predicting the outcomes of the Maude et al. 165 and Jeha et al. 178 studies was assessed by comparing the predicted survival probabilities from the model with the KM data. As shown in Figure 10, the final models appear to provide an accurate prediction of the extracted KM curve for OS in both studies.

FIGURE 10.

Model prediction (lines) vs. extracted KM curves (markers) of OS in all patients across the three evidence sets: (a) minimum data set; (b) intermediate data set; and (c) mature data set.

Treatment acquisition costs

Model inputs: utilities

Per-patient analyses: minimum evidence set

The sequence of assessments starts with a conventional assessment of cost-effectiveness at the patient level based on the minimum evidence set reported in Chapter 5. Disaggregated costs and outcomes are presented in Table 24.

The mean incremental cost of CAR T-cell therapy compared with standard of care over a patient’s lifetime was estimated to be £373,166, with CAR T-cell therapy resulting in an additional 7.46 QALYs. The incremental cost per QALY gained over a lifetime horizon was £49,995 (Table 25), which can be compared against the cost-effectiveness threshold. This can also be expressed as the per-patient net health effect (NHE), including benefits, harms and NHS/Personal Social Services costs. The NHE is the difference between any health gained with the intervention and the health forgone elsewhere in the health-care system and can be expressed in both monetary and QALY terms. With an ICER of approximately £50,000 per QALY, the incremental NHE at a threshold of £50,000 is close to zero (i.e. 0.001 QALYs or £41 per patient), that is, the additional health gained with the intervention is almost exactly offset by health forgone elsewhere.

Given the uncertainties surrounding longer-term outcomes, it may also be informative to consider how incremental NHEs accumulate over time or the ‘investment profile’ with CAR T-cell therapy, shown in Figure 11. The initial per-patient cost for CAR T-cell patients is attributable to the additional acquisition and administration costs of the CAR T-cells and associated HSCT costs. The ‘kink’ in the curve that appears early on represents the associated HSCT costs, which are assumed to be incurred at day 28. These negative NHEs are gradually offset by positive NHEs in later periods resulting from the ongoing mortality benefits assumed from successfully bridging to HSCT. However, it is only after 60 years that the initial losses are sufficiently compensated for by later gains, that is, CAR T-cell therapy appears to be close to breaking even (i.e. NHE ≥ 0).

FIGURE 11.

Investment profile: bridge to HSCT TPP.

Population-level analyses: minimum evidence set

Net health effects can also be presented for a population of patients over time. Although the presentation of population NHEs is not formally requested within the existing NICE methods guide,122 population-based analyses are requested to be submitted to assess population impacts within Section 5.12 (Impact on the NHS). In addition, Section 6.4.1. states that in situations in which the evidence of clinical effectiveness is ‘absent, weak or uncertain’, the committee is requested to ‘balance the potential net benefits to current NHS patients of a recommendation not restricted to research with the potential net benefits to both current and future NHS patients of being able to produce guidance and base clinical practice on a more secure evidence base’ (p. 69-71). 122

Analyses of population NHEs may therefore provide additional information to help inform the committee’s deliberations regarding possible research recommendations (Section 6.4 of the current methods guide122). Analyses of population NHEs require information about the prevalence and future incidence of the target population and a judgement about the time horizon over which the technology will be used in clinical practice. As outlined in Appendix 8, the expected incidence of eligible cases for the exemplar was estimated to be approximately 38 per annum. The technology time horizon was set to 10 years in the base case.

Table 26 reports the population NHE for CAR T-cell therapy over the 10-year technology time horizon. Over this period, the use of CAR T-cell therapy is estimated to generate an additional 2356 QALYs (discounted values) within the population considered compared with the current standard of care. However, as the additional lifetime cost of £117.78M (£141.75M – £23.97M; discounted values) requires other treatments to be displaced and health to be forgone by other patients in the NHS, overall the additional QALYs are exactly offset by health forgone elsewhere. Hence, the incremental population NHE at a £50,000 per QALY threshold is 0.26 QALYs (£12,813).

A series of one-way sensitivity analyses was undertaken to assess the sensitivity of the model results to changes in assumptions. The results of these sensitivity analyses are presented in Table 27.

The one-way sensitivity analyses indicate that the results of the evaluation are sensitive to assumptions regarding the potential for retreatment with CAR T-cell therapy and the assumed discounting rate for health effects in the model. The results of the evaluation are less sensitive to assumptions about the discounting rate for costs, to the impact of remission status on survival in non-HSCT patients and to reducing the cost of standard of care treatment to values consistent with treatment using FLAG-IDA (assuming similar efficacy to that of clofarabine).

If the committee was to consider the criteria met for applying the non-reference case discount rate of 1.5% for both costs and health effects (i.e. when treatment restores people who would otherwise die or have a very severely impaired life to full or near full health and when this is sustained over a very long period, normally at least 30 years), then the ICER would reduce to £35,162 per QALY and CAR T-cell therapy would be associated with an additional population NHE equivalent to 994 QALYs (£49.72M) in comparison to health forgone elsewhere.

Employing the stepwise discounting recommended by the UK Treasury210 to all public sector bodies makes only a small difference to the ICER (£49,601 vs. £49,995), with the incremental population NHE increasing to 18.94 QALYs (£946,799).

Although the results of the evaluation appear to be sensitive to assumptions about the potential for retreatment with CAR T-cell therapy, this was not considered to represent such a challenge in this TPP. CAR T-cell therapy was assumed to be used as a one-off therapy to induce remission and to improve the likelihood and outcomes of HSCT. It was assumed that patients would not receive a repeat treatment in the event of not achieving remission; nor would patients who were successfully treated with HSCT receive further treatments with CAR T-cell therapy.

Probabilistic analysis

The results of the probabilistic analysis are shown in Table 28.

TABLE 28 - Results of the base-case probabilistic analysis at the population level: bridge to HSCT TPP – minimum evidence set

Population level				Cost-effectiveness threshold of £50,000 per QALY gained
Treatment	Cost (£)	QALYs	ICER (£)	NHE, QALYs (£)	Incremental NHE, QALYs (£)	Probability cost-effective (%)	Consequences of decision uncertainty, QALYs (£)
CAR T-cell therapy	141,556,652	2716.4	55,090	–114.8 (–5,738,274)	–215.9 (–10,794,902)	26.1	56.3 (2,813,197)
Standard of care	24,728,297	595.7		101.1 (5,056,627)

The probabilistic ICER increased to £55,090 because of the model non-linearities. Consequently, the population NHE is now negative, with an overall loss to the health system of 215.9 QALYs (£10.79M). At a £50,000 cost-effectiveness threshold, the probability that CAR T-cell therapy is the most cost-effective option is 26.1%.

The cost-effectiveness acceptability planes and curves are presented in Figures 12 and 13, respectively.

FIGURE 12.

Cost-effectiveness acceptability plane: bridge to HSCT TPP – minimum evidence set. WTP, willingness to pay.

FIGURE 13.

Cost-effectiveness acceptability curve: bridge to HSCT TPP – minimum evidence set.

In addition to considering how uncertain a decision is to approve or reject a technology based on expected cost-effectiveness, an assessment of the scale of the likely consequences may also be potentially informative to the committee, particularly in deliberations related to possible research recommendations. An assessment of the potential consequences of uncertainty is important because it indicates the scale of the population NHEs that could be gained if uncertainty surrounding this decision could be resolved immediately. 128 This estimate also represents an expected upper bound to the benefits of more research. This may help to inform subsequent research recommendations. For example, if the maximum potential benefits of further research are considered unlikely to sufficiently justify the research costs, then it may not be worthwhile to issue further research recommendations.

These same consequences are referred to using the term ‘payer uncertainty burden’ (PUB) in the DSU report on managed access. 211 Elsewhere in the literature, these have been defined as the expected value of perfect information (EVPI) and the overall expected opportunity loss. Within the DSU report they are further defined as the value of the risk of making a particular decision because of uncertainty (expressed in either monetary or health units), combining two key concepts: first, the probability that the strategy with the highest expected NHE may not be the optimal strategy (i.e.1 – probability that the intervention is cost-effective based on the probabilistic results) and, second, the consequences of a ‘wrong’ decision in terms of QALYs and NHS costs that could have been saved if the truly optimal strategy had been selected instead.

Assuming a 10-year technology horizon, the consequences of decision uncertainty in the minimum data set are estimated to be 56.3 QALYs (£2.83M). Figure 14 shows how the scale of the consequences of decision uncertainty varies across different cost-effectiveness thresholds, reaching a peak at a £55,000 threshold.

FIGURE 14.

Consequences of decision uncertainty: bridge to HSCT TPP – minimum evidence set.

A summary of the population-level incremental NHEs, net monetary benefits, probability of cost-effectiveness and consequences of decision uncertainty across a range of willingness-to-pay thresholds is presented in Table 29.

TABLE 29 - Consequences of decision uncertainty across different willingness-to-pay thresholds: bridge to HSCT TPP – minimum evidence set

NMB, net monetary benefit.

Cost-effectiveness threshold (£)	Incremental NHE, QALYs	Incremental NMB (£)	Probability cost-effective (%)	Consequences of decision uncertainty, QALYs (£)
20,000	–3720.75	–74,414,973	0	0 (0)
30,000	–1773.61	–53,208,283	0	0 (0)
50,000	–215.90	–10,794,902	26.1	56.3 (2,813,197)
75,000	562.96	42,221,825	94.1	9.5 (710,894)
100,000	952.39	95,238,551	99.3	0.6 (63,592)

At conventional cost-effectiveness thresholds between £20,000 and £30,000 per QALY gained, the probability that CAR T-cell therapy is cost-effective compared with standard of care is 0%. Consequently, because of the high certainty that CAR T-cell therapy is not cost-effective at conventional thresholds (i.e. assuming that EoL criteria do not apply), there are no consequences of decision uncertainty. At a cost-effectiveness threshold of £50,000 per QALY gained, the probability that CAR T-cell therapy is cost-effective compared with standard of care is 26.1%. In this case, the expected population health consequence of decision uncertainty is 56.3 QALYs. These expected consequences can be interpreted as an estimate of the population NHE that could be gained if the uncertainty surrounding this decision could be resolved immediately and provide an expected upper bound on the benefits of more research. The corresponding expected monetary cost of decision uncertainty is approximately £2.8M. At thresholds of £75,000 and £100,000 per QALY gained, the probability that CAR T-cell therapy is cost-effective increases to > 94%. Because there is now high certainty that CAR T-cell therapy is cost-effective at these thresholds, the corresponding consequences of decision uncertainty reduce to < 10 QALYs (or < £1M in monetary terms).

Alternative pricing scenarios probabilistic analysis

A series of alternative pricing schemes has been generated to explore their potential impact on cost-effectiveness and decision uncertainty. These schemes were selected to reflect the possible approaches that have been suggested to address the potential HTA challenges, as highlighted previously in Chapter 3 (see Methods). The schemes considered were:

• A leasing scheme approach based on the approach outlined by Edlin et al. 137 In this scenario, the technology is assumed to be leased from the company. The monthly ‘lease’ payment was established by calculating a stream of payments over the expected survival duration of the patients that has the same expected net present value as the agreed price. Hence, payment was assumed to continue on a monthly basis while a patient remained alive.

• A pay for performance scheme in which payment is made retrospectively only for patients who achieve remission (CR) within a specified period (e.g. 28 days). Alternatively, an initial upfront payment could be made for all, with a separate ‘clawback’ agreed for patients who do not achieve remission.

• A more conventional PAS providing a fixed percentage discount (e.g. 10%).

The probabilistic results based on alternative hypothetical pricing scenarios are shown in Table 30. The scatterplots showing each iteration of incremental costs and incremental effects considered in the probabilistic sensitivity analysis are provided in Figure 15.

TABLE 30 - Impact of different pricing schemes on the cost-effectiveness of CAR T-cell therapy and the associated consequences of decision uncertainty: bridge to HSCT TPP – minimum evidence set

Pricing scenario	Population level				Cost-effectiveness threshold of £50,000 per QALY gained
	Treatment	Cost (£)	QALYs	ICER (£)	NHE, QALYs (£)	Incremental NHE, QALYs (£)	Probability cost-effective (%)	Consequences of decision uncertainty, QALYs (£)
Base case	CAR T-cell therapy	141,556,652	2716.4	55,090	–114.77 (–5,738,274)	–215.9 (–10,794,902)	26.1	56.3 (2,813,197)
	Standard of care	24,728,297	595.7		101.13 (5,056,627)
Leasing method	CAR T-cell therapy	140,082,600	2727.04	54,227	–74.61 (–3,730,478)	–179.94 (–8,997,139)	22.1	22.5 (1,123,900)
	Standard of care	24,678,802	598.91		105.33 (5,266,662)
Payment for remission patients only (70% on average)	CAR T-cell therapy	102,099,708	2708.82	36,430	666.83 (33,341,351)	577.2 (–28,861,808)	96.8	3.9 (195,152)
	Standard of care	24,614,048	581.87		89.59 (4,479,543)
Fixed pricing discount (10%)	CAR T-cell therapy	130,229,928	2707.12	49,857	102.52 (5,125,971)	6.05 (302,586)	51.8	131.2 (6,558,209)
	Standard of care	24,818,199	592.83		96.47 (4,823,385)

FIGURE 15.

Scatterplots of incremental costs and incremental effectiveness across the four pricing scenarios: bridge to HSCT TPP. (a) Base case; (b) leasing method; (c) payment for remission only; and (d) fixed discount. Blue box = mean of the probabilistic sensitivity analysis values. WTP, willingness to pay.

The impact of the different pricing schemes on the sampled outputs of the probabilistic sensitivity analysis is shown graphically in Figure 15. In the base case (fixed cost for CAR T-cell therapy), the cloud of simulated outcomes from the probabilistic sensitivity analysis is flat, such that there is considerable variability around the QALY gains of treatment, but little relative variability around the incremental costs. By introducing a leasing method, the cost of CAR T-cell therapy becomes more closely linked to the effectiveness of treatment, such that the cloud of simulated outcomes from the probabilistic sensitivity analysis is reoriented around the willingness-to-pay threshold. With both the remission and discounted schemes, the cost-effectiveness of treatment is improved and the cloud of simulated outcomes is shifted downwards on the chart.

A comparison plot of the consequences of decision uncertainty across the alternative pricing scenarios for different cost-effectiveness thresholds is shown in Figure 16.

FIGURE 16.

Comparison plot of the consequences of decision uncertainty across alternative pricing schemes: bridge to HSCT TPP – minimum evidence set.

Under a fixed one-off acquisition cost approach, assumed in the main analyses, the NHS bears all of the risks associated with uncertainty surrounding whether the expected benefits of therapy will be realised in routine clinical practice. Hence, the consequences of decision uncertainty to the NHS appear highest with this scheme (56.3 QALYs, £2.81M). The alternative schemes result in reductions in decision uncertainty and associated consequences to the NHS. However, the impact of this and the mechanism by which it is achieved differ across the separate approaches.

The leasing approach results in only a minor difference in the ICER. Similar levels of decision uncertainty also remain (i.e. the probability that the intervention is cost-effective is similar to that under a fixed one-off acquisition cost approach). However, the scale of the consequences of the uncertainty to the NHS is significantly reduced under this scheme. This scheme limits the risk to the NHS of overpaying for a technology that does not achieve the expected outcomes, significantly lowering the consequences of decision uncertainty to 22.5 QALYs (£1.12M).

The use of a pay for performance scheme improves the expected cost-effectiveness and, as a result, reduces both the level of decision uncertainty and the scale of the consequences of this uncertainty. Restricting payment to only patients who achieve remission improves expected cost-effectiveness (£36,430 per QALY), leading to a higher probability of being cost-effective (96.8%), thereby reducing the consequences of uncertainty to 3.9 QALYs (£195,000). The use of a more conventional PAS, based on an assumed 10% reduction in the acquisition cost, works in a similar manner by improving both expected cost-effectiveness (£49,857 per QALY) and the likelihood that the treatment is cost-effective (51.8%). However, as the ICER now lies closer to the threshold in absolute terms, the consequences are increased to 131.2 QALYs (£6.56M).

The comparison plot more clearly shows the impact of the alternative pricing schemes. The alternative schemes affect both the shape of the distribution of the consequences across the separate cost-effectiveness thresholds as well as their position.

Alternative evidence sets probabilistic analysis

All of the analyses reported previously have been based on the minimum evidence set. The impact of the alternative evidence sets on expected cost-effectiveness, the level of decision uncertainty and the scale of the consequences is reported in Table 31 and Figure 17.

TABLE 31 - Probabilistic results across evidence sets: bridge to HSCT TPP

Evidence set	Population level				Cost-effectiveness threshold of £50,000 per QALY gained
	Treatment	Cost (£)	QALYs	ICER (£)	NHE, QALYs (£)	Incremental NHE, QALYs (£)	Probability cost-effective (%)	Consequences of decision uncertainty, QALYs (£)
Minimum (base case)	CAR T-cell therapy	141,556,652	2716.4	55,090	–114.77 (–5,738,274)	–215.9 (–10,794,902)	26.1	56.3 (2,813,197)
	Standard of care	24,728,297	595.7		101.13 (5,056,627)
Intermediate	CAR T-cell therapy	141,556,652	2716.4	55,090	–114.77 (–5,738,274)	–215.9 (–10,794,902)	26.1	56.3 (2,813,197)
	Standard of care	24,728,297	595.7		101.13 (5,056,627)
Mature	CAR T-cell therapy	141,680,276	2764.76	53,462	–68.84 (–3,442,181)	–151.92 (–7,595,782)	28.1	48.1 (2,406,886)
	Standard of care	24,375,545	570.58		83.07 (4,153,600)

FIGURE 17.

Scatterplots of incremental costs and incremental effectiveness across evidence sets: bridge to HSCT TPP. (a) Minimum and intermediate evidence sets; and (b) mature evidence set. Blue box = mean of the probabilistic sensitivity analysis values. WTP, willingness to pay.

As highlighted in Chapter 5, the use of a separate structural/surrogate link within the bridge to HSCT TPP was employed to allow the incorporation of external evidence on the relationship between remission MRD and HSCT status. A limitation of our analysis is that the same external evidence is then used across each of the separate evidence sets. This means that the additional follow-up assumed in both the intermediate and the mature evidence sets is not adequately reflected in the results. Consequently, the ICER and associated decision uncertainty are identical across the minimum and intermediate data sets. Furthermore, the differences in results between these evidence sets and the mature evidence set is driven entirely by the increased precision (i.e. because of higher patient numbers) in the short-term remission, MRD and HSCT rates, as opposed to the additional maturity of follow-up data that may be available.

In practice, the additional follow-up reported in more mature studies could either replace the existing surrogate relationship employed here or be synthesised and combined with the external evidence. Hence, the value that the additional follow-up brings in terms of either confirming an assumed surrogate relationship or increasing the precision around this relationship is not adequately captured in these analyses.

Despite these limitations, the separate evidence sets may still provide an important comparison for the committee to consider, specifically in relation to how its deliberations might be affected in situations in which the same ICER and decision uncertainty are reported but under different circumstances, that is, situations in which the results are based entirely on external surrogate relationships compared with situations in which the results are based on actual observed data from a longer-term trial or follow-up.

As expected, the health consequence of decision uncertainty in the mature evidence set (48.1 QALYs, £2.41M) is lower than that reported in the minimum set (56.2 QALYs, £2.81M) at a threshold of £50,000 per QALY gained. These consequences are reduced by the increased precision associated with the larger sample in terms of the short-term remission, MRD and HSCT rates.

A comparison plot of the consequences of decision uncertainty between the minimum/intermediate evidence sets and the mature evidence set across a range of cost-effectiveness thresholds is provided in Figure 18.

FIGURE 18.

Comparison plot of the consequences of decision uncertainty across alternative evidence sets: bridge to HSCT TPP.

Presentation of the scale of consequences using population NHEs allows some important comparisons to be made across the separate pricing approaches and the different evidence sets. Specifically, these comparisons could provide a more explicit basis for considering the value of direct price reductions that might be realised through a conventional PAS (or less conventional schemes that work by indirectly lowering the effective price) compared with the provision of additional evidence (both precision and maturity) in terms of reducing decision uncertainty and its consequences.

In the bridge to HSCT TPP, significant reductions in the level and scale of the consequences of decision uncertainty (i.e. the risk faced by the NHS) appear to be achieved by more innovative pricing approaches such as pay for performance and leasing approaches than those that might be realised by the provision of further evidence. Such information might provide an important basis for discussions between manufacturers and NICE in terms of how the existing uncertainties that exist might be appropriately managed, ensuring that risks and benefits are more appropriately shared.

Per-patient analyses: minimum evidence set

Again, the sequence of assessments starts with a conventional assessment of cost-effectiveness at the patient level based on the minimum evidence set. Disaggregated costs and outcomes are presented in Table 32. The mean incremental cost of CAR T-cell therapy over an individual patient’s lifetime was estimated to be £503,256 and CAR T-cell therapy resulted in an additional 10.07 QALYs.

The expected cost-effectiveness of CAR T-cell therapy and the per-patient NHEs are shown in Table 33. In common with the previous TPP, the acquisition cost was set such that the ICER (£49,994) was close to the upper limit of NICE’s EoL threshold range (circa £50,000 per QALY gained).

The accumulation of NHEs over time, or, equivalently, the ‘investment profile’ per patient, is shown in Figure 19.

FIGURE 19.

Investment profile: curative intent TPP.

At the start of the time horizon, the initial high costs of treatment are far in excess of the immediate health benefits of treatment, leading to a negative NHE. Over time, the initial negative NHEs are gradually offset by the accrual of the residual health benefits of treatment (i.e. cure). In common with the bridge to HSCT TPP, it is only after approximately 60 years that the initial losses are sufficiently compensated for by later gains such that CAR T-cell therapy appears to be close to breaking even (i.e. NHE ≥ 0).

The shape of the investment profile differs slightly across the separate TPPs, with the early kink that was shown in the bridge to HSCT TPP not evident here. The lack of the kink is due to the small number of patients who are assumed to receive HSCT in the curative intent TPP. Hence, the resulting investment profile is smoother, although higher initial negative NHEs are reported because of the higher acquisition cost assumed within this TPP.

Population-level analyses: minimum evidence set

The expected per-patient effects of treatment were also extended to a population level based on similar assumptions concerning incidence (approximately 38 patients per annum) and the technology time horizon (10 years).

Table 34 reports the population NHEs for CAR T-cell therapy over the 10-year technology time horizon. Over this period, the use of CAR T-cell therapy is estimated to result in an additional 3177 QALYs (discounted values) within the population considered compared with the current standard of care. However, as the additional lifetime costs of £158.84M (discounted values) require other treatments to be displaced and health to be forgone by other patients, overall the additional QALYs are almost exactly offset by health forgone elsewhere. The resulting incremental population NHE is 0.39 QALYs expressed in health terms and £19,269 expressed in monetary terms.

A series of one-way sensitivity analyses was conducted to assess the sensitivity of the model results to changes in assumptions or model settings. The results of these sensitivity analyses are presented in Table 35.

Again, the results of the one-way sensitivity analyses indicate that the results of the evaluation in the curative intent TPP are sensitive to assumptions about the potential for retreatment with CAR T-cell therapy and the assumed discounting rate for health effects in the model. The results of the evaluation are relatively insensitive to assumptions about the discounting rate for costs, the use of a stepped discounting rate (vs. constant discounting rates) and a reduction in the cost of standard of care treatment to values consistent with treatment using FLAG-IDA (keeping the same efficacy).

If the committee was to consider the criteria met for applying the non-reference case discount rate of 1.5% for both costs and health effects, then the ICER would reduce to £35,585 per QALY, and CAR T-cell therapy would be associated with an additional population NHE equivalent to 1291 QALYs (£64.56M) in comparison to health forgone elsewhere.

Employing the stepwise discounting recommended by the UK Treasury again makes only a small difference to the ICER results (£49,606 vs. £49,994), with an incremental population NHE of 25 QALYs (£1.26M) in comparison to health forgone elsewhere.

The sensitivity of the results to assumptions about the potential for retreatment with CAR T-cell therapy was considered to represent a more important issue within this TPP, that is, the longer-term survival benefits are directly linked to the curative potential of the CAR T-cells themselves rather than to an intermediate treatment such as HSCT. Consequently, the potential need to re-administer CAR T-cell therapy over a longer period represents an important additional source of uncertainty within this TPP, particularly for the minimum data set with a relatively short follow-up period.

Probabilistic analysis

The results of the probabilistic analysis are shown in Table 36. The probabilistic ICER increased to £50,906 because of the model non-linearities. Consequently, the population NHE is now negative, with an overall loss to the health system of 56 QALYs (£2.82M). At a £50,000 cost-effectiveness threshold, the probability that CAR T-cell therapy is the most cost-effective option is 50.7%.

TABLE 36 - Results of the base-case probabilistic analysis at the population level: curative intent TPP – minimum evidence set

Population level				Cost-effectiveness threshold of £50,000 per QALY gained
Treatment	E[Costs] (£)	E[QALYs]	ICER (£)	E[NHE], QALYs (£)	Incremental NHE, QALYs (£)	Probability cost-effective (%)	Consequences of decision uncertainty, QALYs (£)
CAR T-cell therapy	183,931,590	3501.50	50,906	–177.13 (–8,856,695)	–56.4 (–2,823,943)	50.7	304.6 (15,229,876)
Standard of care	25,270,727	384.76		–120.66 (–6,032,752)

The cost-effectiveness acceptability planes and curves are presented in Figures 20 and 21, respectively.

FIGURE 20.

Cost-effectiveness acceptability plane: curative intent TPP – minimum evidence set. WTP, willingness to pay.

FIGURE 21.

Cost-effectiveness acceptability curve: curative intent TPP – minimum evidence set.

The consequences of decision uncertainty in the minimum evidence set are estimated to be 304.6 QALYs (£15.23M). Figure 22 shows how the scale of the consequences of decision uncertainty varies across different cost-effectiveness thresholds, reaching a peak at the £50,000 threshold.

FIGURE 22.

Consequences of decision uncertainty: curative intent TPP – minimum evidence set.

A summary of the population-level incremental NHEs, net monetary benefits, probability of cost-effectiveness and consequences of decision uncertainty across a range of cost-effectiveness thresholds is presented in Table 37.

TABLE 37 - Consequences of decision uncertainty across different willingness-to-pay thresholds: curative intent TPP – minimum evidence set

NMB, net monetary benefit.

Cost-effectiveness threshold (£)	Incremental NHE, QALYs	Incremental NMB (£)	Probability cost-effective (%)	Consequences of decision uncertainty, QALYs (£)
20,000	–4816.30	–96,326,095	0.0	0 (0)
30,000	–2171.96	–65,158,711	0.0	0 (0)
50,000	–56.48	–2,823,943	50.7	304.6 (15,229,876)
75,000	1001.26	75,094,517	88.1	66.2 (4,963,418)
100,000	1530.13	153,012,977	94.8	23.4 (2,336,731)

At conventional thresholds of between £20,000 and £30,000 per QALY gained, the probability that CAR T-cell therapy is cost-effective compared with standard of care is 0%. Consequently, there are no consequences of decision uncertainty at these threshold values. At a willingness-to-pay threshold of £50,000 per QALY gained, the probability that CAR T-cell therapy is cost-effective compared with standard of care is 50.7%. In this case, the expected population health consequence of decision uncertainty is 305 QALYs (10 years). The corresponding expected monetary cost of decision uncertainty is approximately £15.23M. At thresholds of £75,000 and £100,000 per QALY gained, the probability that CAR T-cell therapy is cost-effective increases to > 88%. Despite there being high certainty that CAR T-cell therapy is cost-effective at these thresholds, the corresponding consequences of decision uncertainty remain relatively high at 66 QALYs (£4.96M in monetary terms) and 23 QALYs (£2.34M), respectively.

Alternative pricing scenarios probabilistic analysis

The probabilistic results based on alternative hypothetical pricing scenarios are shown in Table 38. The scatterplots showing the iterations of incremental costs and incremental effects for the four pricing scenarios considered in the probabilistic sensitivity analysis are provided in Figure 23. A comparison plot of the consequences of decision uncertainty across the alternative pricing scenarios is shown in Figure 24.

TABLE 38 - Impact of different pricing schemes on the cost-effectiveness of CAR T-cell therapy and the associated consequences of decision uncertainty: curative intent TPP – minimum evidence set

Pricing scenario	Population level				Cost-effectiveness threshold of £50,000 per QALY gained
	Treatment	Cost (£)	QALYs	ICER (£)	NHE, QALYs (£)	Incremental NHE, QALYs (£)	Probability cost-effective (£)	Consequences of decision uncertainty, QALYs (£)
Base case	CAR T-cell therapy	183,931,590	3501.50	50,906	–177.13 (–8,935,381)	–56.48 (–2,902,629)	50.7	304.6 (15,229,876)
	Standard of care	25,270,727	384.76		–120.66 (–6,032,752)
Leasing method	CAR T-cell therapy	181,832,300	3488.85	50,618	–147.79 (–7,389,708)	–38.21 (–1,910,653)	49.2	65.6 (3,277,969)
	Standard of care	25,317,596	396.77		–109.58 (–5,479,055)
Payment for remission patients only (90% on average)	CAR T-cell therapy	167,127,512	3510.80	45,708	168.25 (8,412,636)	266.50 (13,325,042)	63.9	236.1 (11,803,131)
	Standard of care	25,219,827	406.15		–98.25 (–4,912,407)
Fixed pricing discount (10%)	CAR T-cell therapy	167,054,363	3535.49	45,131	194.40 (9,719,974)	305.88 (15,293,860)	64.2	209.1 (10,456,541)
	Standard of care	25,301,914	394.56		–111.48 (–5,573,886)

FIGURE 23.

Scatterplots of incremental costs and incremental effectiveness across the four pricing scenarios: curative intent TPP. (a) Base case; (b) leasing method; (c) payment for remission only; and (d) fixed discount. Blue box = mean of the probabilistic sensitivity analysis values. WTP, willingness to pay.

FIGURE 24.

Comparison plot of consequences of decision uncertainty across alternative pricing schemes: curative intent TPP – minimum evidence set.

As observed in the previous analysis, the fixed one-off acquisition cost approach is associated with the highest potential consequences because of decision uncertainty (304.6 QALYs, £15.23M). As before, the alternative schemes result in reductions in decision uncertainty and associated consequences to the NHS. However, the impact of these reductions and the mechanisms by which they are achieved differ across the separate approaches.

The leasing approach results in only a minor difference in the ICER. Similar levels of decision uncertainty also remain (i.e. the probability that the intervention is cost-effective is similar to that under a fixed one-off acquisition cost approach). However, the scale of the consequences of the uncertainty to the NHS is significantly reduced with this scheme. This scheme limits the risk to the NHS of overpaying for a technology that does not achieve the expected outcomes, significantly lowering the consequences of decision uncertainty from > 300 QALYs in the base case to 65.6 QALYs (£3.28M) with the leasing approach.

Alternative evidence sets probabilistic analysis

The results of the probabilistic analysis are shown in Table 39 and Figure 25. A comparison plot of the consequences of decision uncertainty across the evidence sets is shown in Figure 26.

TABLE 39 - Probabilistic results across evidence sets: curative intent TPP

Evidence set	Population level				Cost-effectiveness threshold of £50,000 per QALY gained
	Treatment	E[Costs] (£)	E[QALYs]	ICER (£)	E[NHE], QALYs (£)	Incremental NHE, QALYs (£)	Probability cost-effective (%)	Consequences of decision uncertainty, QALYs (£)
Minimum	CAR T-cell therapy	183,931,590	3501.50	50,906	–177.13 (–8,935,381)	–56.48 (–2,902,629)	50.7	304.6 (15,229,876)
	Standard of care	25,270,727	384.76		–120.66 (–6,032,752)
Intermediate	CAR T-cell therapy	183,586,917	4296.77	43,344	625.03 (31,251,488)	486.22 (24,311,227)	85.9	40.6 (2,031,623)
	Standard of care	25,264,818	644.10		138.81 (6,940,262)
Mature	CAR T-cell therapy	183,560,268	4307.12	43,252	635.91 (31,795,547)	494.47 (24,723,328)	91.5	14.1 (707,136)
	Standard of care	25,103,273	643.51		141.44 (7,072,220)

FIGURE 25.

Comparison plot of the consequences of decision uncertainty across alternative evidence sets: curative intent TPP.

FIGURE 26.

Scatterplots of incremental costs and incremental effectiveness across evidence sets: curative intent TPP. (a) Base case; (b) intermediate evidence set; and (c) mature evidence set. Blue box = mean of the probabilistic sensitivity analysis values.

As expected, the health consequences of decision uncertainty in the mature evidence set (14.1 QALYs, £707,000) are lower than those reported in the minimum evidence set (304.6 QALYs, £15.23M) at a threshold of £50,000 per QALY gained. These consequences are reduced by the increased certainty surrounding the trajectory of the parametric survival curves and the effect of increased maturity on improving the cost-effectiveness of CAR T-cell therapy. As is evident from Figure 26, the increased certainty over the longer-term survival benefits of treatment (represented by the longer follow-up assumed in the intermediate and mature evidence sets) has a proportionately greater effect in reducing decision uncertainties within the minimum evidence set than the increased precision of greater patient numbers (i.e. reflected only in the mature evidence set).

Use of surrogate end points

The choice of surrogate end points used by manufacturers in their submissions must be researched, explicit and justified. Ideally, a systematic review should be performed to evaluate the strength of the association between the surrogate and the patient-relevant outcome, and the evidence on surrogate validation should be presented according to an explicit hierarchy.

Pivotal study design and the use of historical control data sets

For manufacturer submissions, consideration should be given to the benefits of producing recommendations and/or minimum reporting requirements in terms of the methods used to obtain and analyse single-arm trial data when they are compared with historical control data. When single-arm trial data form the main basis of an assessment, a clear rationale should be given for the type of comparisons made (implicit or explicit) and for the choice of the historical control data that were selected. For example, the gold standard for historical data might be matched data obtained from a patient database (rather than relying on published studies, which might not fit the trial population being studied well enough). When designing trials, manufacturers should bear in mind that multicentre trials are likely to produce more reliable and generalisable results than single-centre trials.

Evidence Review Groups might benefit from using checklists when appraising how historical control data were identified and analysed by manufacturers.

Efficacy estimates

Submission of IPD might be beneficial for ERGs, especially when data sets are small. Use of multivariate meta-analysis can lead to reduced uncertainty around the effectiveness parameter. By allowing all of the relevant data to be incorporated in estimating clinical effectiveness outcomes – including data from surrogate outcomes – multivariate meta-analysis can improve the estimation of health utilities through mapping methods.

Manufacturers should report the data for the full trial population, that is, all eligible patients, including patients who died before they could receive treatment and patients for whom a bespoke (autologous) treatment could not be produced.

The role of any further mandatory trial evidence

Manufacturers should provide details of mandated further studies (e.g. those related to conditional approvals or approvals made through the EMA’s adaptive pathways approach). Future reports from the ADAPT SMART project should provide details about how the use of development plans across target populations agreed upfront with the EMA is working. Guidance may be needed regarding methodological approaches to utilising ‘confirmatory’ trial data in a related indication to update the decision that NICE made for the original indication.

Consideration is also needed regarding the precise role that NICE will have in EMA adaptive pathways processes. For example, what will be the mechanisms by which the EMA updates NICE with new efficacy and safety data for conditionally approved ATMPs (in a timely way) and how will NICE deal with the new data (process wise)?

Extrapolation approaches

Given the inevitable uncertainties that are likely to exist regarding the longer-term benefits of regenerative medicines and cell-based therapies, further methodological research could be usefully undertaken to help inform how these uncertainties might be appropriately quantified in a transparent manner to inform cost-effectiveness analyses. Further research may be particularly helpful to determine the appropriateness of alternative survival modelling approaches to regenerative medicines and cell-based therapies, including more flexible survival models and cure fraction models.

The level of data maturity is an important factor in deriving robust survival projections that are required for cost-effectiveness assessments. When follow-up is immature, a single ‘best-fitting’ survival distribution may not adequately characterise uncertainties over the longer-term extrapolation period. Although the robustness of the ICER estimates to alternative distributions can be explored through separate sensitivity analyses or scenarios, the transparency of the process may be impacted if the weighting of these is not explicitly considered in subsequent policy decisions. The feasibility and appropriateness of model-averaging approaches may also need to be considered more formally. The advantage of these approaches is that the parametric uncertainty associated within each distribution and the uncertainty (or weights) surrounding the choice of preferred method can be more explicitly characterised. However, given the potential complexity in both undertaking these analyses and communicating the results, efforts will need to be made to ensure that models are developed so that informal judgements can be explicitly incorporated in a timely and transparent manner.

Irrecoverable costs and possible learning curve effects

Given the complexity of the treatment pathways that may be required for regenerative medicines and cell-based therapies, manufacturers will need to report clearly the resource and cost assumptions of the different processes required to determine whether the full costs to the NHS have been included and any aspects for which uncertainties may exist. Issues of irrecoverable costs may need to be considered more formally, particularly if a new technology could impose additional infrastructure requirements on the health system. If reimbursement decisions about the technology change before the end of the lifetime of the equipment (e.g. approval is withdrawn), then these costs may not be recovered and hence need to be explicitly considered.

The existence and possible impact of learning curves may also be an important issue for clinical effectiveness and cost-effectiveness assessments. Although the existence of learning curves has received attention in the clinical literature, the relevance of recent work in this area in the context of assessing the cost-effectiveness of medical devices should be considered. 133

Quantification of decision uncertainty

Presentation of the scale of the consequences of decision uncertainty using population NHEs may provide an important additional approach to quantifying decision uncertainty to the assessments already routinely specified within the existing TA methods guide. 122 The implications of existing research128 and research by the DSU211 will also need to be considered by NICE to determine whether further changes to its processes or methods may be helpful for informing the nature of any additional assessments that may be required.

Such assessments could provide a more transparent and explicit basis for discussions between manufacturers, NICE and other relevant stakeholders in terms of how the existing uncertainties might be appropriately managed, ensuring that risks and benefits are more appropriately shared. Broader consideration will also need to be given to approaches that may extend beyond NICE’s existing remit, for example alternative payment schemes. Consequently, other bodies and manufacturers themselves may also have an important role in identifying more innovative approaches to seeking reimbursement that recognise the inherent uncertainties and lead to a more efficient sharing of associated risk.

Existing criteria

The National Institute for Health and Care Excellence’s existing processes also make separate provision for specific disease and technology characteristics that may be relevant to many regenerative medicines and cell-based therapies. Although NICE’s current EoL criteria allow the committee to explore a QALY weighting that is different from that of the reference case, the appropriateness of these criteria may need to be considered in relation to treatments that have curative potential. Further methodological research may also be important to determine whether an alternative weighting approach might be more appropriate for curative therapies. Existing research has identified a potential disconnect between individual and societal preferences concerning valuation of treatment compared with preventative interventions. Further research that is more specifically focused on the concept of cure may provide important additional insights.

Although the NICE methods guide permits the use of a non-reference case discount to be applied in specific contexts, it remains unclear whether regenerative medicines and cell-based therapies would meet the existing criteria (e.g. uncertainties over the projected benefits and/or potentially significant irrecoverable costs). Consequently, NICE may need to provide additional guidance to ensure that manufacturers understand the likelihood of meeting these criteria.

Adjusting using external data

Welton et al. 50 present a Bayesian hierarchical model to model bias in RCTs that are at high risk of bias. The authors developed a mixed-effects model in which treatment effects are considered as fixed and bias effects are considered as random. Estimates of bias in any given meta-analysis were given as a function of the prior distribution, which was estimated from published meta-analyses of RCTs and data from the current meta-analysis. When a meta-analysis contained no information about the size and magnitude of the bias, that is, when there were only high-risk studies, the estimate of bias was based on the prior distribution alone. This method allows treatment effect estimates to include information from the high-risk studies, accounting for the uncertainty in the magnitude of the bias in any particular meta-analysis. This technique was designed with adjustment of RCTs in mind but is extendable to the adjustment of NRSs whereby RCTs represent the low-risk studies and NRSs represent the high-risk studies. An appropriate library of meta-analyses combining data from RCTs and NRSs would, however, be necessary to apply this technique.

Elicitation

Turner et al. ,53 recognising the practical limitations of basing adjustment on external empirical data, proposed an alternative approach in which the direction and magnitude of biases are elicited by reviewers. This method can deal with multiple sources and types of bias including both internal validity bias and external validity bias. In brief, Turner et al. 53 proposed that authors design an idealised study aimed at answering the specific question in mind. This study may not be feasible to carry out and is simply a tool for exploring bias in the completed studies. To identify the potential biases, the completed studies are compared with the idealised study considering a number of potential sources of bias. For each form of bias identified, assessors then elicit the likely magnitude and variance of the bias. These estimates of the magnitude and variance of the potential biases can then be used to adjust treatment effect estimates accounting for both the magnitude and the uncertainty of any potential bias identified. External empirical evidence of bias can be included in the analysis rather than relying on eliciting values, but it is assumed that these data will be largely unavailable.

Regression analysis

Confounding bias occurs in the context of estimating clinical effectiveness when individual patient characteristics, such as age, sex and disease duration, that influence efficacy outcomes are also correlated with treatment received. Regression analysis seeks to directly adjust for these potential confounding variables by building a statistical model54 of the form:

Regression models therefore allow the estimation of the treatment effect conditional on these confounding variables. There are many types of regression model. The choice of any particular model depends on the characteristics of the outcome variable (i.e. continuous or categorical) and on the way that it is mathematically related to the explanatory variables. Typically, for dichotomous outcomes, a logistic regression model is used. For continuous outcomes a linear regression model is used and for time-to-event data a proportional hazards regression (Cox regression) model is used.

Theoretically, regression models can be used to entirely eliminate bias attributable to confounding as long as the appropriate parameters are included with a regression equation. However, in reality, either confounding factors will be unobserved, preventing their inclusion in the regression model, or a lack of understanding of the disease process will mean that we do not know to include them in the regression model. When such unobserved confounders are not included in the regression model, confounding bias can persist. Regression techniques can be used in conjunction with other methods for adjusting of confounding, including propensity scoring and instrumental variables. 54 Regression models also require a minimum number of participants per additional explanatory variable, with a useful rule of thumb being at least 10 observations per explanatory variable. 55 This requirement may limit their application to regenerative medicine where effectiveness estimates can be based on relatively small studies.

Stratification

Stratification involves the division of participants into subgroups with respect to categorical (or categorised quantitative) prognostic factors, for example classifying age into decades or weight into quartiles. The intervention effect is then estimated in each stratum and a pooled estimate is calculated across strata. This procedure can be interpreted as a meta-analysis at the level of an individual study. Major limitations are that it is feasible and meaningful only when effects are consistent across strata and that it can usually be employed for only few variables, as strata increase exponentially in keeping with the number of stratification factors. 54 As such, stratification can only minimise rather than completely remove the bias resulting from confounding.

Matching

Matching involves selecting participants with similar values for important prognostic factors to make the control and treatment groups more similar so that any differences between the treatment group and the control group cannot be a result of differences in the matched variables. Matching can be carried out both prospectively or retrospectively. Matching prospectively can, however, cause significant recruitment problems. Matching retrospectively can also cause problems as it is not always possible to match individuals. In large studies it is often easier to use an unmatched control group and to use regression analysis to adjust for what would have been matched on. 56 Matching can, however, be useful in small studies when there are insufficient participants to adjust for multiple variables at once. 56 As such, matching may be a potentially useful technique for controlling for confounding when using historical controls with the small single-arm studies that have typified regenerative medicine clinical evidence. Although matching can be used to reduce confounding bias, it is unlikely to completely account for all difference because of unobserved confounding.

Instrumental variables analysis

Instrumental variables techniques attempt to approximate the experimental approach by using an instrument variable or variables. A parameter is considered a valid instrument if it meets the following two conditions:

1. the instrument must be correlated with the receiving of treatment (or exposure)

2. the instrumental variable must be independent of (uncorrelated with) unobserved confounders.

When a valid instrument exists the instrumental variables approach leads to unbiased estimates equivalent to those from a randomised study. Indeed, randomisation can be thought of as the perfect instrumental variable as it is by definition perfectly associated with treatment allocation and independent of unobserved heterogeneity. The problem with the instrumental variables approach, however, is that identification of a valid instrument is often difficult. Furthermore, although the first requirement of a valid instrument is easily tested, the second requirement is essentially untestable and therefore we can never be certain that an instrument is valid. The application of an instrumental variables approach also leads to significant reduction in the power to detect a difference, particularly when the instrumental variable is poorly correlated with treatment allocation. This issue may be particular problematic in regenerative medicine where studies are often small with a low power to detect differences between alternative treatments.

Propensity scoring

Propensity scoring, rather than being a single method, is a suite of methods that considers confounding bias as a form of selection bias in which treatment allocation is acknowledged to be non-random and which considers that treatment selection is often influenced by a patient’s characteristics. 57 All propensity scoring methods seek to model this process of treatment selection and estimate the propensity to receive treatment based on baseline patient characteristics. Conditional on the propensity score the distribution of baseline characteristics will be similar in both the treatment group and the control group. Therefore, in patients with similar propensity scores patient characteristics will be the same, independent of whether treatment was received. The propensity is typically estimated using a logistic regression model, although other methods have been applied. The estimated propensity score can be used to remove the effects of confounding in four different ways. 57 These are described very briefly below:

1. Matching – the propensity score is used to match participants in the treatment and control groups who have similar values of the propensity score.

2. Stratification – subjects are ranked on the propensity score and stratified into groups, typically quintiles. Stratum-specific mean differences are then calculated and these differences are effectively meta-analysed to estimate an overall difference in means.

3. Inverse probability of treatment weighting (IPTW) – this involves using the propensity score as a weight such that an individual participant’s weight is equal to the inverse of the probability of receiving the treatment.

4. Covariate – the propensity score is added as a covariate within a regression equation. The propensity score can be added either with or without additional explanatory variables.

A number of studies have sought to compare propensity scoring methods to ascertain which is the most effective at removing confounding bias. 57–61 These studies have shown matching and IPTW to be more effective than stratification or covariate adjustment. 58–61 The principal advantage of propensity scoring over other adjustment methods such as regression analysis is that it can be used even with small sample sizes and therefore may be particularly relevant to regenerative medicine. Propensity scoring also has a number of disadvantages. First, propensity scoring controls only for differences in observed variables and does nothing to remove bias resulting from unobserved characteristics. Second, including variables that affect whether a treatment is received but not the outcome of interest increases the variance of the estimated treatment effect without a concomitant reduction in bias. This is problematic as sometimes it can be difficult to establish which variables will impact only on which treatment is received. 57

Propensity scoring compared with regression analysis

Six studies62,64–68 compared propensity scoring with regression analysis. The conclusions from these studies were inconsistent. Two studies66,68 concluded that estimates obtained from regression analysis are similar to those obtained using propensity scoring. However, two studies64,67 also came to the opposite conclusion, that estimates obtained from regression analysis and propensity scoring differ significantly. One simulation study65 comparing the two methods considered propensity scoring to be the superior method, whereas another study62 found that propensity scoring is superior when the number of events per confounder is low. The disparate results of these studies means that it is difficult to draw conclusions regarding the relative performance of the different approaches, but Kurth et al. 64 make an important observation that potentially explains these different results. Kurth et al. 64 note that each method of adjustment answers subtly different questions as it makes different assumptions. This inevitably means that different methods of adjustment will yield different results. Kurth et al. 64 advise that researchers need to consider carefully the population for which an overall treatment estimate is most appropriate.

Regression analysis compared with instrumental variables analysis

Only two studies compared regression analysis with instrumental variable methods. 63,67 Crosby et al. 63 found that results from regression analysis and instrumental variable methods differed somewhat and suggested that instrumental variable methods are potentially superior. Stukel et al. 67 compared all three methods of adjustment (regression analysis, instrumental variables and propensity scoring) and concluded that instrumental variables may lead to less biased estimates of treatment effects. Although the evidence on instrumental variables is limited it nevertheless suggests that instrumental variables may offer advantages over other methods and may produce the least biased estimates.

Propensity scoring compared with instrumental variables analysis

A recently published systematic review found 55 comparisons (37 studies) between propensity scoring and instrumental variables analysis. 69 The review found there to be a slight/fair agreement between the methods (Cohen’s kappa coefficient = 0.21, 95% CI 0.00 to 0.41). In 23 cases (42%) the results were non-significant using one method but significant using the other method; using instrumental variables methods the results were non-significant in most cases (87%). The study authors recommended caution when interpreting the results of these analyses and that further research is needed to clarify the roles of these methods.

In addition to the seven empirical studies identified,62–67,69 a discussion paper by Biondi-Zoccai et al. 70 provides a useful overview of the alternative methods of adjustment. Biondi-Zoccai et al. 70 concluded that there is no clearly superior method, noting that ‘both standard multivariable methods and propensity scores have key limitations, and none is able to take into account unknown confounders’ (p. 736). Biondi-Zoccai et al. ,70 however, go on to suggest that propensity scoring methods may have advantages over regression methods when the sample size is small and that, although instrumental variables methods are not without their limitations, they are the only methods that allow for unobserved confounding to be adjusted for.

Adjustment methods applied specifically to single-arm trials

Hamre et al. 71 carried out a study aimed at improving methods to minimise bias in single-arm studies. Four bias factors were suppressed stepwise: attrition bias (by replacing missing values with the baseline value carried forward), bias from natural recovery (by sample restriction to patients with disease duration of 12 months), regression to the mean because of symptom-driven self-selection (by replacing baseline scores with scores 3 months before enrolment) and bias from adjunctive therapies (by sample restriction to patients not using adjunctive therapies). In the cohort analysed, these four bias factors could together explain a maximum of 37% of the 0- to 6-month improvement in disease score. However, this method has not been widely tested on other cohorts.