Notes
Article history
The research reported in this article of the journal supplement was commissioned and funded by the HTA programme on behalf of NICE as project number 08/84/01. The assessment report began editorial review in July 2009 and was accepted for publication in August 2009. See the HTA programme website for further project information (www.hta.ac.uk). This summary of the ERG report was compiled after the Appraisal Committee’s review. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the referees for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.
Declared competing interests of authors
none
Permissions
Copyright statement
© 2010 Queen’s Printer and Controller of HMSO. This journal is a member of and subscribes to the principles of the Committee on Publication Ethics (COPE) (http://www.publicationethics.org/). This journal may be freely reproduced for the purposes of private research and study and may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NETSCC, Health Technology Assessment, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.
2010 Queen’s Printer and Controller of HMSO
This paper presents a summary of the evidence review group (ERG) report into the clinical effectiveness and cost-effectiveness of azacitidine (aza) compared with conventional care regimes (CCR) for higher risk patients with myelodysplastic syndrome (MDS), chronic myelomonocytic leukaemia (CMML) and acute myeloid leukaemia (AML), based on the evidence submission from the manufacturer to the National Institute for Health and Clinical Excellence (NICE) as part of the single technology appraisal process. The patient outcomes governing relative effectiveness and cost-effectiveness were defined as overall survival, time to progression (TTP) to AML, adverse events and health-related quality of life (HRQoL). The clinical evidence was derived from open label randomised controlled trial referred to as study AZA-001. It compared aza with CCR in 358 patients with higher risk MDS, CMML and AML 20–30% blasts. The outcomes reported in AZA-001 included overall survival, TTP to AML and adverse events. No HRQoL results were reported; however, outcomes likely to impact on HRQoL were provided. The results showed that: the median overall survival was 24.5 months on aza, compared with 15.0 months in the CCR group (p = 0.0001); the response rates were low (complete remission 17% aza versus 8% CCR); the median time to transformation to AML was greater in the aza group (17.8 versus 11.5 months; p < 0.0001); and of patients who were red blood cell (RBC) transfusion-dependent at baseline, 45% of those on aza became RBC transfusion-independent during the treatment period, compared with 11.8% in the CCR group (p < 0.0001). The ERG reran the submission’s search strategies after some modifications incorporating minor improvements. The ERG analysed the submitted economic model (model 1) and identified a number of inconsistencies and errors within the model. The manufacturer submitted a revised model for analysis by the ERG. Using the issues identified in the earlier analysis, the ERG conducted those repairs to the revised model that were feasible within time constraints. The ERG ran this version in probabilistic sensitivity analyses to generate cost-effectiveness acceptability frontiers. The results of these exploratory analyses indicated that: for standard-dose chemotherapy (SDC)-treated patients, of six treatment options available, best supportive care (BSC) was likely the most cost-effective option up to a threshold of £51,000/quality-adjusted life-year (QALY) [beyond £51,000/QALY, aza + low-dose chemotherapy (LDC) became cost-effective]; for LDC-treated patients, of four options available, BSC was again the most cost-effective option up to a willingness-to-pay threshold of £51,000/QALY (aza + LDC became cost-effective after £51,000/QALY); for BSC-treated patients, aza + BSC became cost-effective relative to BSC at a threshold of about £52,000/QALY. The ERG considers these results exploratory and considers that they should be viewed with caution. The AZA-001 study showed that, compared with CCR, those MDS patients receiving aza had prolonged median survival, had delayed progression to AML, had reduced dependence on transfusions and had a small improvement in response rate. Given the general paucity of economic modelling work in MDS and the limitations of the submitted industry model there is an evident need for an independent cost-effectiveness analysis of aza in MDS. At the time of writing, the guidance appraisal consultation document issued by NICE on 4 March 2010 states that azacitidine is not recommended as a treatment option for people not eligible for haemopoietic stem cell transplantation with the the following conditions: intermediate-2 and high-risk MDS according to the International Prognostic Scoring System, CMML with 10-29% marrow blasts without myeloproliferative disorder, or with AML with 20-30% blasts and multilineage dysplasia, according to World Health Organization classification.
Introduction
The National Institute for Health and Clinical Excellence (NICE) is an independent organisation within the NHS that is responsible for providing national guidance on the treatment and care of people using the NHS in England and Wales. One of the responsibilities of NICE is to provide guidance to the NHS on the use of selected new and established health technologies, based on an appraisal of those technologies.
NICE’s single technology appraisal (STA) process is specifically designed for the appraisal of a single product, device or other technology, with a single indication, where most of the relevant evidence lies with one manufacturer or sponsor. 1 Typically, it is used for new pharmaceutical products close to launch. The principal evidence for an STA is derived from a submission by the manufacturer/sponsor of the technology. In addition, a report reviewing the evidence submission is submitted by the evidence review group (ERG), an external organisation independent of NICE. This paper presents a summary of the ERG report2 for the STA entitled ‘Azacitidine (aza) for the treatment of myelodysplastic syndrome (MDS), chronic myelomonocytic leukaemia (CMML) and acute myeloid leukaemia (AML)’.
Description of the underlying health problem
The following is taken from the NICE scope for this STA.
The MDSs are a diverse group of haematological disorders in which the bone marrow functions abnormally and insufficient numbers of mature blood cells are produced. Red blood cells (RBCs), white blood cells and platelets may all be affected by MDS, resulting in life threatening disease, with anaemia and increased risk of bleeding and infections. MDS affects patients’ quality of life owing to debilitating symptoms such as fatigue and dyspnoea, treatment regimens involving hospitalisation with intravenous drug infusions and blood transfusions, and complications such as severe infections.
Myelodysplastic syndromes are subdivided using the International Prognostic Scoring System (IPSS), and the French–American–British (FAB) and World Health Organization (WHO) classification systems. Based on the proportion of leukaemic cells (or ‘blasts’), the presence of chromosome 7 abnormalities and the presence of blood cytopenia, the IPSS classifies outcome as either low-risk, intermediate-I risk, intermediate-II risk or high-risk. It is estimated that higher risk MDS subgroups (intermediate-II and high-risk) form approximately 22% and 7% of the MDS population, respectively. The FAB system divides MDS into five subgroups, including CMML, which is characterised by high numbers of white blood cells in the blood and bone marrow. The WHO system, which divides MDS into eight subgroups, does not class CMML as a type of MDS, but rather within a new category of myelodysplastic–myeloproliferative overlap syndromes.
Myelodysplastic syndromes are associated with an increased risk of transformation to AML. AML is a progressive form of MDS characterised by rapidly growing cancer of the blood and bone marrow. Around 30% of patients with MDS will progress to AML.
There were 1993 people newly diagnosed with MDS in England in 2004, with over 90% of patients aged over 60 years at the time of diagnosis. Median survival of patients with MDS is around 20 months, but can be less than 6 months for high-risk subgroups. Establishing the presence of chromosome 7 abnormalities is important as this is associated with rapid progression to AML.
The mainstay of treatment for MDS is best supportive care (BSC) (transfusions, growth factors, antibiotics) to control the symptoms of bone marrow failure, and low-dose standard chemotherapy for some patients. Stem cell transplant is not an option for the majority of patients as the patient’s age and/or comorbidities usually precludes this treatment option.
Scope of the ERG report
The scope for this STA was to address the clinical effectiveness and cost-effectiveness of aza relative to CCR, particularly BSC, low-dose chemotherapy (LDC) and standard-dose chemotherapy (SDC) in patients with higher risk MDS, CMML and AML with 20–30% blasts. The patient outcomes governing relative effectiveness and cost-effectiveness were defined as: overall survival, time- to-progression (TTP) to AML, adverse events and health-related quality of life (HRQoL).
The marketing authorisation indicates the dose and route of aza to be 75 mg/m2 subcutaneously daily for 7 days followed by a rest period of 21 days (28-day treatment cycle). It is recommended that patients be treated for a minimum of six cycles, continuing for as long as the patient continues to benefit or until disease progression. The unit cost of aza is £321/100 mg.
The key source of evidence on clinical effectiveness was an open-label randomised controlled trial (RCT) by Fenaux et al. 4 referred to as study AZA-001. It compared aza with CCR in 358 patients with higher risk MDS, CMML and AML 20–30% blasts. The outcomes reported in AZA-001 included overall survival, TTP to AML and adverse events. No HRQoL results were reported; however, outcomes likely to impact on HRQoL were provided (e.g. freedom from transfusion and rates of infection requiring intravenous antibiotics).
The manufacturer submitted a de novo economic model that was used to estimate the cost per quality-adjusted life-year (QALY) gained from aza in comparison with BSC, LDC and SDC. HRQoL utilities were obtained by mapping with a published algorithm to convert European Organisation for Research and Treatment of Cancer scores in Study CALBG 9221 into European Quality of Life-5 Dimensions values. Resource utilisation was based on expert opinion gathered from consultant haematologists in the UK, and costs were obtained from standard sources.
Methods
The ERG report comprised a critical review of the evidence for the clinical effectiveness and cost-effectiveness of the technology based upon the manufacturer’s/sponsor’s submission to NICE as part of the STA process. This work was undertaken in 8 weeks, beginning from 7 June 2009.
Because of the central importance of the AZA-001 study, it was formally fully appraised by the ERG, taking advantage of responses to requests for clarification from the manufacturer.
The ERG reran the submission’s search strategies after some modifications incorporating minor improvements.
The ERG analysed the submitted economic model (model 1) and identified a number of inconsistencies and errors within the model. The manufacturer submitted a revised model for analysis by the ERG. Using the issues identified in the earlier analysis, the ERG conducted those repairs to the revised model that were feasible within time constraints.
The submitted models estimated overall survival using log-logistic (baseline) and Weibull distribution fits. When estimating uncertainty the models assumed a one-to-one and approximately linear relationship in these parameters. The ERG considered that this likely underestimated variation in these parameters and therefore incorporated the actual correlation between parameters where made available.
The ERG ran an ERG-repaired version of the second submitted model encompassing both the original two-way comparisons proposed by the manufacturer and additionally probabilistic sensitivity analyses that, in ERG judgement, might more accurately reflect the full range of clinical options available for the patient population. Other changes were made to the model, but a detailed account cannot be provided here.
The ERG extracted overall survival data for all the patient subgroups examined in the manufacturer’s economic model and prepared Kaplan–Meier plots to indicate the inherent uncertainty in the observed data.
The ERG explored curve fits to the observed overall survival for patient subgroups using a greater range of distributions than those provided in the manufacturer’s submission.
The ERG tested the face validity of the curve fits for overall survival submitted by the manufacturer by extrapolating to the base-case time horizon (25 years) rather than to the 7.7 years shown in the manufacturer’s submission. The ERG also compared extrapolations of alternative curve fits.
The ERG extracted data relating to survival in the AML state and attempted to replicate the values provided in the manufacturer’s submission.
The ERG extracted observed data for TTP to AML and compared observed TTP with the modelled TTP used in the manufacturer’s economic model.
Results
Summary of submitted clinical evidence
The key source of evidence on clinical effectiveness was the AZA-001 open-label RCT4 comparing aza with CCR in 358 patients with higher risk MDS, CMML and AML 20–30% blasts.
The AZA-001 study showed that:
-
The median overall survival was 24.5 months on aza, compared with 15.0 months in the CCR group (p = 0.0001).
-
The response rates were low (complete remission 17% aza versus 8% CCR).
-
The median time to transformation to AML was greater in the aza group (17.8 versus 11.5 months; p < 0.0001).
-
Of patients who were RBC transfusion-dependent at baseline, 45% of those on aza became RBC transfusion-independent during the treatment period, compared with 11.8% in the CCR group (p < 0.0001).
Summary of submitted cost-effectiveness evidence
The first model submitted by the manufacturer provided the following base-case incremental cost-effectiveness ratio (ICER) values for investigator pre-selected subgroups:
The ERG concluded that this model was internally incomplete and was not fully executable, and apprised the manufacturer of a large number of errors and inconsistencies that resulted in submission of a second model that was accompanied by the base-case ICER values shown below:
Commentary on the robustness of submitted evidence
Concerning clinical effectiveness, the AZA-001 study was open to bias, particularly from lack of blinding and uncertainty about losses to follow-up. In addition there was no direct evidence on impact on HRQoL. There is no evidence for differences in effects between investigator pre-selected treatment groups.
With regard to cost-effectiveness, the ERG had serious concerns regarding the validity of survival inputs into the model. However, the overwhelming observation concerned the errors in the submitted model which were sufficiently severe and numerous that the credibility of the estimates of cost-effectiveness provided in the manufacturer’s submission was completely undermined.
Deficiencies identified in the first submitted model included serious coding errors preventing control of model assumptions. When these were corrected, the model was not functional and did not produce results under its base-case assumptions. Other issues included: the non-discounting of all cost data; minor deficiencies in the discounting of utility data; a lack of functionality to reproduce selected analyses in the manufacturer’s submission; a large amount of redundant material within the model; and incorrect or inappropriate characterisation of uncertainty in cost, utility and survival estimates.
The manufacturer was apprised of the ERG’s concerns regarding the model and submitted a second ‘modified’ model.
Although functionality was partially restored and discounting was improved in the second model, the ERG considered that several serious concerns remained unaddressed, the most important of these being: failure of the model to reflect treatment options available in clinical practice; mischaracterisation of the uncertainty in survival analyses; lack of face validity regarding base-case inputs for overall survival; and questionable reliability regarding the TTP to AML.
In view of these concerns, the ERG had little confidence in the deterministic or probabilistic analyses submitted.
The ERG fixed the deficiencies remaining in the model as much as was possible within the remit of the STA and ran this version in probabilistic sensitivity analyses to generate cost-effectiveness acceptability frontiers. The ERG considered these analyses better reflected the treatment options likely to hold in clinical practice than did the two-way comparisons undertaken in the manufacturer’s analyses for those who would otherwise receive chemotherapy options (SDC, LDC). The results of these exploratory analyses indicated that:
-
For SDC-treated patients, of six treatment options available, BSC was likely the most cost-effective option up to a threshold of £51,000/QALY. Beyond £51,000/QALY, aza + LDC became cost-effective.
-
For LDC-treated patients, of four options available, BSC was again the most cost-effective option up to a willingness-to-pay threshold of £51,000/QALY (aza + LDC became cost-effective after £51,000/QALY).
-
For BSC-treated patients, aza + BSC became cost-effective relative to BSC at a threshold of about £52,000/QALY.
The ERG considers these results exploratory and considers that they should be viewed with caution because of concerns regarding various biases relating to the TTP to AML, the uncertainty associated with the parameters describing fitted curves for overall survival from the trial, the effect of age-related non-MDS/AML mortality, and the impact of revised Health Resource Group figures.
Conclusions
The AZA-001 study showed that, compared with CCR, those MDS patients receiving aza had prolonged median survival (by about 9 months), had delayed progression to AML, had reduced dependence on transfusions and had a small improvement in response rate. As an open-label design, this study was at risk of bias and there was concern regarding losses to follow-up; these considerations may indicate some overestimation in the survival benefit of aza.
Aza reduces the requirement for transfusion and for intravenous antibiotic administration, and the claim has been made that ‘azacitidine results in a marked improvement in patient well-being’. There is no direct research evidence about well-being of the patient population of interest in this STA, and research on quality of life for MDS patients is clearly required.
The economic models submitted for assessment were flawed and the cost-effectiveness of aza versus CCR was unlikely to be reliably estimated using the manufacturer’s submitted models. Exploratory analyses using an improved version of the manufacturer’s model indicated that in various scenarios aza was unlikely to become cost-effective relative to competing treatment strategies at a willingness to pay of less than £51,000/QALY.
Given the general paucity of economic modelling work in MDS and the limitations of the submitted industry model there is an evident need for an independent cost-effectiveness analysis of aza in MDS.
Note
Because of the extensive inconsistencies and errors within the model first submitted by the manufacturer the ERG presented a relatively brief initial report to NICE indicating that due to model inadequacies no reliance could be placed on the submitted cost effectiveness estimates. The report also encompassed a critical appraisal of the single RCT used in the manufacturer’s submission. This report was sent to NICE in line with contractual time lines. Subsequently ERG received a second economic model submitted by the manufacturer. This second model was appraised by the ERG and an addendum to the original ERG report was then submitted to NICE in time for the first committee meeting. This addendum contained a substantial critique of the survival analyses underpinning the second economic submission together with an appraisal of the economic model which unfortunately retained several deficiencies.
Summary of NICE guidance issued as a result of the STA
At the time of writing, the guidance appraisal consultation document iKey references ssued by NICE on 4 March 2010 states that:
-
1.1 Azacitidine is not recommended as a treatment option for people who have the following conditions and are not eligible for haemopoietic stem cell transplantation: intermediate-2 and high-risk myelodysplastic syndromes according to the International Prognostic Scoring System (IPSS); chronic myelomonocytic leukemia with 10-29% marrow blasts without myeloproliferative disorder or acute myeloid leukemia with 20-30% blasts and multilieage dysplasia, according to the World Health Organization classification.
-
1.2 People with conditions stated in 1.1 who are currently receiving azacitidine for myelodysplastic syndromes, chronic myelomonocytic leukemia or acute myeloid leukemia should have the option to continue treatment until they and their clinicians consider it appropriate to stop.
Disclaimers
The views expressed in this publication are those of the authors and not necessarily those of the HTA programme or the Department of Health.
Key References
- NICE . Guide to Single Technology (STA) Process 2006. www.nice.org.uk/page.aspx?o=STAprocessguide.
- Edlin R, Connock MJ, Round J, Tubeuf S, Fry-Smith A, Hyde C, et al. Azacitidine for the Treatment of Myelodysplastic Syndrome, Chronic Myelomonocytic Leukaemia and Acute Myeloid Leukaemia 2009. http://guidance.nice.org.uk/TA/Wave18/19.
- Celgene . Single Technology Appraisal (STA) of Vidaza® (azacitidine); [for the Treatment of Myelodysplastic Syndrome, Chronic Myelomonocytic Leukaemia and Acute Myeloid Leukaemia] 2009. http://guidance.nice.org.uk/TA/Wave18/19.
- Fenaux P, Mufti GJ, Hellstrom-Lindberg E, Santini V, Finelli C, Giagounidis A, et al. Efficacy of azacitidine compared with that of conventional care regimens in the treatment of higher-risk myelodysplastic syndromes: a randomised, open-label, phase III study. Lancet Oncol 2009;10:223-32.