Evidence Review Group approaches to the critical appraisal of manufacturer submissions for the NICE STA process: a mapping study and thematic analysis

Topic area

Single technology appraisals are categorised by topic on the NICE website. There are 21 topic categories. Table 1 shows the categorisation of topics identified in the mapping exercise. Some topics are listed under more than one category, for example ‘pemetrexed for the first-line treatment of locally advanced or metastatic non-small cell lung cancer’7 is listed under cancer and respiratory. Thirty-one STAs were listed in more than one category. The majority of topics were cancer topics. Under the cancer category, some STAs, such as those concerning satraplatin for hormone-refractory prostate cancer and capecitabine for advanced pancreatic cancer, were not listed on the NICE website and the topic category was assigned on the basis of the title of the appraisal by the authors of this report.

Single technology appraisal report and project status

Of the 95 STAs identified, 36 were categorised as completed STAs (38%), 28 were in progress (29%), 22 (23%) had been suspended, eight terminated (8%) and one had been completed but subsequently the guidance was withdrawn (1%). Three of the 95 STAs (satraplatin for hormone-refractory prostate cancer, etanercept for moderate to severe chronic plaque psoriasis in children and adolescents and rimonabant for type 2 diabetes mellitus) had passed through the topic selection process but were not referred to NICE. See Figure 2 for STA project status.

FIGURE 2.

Single technology appraisal project status.

Of the 28 in-progress STAs, eight had completed ERG reports, four had ERG reports not yet completed and 16 ERG reports had not yet commenced (Figure 3).

FIGURE 3.

Evidence Review Group report status.

Of the 31 STAs that had been suspended or terminated or had their guidance withdrawn, three ERG reports had been completed. For two of these appraisals (mifamurtide for osteosarcoma and romiplostim for thrombocytopenic purpura) the appraisals were suspended due to delays in drug launch. For the third appraisal (rimonabant for type 2 diabetes mellitus) the guidance was withdrawn.

Manufacturer

Sixty-four STAs had a named manufacturer, while the remaining 31 did not have a manufacturer reported on the NICE website, usually because the STA process for those topics was still in the early stages. Twenty-seven manufacturers have been involved in the STA process to date. Table 2 lists manufacturers and the numbers of STAs they have been involved in.

Evidence Review Group team

Sixty-three of the STAs had been assigned an ERG team. Five teams undertook 10 or more ERG reports and the remaining two teams undertook three and five reports (Figure 4).

FIGURE 4.

Evidence Review Group teams.

Incremental cost-effectiveness ratios reported

Thirty-six completed STAs and seven in-progress STAs contained figures for the base-case incremental cost-effectiveness ratios (ICERs) reported by the manufacturer within their submission. Thirty-three completed STAs and seven in-progress STAs included figures for base-case ICERS reported by the ERG within their report.

Significant differences were seen when the manufacturer and ERG base-case ICERs were compared. The manufacturer base-case ICERs were > £30,000 per quality-adjusted life-year (QALY) in 13 cases and < £30,000 per QALY in 30 cases. In contrast, the ERG base-case ICERS were reported as > £30,000 per QALY in 25 cases and < £30,000 per QALY in 15 cases (Figure 5).

FIGURE 5.

Base-case ICERs reported by ERG and manufacturer.

In three ERG reports, a clear figure for the base-case ICER calculated by the ERG was not reported. 8–10 One ERG report stated that a £30,000-per-QALY threshold analysis was conducted (natalizumab for the treatment of adults with highly active relapsing–remitting multiple sclerosis8); one ERG report estimated two base-case ICERs of > £30,000 and > £100,000 per QALY (pemetrexed for the treatment of relapsed non-small cell lung cancer9); and one ERG report stated that the ERG was unable to provide revised ICERs (pemetrexed for the first-line treatment of locally advanced or metastatic non-small cell lung cancer10).

Appraisal consultation document decision

The AC can make various recommendations in the ACD. Obviously there can be a clear ‘yes’, a clear ‘no’ or a ‘minded’ opinion. A ‘minded’ opinion usually means that the AC is minded to say ‘yes’ or ‘no’ but that it requires further information, usually from the manufacturer, to make its decision at the subsequent AC meeting. There were 38 STAs with ACD decisions available on the NICE website (31 completed STAs and seven in-progress STAs). Nineteen had a ‘no’ recorded as the ACD decision, nine were recorded as ‘yes’ decisions and seven had ‘minded no’ decisions within the ACD (Table 3). Two STAs had both a ‘yes’ and a ‘minded no’ decision recorded within the ACD and one STA had both a ‘no’ and a ‘minded no’ recorded in the ACD.

Appraisal consultation documents were not made available on the NICE website for five completed STAs11–15 (trastuzumab as adjuvant therapy for early-stage breast cancer;11 varenicline for smoking cessation;12 dabigatran etexilate for the prevention of venous thromboembolism in patients undergoing elective hip and knee surgery;13 rituximab for the first-line treatment of low-grade follicular non-Hodgkin’s lymphoma;14 and rivaroxaban for the treatment of venous thromboembolism15). However, ACDs are issued only if the AC recommendations are restrictive or if the manufacturer is requested to provide further clarification, and therefore ACDs are not issued for all STAs.

Two STAs had two ACDs, of which one had a ‘no’ decision recorded in its first ACD and then a ‘yes’ decision in the second ACD,16 and the other had a ‘no’ decision recorded in its first ACD and then a ‘yes’ decision with conditions in the second ACD. 17

Final appraisal determination decision

Thirty-six completed STAs and one in-progress STA had FAD decisions. Twenty-five STAs were recorded as ‘yes’ decisions, whereas 12 were recorded as ‘no’ decisions. Thirty-two STAs had both ACD and FAD documents available on the NICE website. Table 4 shows the initial ACD decision (taken from the first ACD if more than one ACD) and subsequent FAD decision in the 32 STAs with both ACDs and FADs. All ‘yes’ decisions and 10 out of 14 ‘no’ decisions remained the same between the ACD and FAD. ‘Minded no’ and ‘yes/minded no’ decisions within the ACDs almost always became ‘yes’ decisions within the FAD. The one exception was a ‘minded no’ to ‘no’ decision between the ACD and FAD. In this case, a subsequent appeal was upheld. After additional work was undertaken by the DSU, the AC met again and the ‘no’ decision in the FAD was subsequently changed to a ‘yes with conditions’ (erlotinib for the treatment of relapsed non-small cell lung cancer18).

Referral date to final appraisal determination

The NICE STA process guide3 states on p. 18, in section 3.2.1, that the STA process begins ‘after NICE has received formal referral from the Secretary of State for Health’. However, no guidance is given with regards to the length of the STA project between the referral date and the FAD. Week 0 on the STA project timeline is when manufacturers are invited to complete an evidence submission or when the final scope is issued. The time from final scope to FAD should be 34 weeks or less, as stated in the 2009 STA process guide3 (when there is no appeal). In the original STA process guide (2006)19 this was stated to be 35 weeks.

The time period between the referral date and final scope date (i.e. week 0) ranges considerably between 4.4 and 83 working weeks (see Table 5). The length of the entire STA project was taken as the number of working weeks between the referral date and the FAD date. The mean number of working weeks between the referral date and the FAD in the 36 completed STAs was 54.9. This ranged considerably between 30.4 and 131.0 working weeks. The majority of completed STAs (27/36) took between 39 and 60 weeks to complete from their referral to FAD.

Final scope to final appraisal determination

The 35-week final scope to FAD time period for a completed appraisal was used in this analysis, as some of the STAs were completed before 2006. A 2-week grace period was added to allow for holiday periods and other delays, making the cut-off point 37 weeks for a completed, on-time STA. Scopes (and scoping workshops) were not part of the initial STA process and seven STAs did not have a final scope available on the NICE website. It was therefore not possible to calculate the time period between the issue of final scope and FAD in these seven STAs. Eight of the 29 STAs where it was possible to calculate time to completion were completed within 37 working weeks, i.e. 28% of STAs were completed within a period that could be considered ‘on time’ when the final scope date is taken as the start of the process and there is no appeal process. The mean number of working weeks between the final scope and FAD in the 36 completed STAs was 44.0. This ranged considerably from 26.2 to 122.2 weeks.

The scope to FAD period for 12 STAS was delayed between 2 and 5 weeks. Possible reasons made available on the NICE website for the 2- to 5-week delay were provided for two STAs only and included appraisal suspension (omalizumab for severe persistent allergic asthma20) and transfer from the 10th to the 12th wave (natalizumab for the treatment of adults with highly active relapsing–remitting multiple sclerosis8). A third STA was recorded to have a 2- to 5-week delay between scope and FAD (docetaxel for adjuvant treatment of early breast cancer21). However, this STA had been transferred from the MTA to the STA process and thus the final scope date related to that of the original MTA (scopes were not initially provided for STAs). There did not appear to be any reasons provided on the NICE website as to why the remaining nine STAs were delayed between 2 and 5 weeks. However, it was noted that four of the STAs required extra work from the manufacturer22–25 (adalimumab for the treatment of moderate to severe psoriatic arthritis,22 rituximab for the treatment of refractory rheumatoid arthritis,23 rituximab for the treatment of recurrent or refractory stage III or IV follicular non-Hodgkin’s lymphoma,24 and entacavir for the treatment of chronic hepatitis B25) and extra work was undertaken by the ERG for one STA26 (cetuximab for the treatment of metastatic and/or recurrent squamous cell carcinoma of the head and neck26).

The scope to FAD period was delayed by > 5 weeks for two STAs,7,27 for which no apparent reasons were provided on the NICE website (ustekinumab for the treatment of moderate to severe psoriasis27 and pemetrexed for the first-line treatment of locally advanced or metastatic non-small cell cancer7). A further seven STAs had scope to FAD periods delayed by > 10 weeks. Reasons for scope to FAD periods being delayed by > 10 weeks were provided for six of these STAs and included appeal proceedings (erlotinib – non-small cell lung cancer,18 abatacept,28 febuxostat29), additional work undertaken (erlotinib18), end-of-life consideration with production of two ACDs (lenalidomide16), first AC meeting cancelled owing to clarification of European regulatory timings (febuxostat29), timings reset at the request of the manufacturer to allow for the outcome of discussions with the Committee for Medicinal Products for Human Use of the European Medicines Agency to be incorporated in the MS (abatacept28), third and fourth AC meetings (cetuximab for colorectal cancer30) and a FAD returned with comments after the second AC (sunitinib31). There were no apparent reasons stated on the NICE website to explain why the scope to FAD period for the infliximab for ulcerative colitis32 STA was delayed by 14.2 weeks.

In addition, seven STAs are delayed and were still in progress at the time of data extraction (August 2009):

• Topotecan for the treatment of recurrent and stage IVB carcinoma of the cervix (third AC meeting for end-of-life consideration). 33

• Tocilizumab for the treatment of juvenile idiopathic arthritis (delay due to pricing issues, guidance delayed as a result). 34

• Bevacizumab in combination with non-taxane chemotherapy within its licensed indications for the first-line treatment of metastatic breast cancer (rescheduled to align with regulatory approval). 35

• Trabectedin for the treatment of advanced metastatic soft tissue sarcoma (extension granted for manufacturer to meet full requirements of the STA process). 36

• The clinical effectiveness and cost-effectiveness of erlotinib monotherapy for maintenance treatment of non-small cell lung cancer after previous platinum-containing chemotherapy (rescheduled to align with regulatory expectations). 37

• The clinical effectiveness and cost-effectiveness of erlotinib in combination with bevacizumab for maintenance treatment of non-squamous advanced or metastatic non-small cell lung cancer after previous platinum-containing chemotherapy (rescheduled to align with regulatory expectations). 38

• Rituximab for relapsed treatment of chronic lymphocytic leukaemia (delays due to variation to marketing authorisation). 39

If these delays had been taken into account then the overall time frame for the 95 STAs would have been lengthened.

Manufacturer submission to Evidence Review Group report submission

The appendix B document in the NICE STA process guide3 (p. 56) allows 8 weeks between the manufacturer report submission and ERG report submission. The NICE guidance also stipulates that ERG teams will be given a minimum of 8 weeks to complete the ERG report. Eighteen (50%) of the completed STA ERG reports were completed within 8–10 working weeks and a further 14 (39%) STA ERG reports were submitted 10–12 working weeks after receipt of the MS. The mean number of working weeks between the manufacturer report submission and ERG report submission in the 36 completed STAs was 10.5. This ranged considerably between 8.2 and 23.8 working weeks. The majority of STAs (32/36) took 8–12 working weeks between the manufacturer report submission and ERG report submission.

Evidence Review Group report submission to first Appraisal Committee

The first AC is scheduled to take place 4 weeks after the ERG report has been submitted, and this was the case for 5 out of the 36 completed STAs. The first AC meeting was 4–5 weeks after the ERG report submission in 15 STAs and 5–7 weeks in 13 STAs. The mean number of working weeks between the ERG report submission and first AC in the 36 completed STAs was 5.3. This ranged considerably between 3.4 and 11.2 working weeks.

First Appraisal Committee to second Appraisal Committee

The second AC is scheduled to take place 8 weeks after the first AC. The mean number of working weeks between the first AC and second AC in the 36 completed STAs was 10.4. This ranged considerably between 8.2 and 22 working weeks. The time period between the first AC and second AC in two-thirds of completed STAs (24/36) was between 8 and 10 working weeks.

Second Appraisal Committee to final appraisal determination

The FAD is published (if no appeal is received) 10 weeks after the second AC meeting, although a second AC meeting is not held if an ACD is not produced at the first AC meeting. In all but four STAs this was achieved. The mean number of working weeks between the second AC and the FAD in the 36 completed STAs was 8.2 working weeks. This ranged considerably between 3 and 29.8 working weeks. The time period between the second AC and the FAD was 3–5 weeks in six STAs and 5–9 weeks in 20 STAs.

Manufacturer

After the first AC meeting, in 11 out of 36 of the completed STAs and in one out of seven of the in-progress STAs, the manufacturer was requested to complete further work before the second AC meeting.

Evidence Review Group

Before the first AC meeting, in 5 out of 36 of the completed STAs and in three out of seven of the in-progress STAs, the ERG was requested to complete work in addition to the ERG report. After the first AC meeting, in 4 out of 36 of the completed STAs and in one out of seven of the in-progress STAs, the ERG was requested to complete further work before the second AC meeting.

Decision Support Unit involvement

Three STAs had involvement from the NICE DSU:

• cetuximab for first-line treatment of metastatic colorectal cancer30

• erlotinib for the treatment of relapsed non-small cell lung cancer18

• lapatinib for breast cancer for use in women with previously treated, advanced or metastatic breast cancer. 40

In one of these STAs it appears that the decision of the AC changed from ‘no’ in the ACD to ‘yes’ with restrictions in the FAD (cetuximab30). For one of the STAs (erlotinib18) the decision changed from ‘minded no’ at ACD to ‘no’ at FAD; however, after additional work undertaken post appeal the decision was changed to ‘yes’ but with restrictions. For lapatinib,40 the decision remained ‘no’ at FAD although at the time of writing this STA is still ongoing.

Dates

Full dates (day/month/year) were often not recorded within documents published on the website; often only the month and year were available and this made calculation of timings difficult. Where exact dates were not published within documents, such as the scope and FADs, we used the 28th of each month as the date of issue. Therefore, there is some inaccuracy and underestimation within these analyses.

There were also other inconsistencies with dates. Often the month of issue within the scope and FAD documents was not the same as the month stated on the website. Examples of this are romiplostim for the treatment of chronic immune or idiopathic thrombocytopenic purpura (ITP)44 and paclitaxel for the adjuvant treatment of early breast cancer. 45

Documentation

Some documents were missing from the NICE website, such as ACDs and final scopes. This could be due to the fact that initially scoping documents and ACDs were not part of the STA process and were later additions. In some appraisals there were two FAD documents and it was not always clear why.

Transparency and inconsistencies

Reasons for delays, suspensions or terminations are not always clear on the NICE website. As each appraisal does not have a unique identifying number, and there are several appraisals for the same drug within different conditions, it was sometimes difficult to find information on a specific appraisal.

Three STAs in this mapping exercise had passed through the topic selection process but were not referred to NICE: two of these are listed on the NICE website, although difficult to find, while one is not mentioned at all (etanercept for moderate to severe chronic plaque psoriasis in children and adolescents).

As described previously, there is inconsistency in the way STAs are categorised by topic. The terms ‘suspended’ and ‘terminated’ are not used consistently. NICE continues to monitor topics that are referred to it even if the appraisal has ended. STAs also occasionally have different titles within different documents relating to the same appraisal that occasionally caused confusion.

Description of manufacturers’ submissions

The numbers of pivotal trials included in the MSs are shown in Figure 6. Less than half of MSs included only one pivotal trial.

FIGURE 6.

Number of pivotal trials included by manufacturers.

Eight manufacturers (27%) included a meta-analysis in their submission, although as 14 out of 30 submissions included only one pivotal trial then a meta-analysis would not have been appropriate in these submissions. Nineteen manufacturers (63%) undertook an indirect comparison, 29 (97%) undertook SAs and 28 (93%) carried out probabilistic sensitivity analyses (PSAs).

Description of Evidence Review Group approaches to critical appraisal of clinical evidence

In 26 (87%) ERG reports the team commented on the searches in the MS and 14 (47%) replicated the searches. Methods used to critically appraise the manufacturer’s systematic review were not always clear. Thirteen (43%) ERG reports stated that a published critical appraisal tool was used to assess the quality of the review. These included the QUOROM (quality of reporting of meta-analyses),49 CRD,50 BMJ (British Medical Journal)51 and ScHARR52 tools. Figure 7 shows the tools used in the 13 ERG reports reporting that published tools were used.

FIGURE 7.

Published tool used to appraise quality of manufacturer’s systematic review.

Only 11 (37%) used a published critical appraisal tool to assess the quality of the manufacturer’s included trials. In 12 (40%) methods for assessing the quality of the trials were unclear. Figure 8 shows the tools used in the 11 ERG reports reporting the use of a published tool, which included the CASP (Critical Appraisal Skills Programme),53 NICE1 and CONSORT (consolidated standards of reporting trials)54 checklists.

FIGURE 8.

Tools used in 11 ERG reports to assess trial quality.

In 11 (37%) ERG reports the team undertook additional analysis of the clinical data. Table 6 shows the reported number of clinicians used by the ERGs, either listed as an author or mentioned in the acknowledgement section in the 30 ERG reports.

Description of Evidence Review Group approaches to critical appraisal of cost-effectiveness evidence

Sixteen (53%) of the ERG reports commented on the manufacturers’ searches for economic evidence and five (17%) replicated the searches. Only two (7%) of the ERG reports stated that a published tool was used to critically appraise the quality of included cost-effectiveness studies. In both cases the tool was the Drummond checklist. 55 Fourteen (47%) reports stated that the MS followed the NICE reference case1 (see Appendix 3). Table 7 shows the reported critique of the MSs in the ERG reports from the point of the view of items in the NICE reference case. Table 8 provides an overview of the elements used in the critique of the submitted model(s) by the ERGs. In the 30 ERG reports included in this study, 23 (77%) reported that extra cost-effectiveness analyses were undertaken. Table 9 shows the frequency of various approaches reported by the ERG teams to assess the models presented in the 30 MSs.

Processes

The ‘processes’ theme emerged from five subthemes, which were interpretations of ERG comments on the conduct or performance of the reviewing and modelling in the submissions and, in particular, the conduct of the analyses. How well or how badly the manufacturer performed these key processes for the submission was a frequent source of criticism or comment within ERG reports. The five subthemes identified from the data were the conduct of both the review and the modelling processes; the inadequacy or inappropriateness of the analyses performed; and reported concerns with the data that were used in the analyses.

More than half of the ERG reports (17/30) explicitly criticised the conduct of the review, for example the quality of the searching, screening or quality assessment processes (1, 2, 5, 11, 13–17, 20, 27); the definition or application of inclusion criteria, especially for indirect comparisons (5, 7, 13, 15–17, 19, 20); the presence of errors in, or a failure to perform, meta-analyses (5, 16, 19); failure to control for confounders (6, 28) or perform a key subgroup analysis (28); or even the complete absence of a formal systematic review (9, 30). On some occasions, elements of the review process, such as searching and data extraction, prompted positive comments from an ERG (8, 10, 19, 20, 22, 25) or a statement that the review generally was reasonable or good (21, 26).

Evidence Review Group reports rarely commented explicitly positively on the approach taken in the submission to reviewing, modelling or analysis. However, half of all ERG reports (15/30) did pass positive comments on the appropriateness of the structure of the economic model presented or the reasonableness of the modelling approach taken (8, 9, 12, 14, 16–19, 21–23, 25–27, 30), albeit often with some criticism of certain elements of the model. Such weaknesses, highlighted in 20 ERG reports, included a failure to incorporate or capture adequate levels of uncertainty (1, 10, 17); calculation errors in the model requiring revision or correction (2, 3, 25); other technical, structural or design errors in the model or SA (4, 5, 7, 10, 11, 15, 17, 21, 22, 28, 30); and the failure to control for confounders (6) or queries over a key assumption (13), missing data (13, 17, 24, 29) or the absence of a SA (16).

The analyses performed by the manufacturers and reported in the submissions constitute another element of the effectiveness review process that was frequently critiqued by the ERGs. Criticisms of the inadequacies of the analyses presented consisted of the failure to address heterogeneity of trials (5); the failure to perform a relevant analysis, for example a meta-analysis or SA or PSA (6, 7, 16, 18, 19, 21, 23, 27); no validation of the model (6, 13); or the inadequate analysis of safety outcomes (15). However, it was not simply the failure to perform a necessary analysis, but rather the performance of an inappropriate analysis also, that generated criticism in 17 ERG reports, i.e. that the combination, pooling or comparison of effectiveness data was highly questionable (2, 7, 11, 17, 20, 21, 23–25) or that the methods used for the reported analysis (6, 8, 11, 13, 15, 20, 21, 27) or modelling (1, 5, 7, 29) were deemed to be inappropriate.

The other major criticism relating to analyses contained within submissions focused on issues with the data being used, especially the data used in the cost-effectiveness model. Two-thirds of ERG reports (20/30) contained such criticisms. These criticisms related to the efficacy data being used from both direct and indirect comparisons (8, 9, 14, 17, 18, 20, 23, 24); the cost data used (1, 5, 7, 18–20, 30); the utility or QoL data (1, 25, 27, 29, 30); the data on population, comparators, outcomes or various model parameters (5, 6, 8–10, 12, 18, 19, 23, 28); the use of unpublished data (23, 25, 29); and the data being used in the SAs or PSAs (7, 21). By contrast, the data used in the direct and indirect comparisons performed for the effectiveness review generated explicit criticism in only four ERG reports (5, 24, 25, 27).

Reporting

The ‘reporting’ theme was derived from explicit comments made in ERG reports on the quality of the manufacturers’ description of the conduct of both the reviewing and the modelling. Only in a small number of cases did ERGs make explicit, positive comments on the description of the technology or processes within MSs. Nine ERGs commented on the accuracy and adequacy of the background section provided in the submission (6–8, 10, 11, 13, 20, 27, 29), albeit in some cases stating that further information would still have been useful (11, 13, 20, 27). The reporting of the searches for the effectiveness review was praised in four ERG reports (3, 21–23) and the description of the model and its data sources in five ERG reports (21–23, 26, 29). One ERG specifically described good reporting of statistics (1), and another a well-reported multiple treatment comparison process (18).

It was far more common, however, for ERGs to perceive as inadequate the reporting or description of the processes being undertaken or the sources of data being used. A total of 26 out of 30 ERG reports described some form of inadequacy in the reporting of a submission. These included poor descriptions of the searches undertaken, prohibiting replication, for both clinical effectiveness and cost-effectiveness reviews (2, 4–6, 8, 20, 26, 27); an explicit criticism in seven ERG reports regarding the lack of transparency of the clinical review processes generally, for both direct and indirect comparisons (e.g. screening, extraction and quality appraisal) (2, 5, 7, 11, 12, 20, 23); and, quite frequently, in eight ERG reports, inadequate reporting concerning the number of included studies and their characteristics for both clinical effectiveness and cost-effectiveness reviews (2, 4, 18, 19, 22–25). Five ERGs also considered the background section to be inadequate and lacking key information (14–16, 29, 30). The description of the analyses in submissions was also a major source of criticism, i.e. the failure to report data and data sources in full (1, 29, 30); the failure to describe all of the methods being used in direct (5, 20, 24, 27) or indirect comparisons (12, 17–19); the failure to provide adequate descriptions of the analyses more generally (6, 11, 18), and the poor reporting of included studies and data in tables (4, 5, 16). However, the submissions’ reporting of the cost-effectiveness model was the most common criticism by ERGs. Eleven submissions were deemed to be affected by such inadequacies as a failure to describe adequately the parameters or assumptions behind the model, the generation or source of various values, or the impact of bias from missing data (4, 11, 17–22, 27, 28, 30).

Satisfaction of objectives

The ‘satisfaction of objectives’ theme emerged from issues surrounding the relationship between the decision problem and whether, or how far, the elements of the decision problem were satisfactorily addressed in the MSs. The five subthemes identified under this theme relate to the objectives determined by the decision problem, i.e. the population, intervention, comparator, outcomes and the NICE base case.

Ten submissions were not criticised for the relevance of their included trial populations to the populations required by the scope, but 20 out of the 30 ERG reports did raise issues with the trial populations being presented and considered in the submission (1–3, 6, 7, 9–11, 15, 16, 18–22, 25–28, 30). These concerned differences between the population defined in the decision problem and the population being evaluated in the submissions’ trials (3, 9–11, 19, 20, 22, 25–27, 30); differences between the UK population and the trial population based on age, treatment or current practice (1, 2, 6, 7, 15, 27); differences between the licensed population and the trial population (9, 28); a failure to consider the effect of the treatment on different, relevant subgroups of patients (1, 7, 18, 19, 21, 30), including in the model (18); and, finally, problems with the definition of the population in the submission (16, 20, 26, 30), again including in the model (26).

The second major disparity between submissions and the requirements of the decision problem concerned the comparators. Twelve ERG reports found nothing to criticise in this regard, but 18 did raise various issues with the comparators considered in submissions. The principal issue involved a submission’s failure to consider one or more of the comparators designated in the decision problem, or the submission’s use of a combination of comparators not admitted in the decision problem: this was raised in 13 reports (1, 3–5, 7–9, 11, 13, 15, 17, 22, 30). In two cases, this issue was raised for the cost-effectiveness model too (8, 11). Three reports were concerned that the comparator being evaluated was not in use in the UK (6, 15, 26) or that a non-optimal dose of the comparator was being assessed (26), and two reports also identified problems with a lack of definition for the chosen comparator (25, 30). One submission reported a non-optimal treatment duration for the comparator in the model (28) and one conducted a mixed-treatment comparison (MTC) using comparators other than the principal comparator named in the scope (18).

By contrast, few ERGs reported a submission’s failure to satisfy the intervention or outcome elements of the decision problem. Twenty-one ERG reports found no problem with the intervention being evaluated in the submission. In nine instances the actual definition of the intervention was an issue, i.e. the inclusion or failure to include the intervention as combination therapy or monotherapy (1, 16, 18, 29, 30); differences between the trial interventions and UK practice (22, 26); differences between the licensed intervention and the trial intervention (3); and a specific instance where the dose being evaluated was an issue (11). Twenty-one reports (1–7, 9, 10, 12, 13, 18, 19, 21, 23–28, 30) also did not have any criticism to make of the outcomes presented in the submissions. The principal issue in the remaining reports concerned uncertainties regarding the appropriateness of the outcome being measured (8, 11, 14, 20, 22, 29). Otherwise, comments were limited to a lack of clarity on how QoL was being measured (14, 16), the inadequacy of the safety measures presented (15), a failure to report outcomes (17) or concerns about a trial’s lack of power to detect the secondary outcomes being presented (17).

The vast majority of the issues relating to the satisfaction of objectives were raised in the appraisal of the clinical effectiveness evidence. Very few, for example only those relating to the measurement of QoL as an outcome (14, 16), were issues raised principally or exclusively in the cost-effectiveness sections of the reports. In relation to satisfying objectives, the principal focus of the cost-effectiveness section concerned the submission’s failure or otherwise to satisfy the requirements of the NICE base case when developing the model. The majority of submissions appear to have adhered to the NICE base-case scenario and prompted no criticism: only eight ERGs reported either multiple deviations from the base case (23, 29) or specific deviations in relation to either the calculation of utilities (3, 5, 13) or the comparator (17, 20, 27).

Reliability and validity

The ‘reliability and validity’ theme emerged from four subthemes that were interpretations of ERG comments on the robustness or limitations of the submissions’ findings. The four subthemes are uncertainty due to the possibility of bias or an exaggerated estimate of effect; uncertainty due to the absence of evidence; the reliability (rather than uncertainty) of the findings; and external validity, i.e. how far the ERG considered the submission to be externally valid for the intended population and service.

The level of uncertainty surrounding the findings presented by the submissions was a frequent cause of criticism in ERG reports. A total of 27 out of 30 ERG reports (1–13, 15–18, 20–24, 26–30) stressed the presence of bias within the analyses, thus highlighting the lack of certainty surrounding the results presented in the submissions. The principal causes of uncertainty identified by ERGs concerned the possible exaggerated effect of the technology from the analysis (13, 15, 21, 27, 29), especially the relative efficacy of the technology versus relevant comparators (1, 5, 9, 11, 12, 23, 24); the safety of the technology (11); and uncertain levels of risk for different populations (30). These uncertainties within the clinical effectiveness analyses impacted on the models, which were also subject to other biases, such as uncertainties due to issues with the parameters or values in the model (3, 4, 6, 7, 15, 17, 20, 24, 28, 29); an exaggeration or overestimate of benefits or costs (1, 2, 8, 9, 18, 22, 23); excessive extrapolation from the data (2, 9, 16, 26); errors in the model itself (10, 13, 16); and uncertainty due to the model’s high degree of sensitivity to values or assumptions within the model (2, 5, 8, 12, 20, 21).

By contrast, few ERG reports stated that it was the lack of evidence rather than the quality of the evidence or its analysis that generated uncertainty in the submissions’ findings. Only seven ERG reports stated that the available trial evidence was insufficient to generate robust findings on follow-up or treatment duration (3, 4, 14, 16, 28), including survival in the model (1, 14). There was also uncertainty regarding relative efficacy of technologies as no head-to-head trial had been performed (3, 23, 30) or because the findings were based on only a single trial (21). ERG reports did sometimes also explicitly comment that the findings of the effectiveness or cost-effectiveness analyses were strong and reliable, but this was relatively infrequent. In seven reports, ERGs stated that the submission offered a convincing case for the technology (3, 5), an unbiased estimate of treatment effect (17–19) or a fair interpretation of the trial data (26, 28). In terms of the cost-effectiveness analysis, only three ERG reports considered that the submission presented a reasonable estimation of the cost-effectiveness of the technology versus relevant comparators (10), that the modelling of the costs of the technology appeared sound (15), or that the model was superior to previous published models (28).

Finally, the external validity of the submission’s findings was not criticised in 13 ERG reports, but was certainly queried explicitly by 17 ERG teams. These issues principally related to differences between the trial and UK populations (1, 6, 12, 15, 26, 27), including subgroups of UK patients likely to receive the treatment (7); differences between the treatment practices being evaluated in trials and clinical practice in the UK (2, 8, 14, 15, 22, 26) and those in Europe (5); or, simply, differences between the trials and ‘real-world’ clinical practice generally (10). One model was also criticised for not reflecting ‘real-world’ decisions (11), two others for using non-UK sources of data for the model (13, 16) and one ERG reported that the efficacy findings could not be extrapolated to all of the PSA populations (28).

Content

The ERGs often commented on the amount and quality of the trial evidence that formed the basis for the submission, as this affected both the internal and external validity of the submission and its reviews. However, unlike the other themes, this element is not readily addressed by the improved conduct of reviews, analyses, modelling or reporting of the processes used. Nevertheless, it is an element of the submission that attracted comment from ERGs and so is represented in this analysis. The limited amount of relevant trial evidence, for example the fact that the submission may be based on the evaluation of a single randomised controlled trial (RCT), or perhaps two such trials only, was explicitly raised as a point by 10 ERGs (1, 3, 7, 17–19, 21, 22, 26, 28). The absence of any head-to-head trials of relevant technologies was another issue that was sometimes specifically mentioned by ERGs (3, 9, 22). The quality of the included trials was also often commented on by ERGs, as this had implications for the validity of the review, model and submission: included trials, even if there was only one, were more often explicitly reported as being of good or reasonable quality (1, 8–12, 17, 18, 22) than low quality (14). Small sample sizes in trials (22) and limited follow-up (20, 26) were other factors affecting the amount and quality of the evidence that drew comments from ERGs.

General queries

As would be expected, there were instances when there was a need to clarify the specific data presented in the submission:

Table 5.7.1 – provide a definition of (any) severe adverse event.

Please be explicit that the **** trial is currently published as an abstract only. This is important since a great deal of the data reported to be from this study are marked confidential in the submission and cannot be verified with reference to a published paper.

Please explain the differences, if any, between the populations receiving the different doses of *** (Table 12).

Please clarify how the **** risk reduction values used in the model were derived from the *** trial. It does not seem as though they were drawn from the published paper.

Please provide a list of the 18 RCTs (section 5.2.1, page 17) which were excluded because the population was not aligned with the EU/UK licence; with the specific reason (s) for exclusion for each one.

The values of n change for the different analyses due to withdrawals, crossovers and pooling of different studies (as highlighted in Table 1). Not only are the reasons for these differences complex there are occasions where it is not possible to determine why certain patients were included and excluded e.g. the long term efficacy following repeated treatments.

As can be seen from these examples, the lack of information or inconsistency of the provided data would limit the ability of the ERG to move forward with assessment of the MS.

In addition to these general queries, there were three STA clarification letters that included specific questions requiring clinical expertise and related to issues of patient pathway/response and drug dosage:

Please clarify if a switch from *** to *** is expected to happen in clinical practice and provide a rationale for why a sequential approach of prescribing *** to all patients and then *** to patients who do not respond was not considered.

The diagnostic criteria for *** have not been made clear in the submission. Clear diagnostic criteria are required to identify the patient population and give an indication of incidence – is this available from the manufacturer?

Based on the figures presented in the ‘validation’ sheet of the submitted model it can be seen that the patients in the comparator arm in the USA trials have a better prognosis than the patients in the comparator arm from the *** study (84.8% in the joint US analysis compared with 77.4% from the *** trial). A more detailed discussion of this issue and why these differences exist would be most useful to the Committee in its discussions.

Please clarify whether there is evidence for no dose adjustment in the elderly? Reference 9 appears to be based on clinical opinion rather than trial evidence.

Please clarify in a clinical context why within the model the dosing of *** is allowed to be altered (letting patients move from 80 mg to 120 mg) but the dose of *** remains constant.

Additional clinical data

In addition to requests for additional data, there were indications from the clarification points that there were data that the ERG might have expected to see in the submission but which were omitted from the MS (e.g. selective reporting):

In the *** trial, the most common adverse event leading to withdrawal was abnormal liver-function test results which accounted for withdrawal of 5 patients … In addition, more patients receiving *** (80 or 120 mg/d) discontinued the study because of rashes. It is not clear, why this has not been reported in the STA submission.

Only data for the last 4 weeks of the *** and *** trials have been reported (Figures 5–6 do not report n/N at each time point). It is noteworthy, that in the *** trial, nearly 70% of patients in the *** groups experienced *** flares after discontinuing prophylaxis i.e. between week 9 and 52 (no data reported for *** trial from week 1 to 24).

Please provide further detail of discontinuation rates for the *** trial? It is unclear why this was not reported in the submission. Please provide a table with baseline characteristics for the trials other than ***. Section 2.3.2 only provides baseline characteristics for ***. Although it is appreciated that only limited information may be available for ***, NEJM [New England Journal of Medicine] articles are available for ***, *** and ***.

Present response results (as in Table 5.6.3 and including change in *** scores) for the 12 week trials excluded from the cost-effectiveness study reported in Table 5.6.2.

Please provide the response data that are entered into the model, including the adjustment for the differing skin involvement in the patient populations of the trials.

Please provide appropriate data in table format; section 5.4.5 (secondary endpoints: proportion of patients requiring treatment for ****) has been poorly reported with important omissions. It appears that limited/selective data have been reported, particularly on the number of patients that experience **** from day 1 to end of study.

Selective reporting has been recognised as an important issue in relation to clinical trial reports and the MSs demonstrated that the ERGs had identified instances of such reporting. 57

Additional clinical analysis

These clarification points represent examples of requests for additional clinical analysis. As can be seen, a number of these requests would require a considerable amount of additional analysis to be carried out by the manufacturer:

Please provide the additional analyses of clinical trial data relating to the table of assumptions (on page 109) about scheduling of response rates.

Please provide working Excel spreadsheet which describes how investigator-assessed response rates were pooled for use in the SA reported in table. In particular, please provide a copy of time-to-event analyses for overall survival and time-to-disease progression in trials.

Please provide a full indirect comparison, including methodology, between *** and ***/***. If this is not available, please provide pairwise comparisons (including associated Confidence Intervals) for: weight loss; waist circumference; change from baseline in BMI; and the proportion with 5% and 10% weight loss.

Could you provide a meta-analysis of the safety and efficacy evidence (as noted in guidance section 5.5) using individual patient level data or aggregate data for all outcomes of interest and not just significant results?

Both fixed effect and random effects model based results are required for all analyses. [Requests of this kind were seen in five letters.]

Please undertake a full synthesis of PsARC response and other outcome parameters at 12 weeks for ***, *** and *** using data from all available studies (including both 12 and 24 week studies).

Please provide a subgroup analysis of the co-primary results.

Please provide statistical evaluations of heterogeneity for the studies from which absolute efficacy estimates were pooled.

Please provide either forest plots, or the results of the tests for heterogeneity, for all the pooled results in Table 20.

Analysis decisions/rationale

The majority of issues in this category related to requests for information explaining how analyses were carried out or why various decisions were taken in relation to the analysis that was presented in the submission:

Please provide additional clarification on the approach used to estimate separate treatment effects for each subgroup (main subgroups and those considered in the sensitivity analyses). In particular, clarify whether the data was from the trials was simply split into the different subgroups or whether formal statistical interactions were estimated and employed.

Please provide a rationale for presenting the pooled analysis of individual patient data from the *** and *** trial and discuss the methodological limitations of doing so.

Expand description of the mixed treatment comparison (MTC). Specify which studies have been included and explain process for estimation of adjusted indirect comparison.

Please explain the process of ‘estimation by adjusted indirect comparison’ used to generate RRs for *** and extended *** versus nil in the single intervention meta-analyses, and why no adjustment was possible for ***.

Please provide reasons why you have not considered using a ‘multiple treatment comparison’ approach to answering all the comparisons presented in your decision problem.

Please provide a more descriptive explanation of the method of indirect comparison used to compare *** with *** and provide a critique of the pros and cons of this approach.

Please provide more detail about how and why the adverse event data for *** was pooled (page 89).

Please provide further rationale for the use of last observation carried forward to account for missing data and provide an alternative analysis of the clinical data using a best case/worst case scenario.

Please provide a rationale for not undertaking any subgroup analyses when in this disease’s context specific subgroups may imply different disease management (e.g. patients with renal impairment) and eventually require different clinical endpoints (e.g. patients with **** requiring a quicker reduction in sUA levels) from the overall population.

Provide pooled analyses for primary endpoints and safety data from studies, or an explanation for not doing so. Include analysis of bleeding outcomes. Full description of methodology should be provided.

Currently, pooled analyses have only been performed on the secondary efficacy endpoint and no explanation has been given for not performing this analysis on the primary efficacy endpoint (or safety endpoints).

Please clarify which analyses used IPD data, whether data relating to subgroups was obtained from different trials, and which trials contributed to which outcomes.

As can be seen the requests range across a number of issues including, subgroup analysis, methods for indirect and MT comparisons, endpoints and outcomes.

Clarification/explanations

As would be expected, this is the largest category and spans all aspects of the submissions. In terms of clarification of calculations, the queries ranged from requests for simple explanations of the sources of data to more complex explanations that were related to estimates used in the model:

Please provide a step by step explanation of how you got from the initial figure (–0.15) to a treatment effect of –7.32% and why you did not consider the adjusted figure in the same document (–0.21) appropriate for use.

Please clarify the methodology for obtaining the CEACs. Traditionally this would not have more than 3 dp as only 1,000 simulations were provided, whereas the figure reported for being less than £20k at 24 months is 0.6234.

Please clarify whether cause-elimination approaches were used to estimate these mortality estimates.

Please explain whether the following analysis does not constitute double counting? Cell J54 on the ‘Outputs sheet’ already contains the first 5 years of utility. In cell L31 lifetime and 0–5 years are added together.

Other queries highlighted inconsistencies between the clinical and economic sections of the submission as well as inconsistencies within the economic sections:

Why are the inclusion criteria, namely the duration of study, different between the clinical (no restriction) and cost (minimum 12 weeks) section?

The odds ratio for death at 6 months following *** event is taken from a meta analysis that includes the *** 1993 study. This study was excluded in the clinical effectiveness section. Please explain why this study has been included in the economics.

Please provide clarification on Table 6.4 in the context of the model. The mean data for *** is stated in the table as 0.2016, but is in the model as 0.2108. The distribution from the raw data is also unlikely to be normal. Clarify that the *** data is also correct within the model.

Clarification issues also included queries related to the sources of data and their use within the economic analysis:

Please provide clarification on the data and sources for these assumptions and how these compare with epidemiological studies of *** levels in patients treated for ***.

Please confirm that survival analyses for 2-arm model use only data from time of second randomisation. Was the same truncation point (1500 days from first randomisation) used for these analyses as for the 4-arm model?

Please explain further how you used the above information to ‘identify the most likely distribution of responses’ for the rest of treatment options, and how you sample the type of response for the next 5 timepoint from the joint distribution of ***, *** and ***.

Justify combining of minor bleed and clinically relevant bleed in the economic evaluation.

Please justify the selection of an initial 6-month cycle when the majority of trial outcomes data relates to 90 days results.

In addition, and similar to the clinical section, there were queries related to why explicit clinical and modelling decisions were taken:

Please provide further rationale for the use of your choice of comparator considering this argument.

Please provide justification for excluding strategies in which the drug or comparator drug is discontinued if a specific target is not achieved.

Please explain why your indirect comparison data is not used in the model.

Please provide a rationale for the inclusion or exclusion of variables used in the PSA.

Please clarify the clinical justifications of transition probabilities from the ‘Flare d/t resistance’ health state to other states (Response, CHB Resist Salvage Tx, CHB no TX, etc.).

Data/methods

Data queries were broad and, as can be seen from the examples, the majority of the requests were critical to the ERG as they attempted to appraise the submission:

Please provide the absolute event numbers on which these transition probabilities are based and the details of the methodology used to obtain the probabilities.

Please make clear where in the model (which cells) were used to adjust the utility for surgery in the first year.

Please define what counts as an event (e.g. first or subsequent? Local, regional, distant recurrence, contralateral?) for each clinical outcome used in the model, including whether death without breast cancer recurrence, or the diagnosis of a non-related cancer counts as an event.

Please provide further clarification of the meaning of the adverse event rates used in the model (for example does the adverse event data refer to the number of events, or the number of patients for whom any event occurred at any time).

There is no information in the submission concerning the timing of, and reason for, withdrawals in the two arms of the trial. The ERG request a detailed table of withdrawals by week and by reason (serious AE, reaction, clinical advice, patient request, lack of efficacy, etc.).

Further analysis

The queries raised in this section highlight some of the additional analyses that were requested by the ERGs. The most commonly requested additional analysis related to SA around various parameters. One example requested four such analyses, considering cost, outcomes, time horizon and confidence intervals, and demonstrates that important analyses were missing from the original submission:

Please provide additional analysis including:

1. increased drug cost including cost of administration to £25,000 and £30,000

2. improving/reducing the survival estimates in the comparator arm

3. sensitivity analysis around extrapolation of benefits of *** beyond the trial period, by setting the relative risk of *** equal to 1 after (a) 10 years (b) 5 years

4. widening the confidence intervals around RFS, DDFS and RFM curves over time to reflect uncertainty in these measures.

Other examples include:

Re-run the base case analysis based on a complete synthesis of all 12-week *** data for all comparators (based on all trials of at least 12 weeks duration).

Re-run the base case analysis without the ‘conventional ***s’ drug cost.

Could you provide SA on the extension of the first phase of the model up to 6 months, noting that *** Guidelines state that initial prophylactic treatment can go up to 6 months.

Please provide a SA on the effect of vial sharing on the cost-effectiveness.

Specific requests for cost-effectiveness analysis indicate that there may have been cases when basic information/analysis was missing from the original submission, as indicated by the requested information in the following examples from a single clarification letter:

Please present summary cost-effectiveness results for the base-case populations and the populations considered in the SA.

We request that the cost-effectiveness results are also presented based on this re-run synthesis. We would be grateful if ***would replicate Tables B1.2, B1.3, B1.4, B1.5 and C4.4 for this analysis.

There are examples where the ERG requested a different modelling/analysis approach:

Please provide a simultaneous comparison of *** (lifetime treatment duration), *** (lifetime treatment duration), *** (1-year treatment duration) and diet/exercise as opposed to separate pairwise comparisons.

Please assess the use of relative risks compared to odds ratios on the central estimates and SA results.

Provide incremental cost-effectiveness ratios (ICERs) using both random effects and fixed effect results.

A mean of 307 days is quoted in the submission for the time between doses of ***. This is used to estimate the cost per treatment. The ERG request a detailed table AND Kaplan Meier analysis of the time to second treatment in the intervention arm of the trial, censoring all treatment terminations. We also request the provision of total numbers of first and second treatments given in the intervention arm of the trial.

Please perform a full PSA across a wider range of parameters, including as a minimum all of those which are varied in the one- and multi-way sensitivity analyses. Please state the number of iterations performed.

In addition, present the associated cost-effectiveness acceptability curves and report the probabilities that each intervention is cost-effective at a threshold of £20,000, £30,000 and £40,000 per QALY for the base-case populations.

Other clarification letters requested analyses to be carried out related to different subgroups of patients:

Please present summary cost-effectiveness results for the base-case populations and the populations considered in the SA.

Please consider subgroup analyses for patients in different OTT intervals (e.g. 0–90 and 91–180 minutes).

Please provide separate incremental cost-effectiveness ratios for patients with recurrent cancer and for those patients with metastatic cancer

There are other interesting queries that raise the question of what data were used in the original submission to populate the economic model:

Please provide additional cost-effectiveness results showing SA with utilities derived from the clinical trial data.

Searching

• Re-running the manufacturers’ searches may be considered but as a minimum it is important to look at the numbers of identified studies in the MS. Depending on circumstances there may be time to critique and correct the searches if necessary. Comments should be made on the adequacy of the search strategy and database selection.

• Searches for ongoing studies and background information may be considered.

• There may be specific tasks that arise, for example finding a study hinted at in the MS, finding utility information, key parameter values or other economic evaluations and checking consistency in different sections of the MS related to the searching.

Critical appraisal tools

• The use of critical appraisal tools for assessing the quality of the manufacturer’s systematic review and model is considered to be important, as this increases transparency and provides a framework or starting point for appraising the MS. However, the use of a critical appraisal tool alone is not sufficient and teams should not be constrained by them.

• Critical appraisal tools are needed to assess the manufacturers’ included studies.

• There is a need to focus on those studies that are driving the results. Levels of critical appraisal may depend on their importance in the decision. This is not always clear and can change.

• The use of critical appraisal tools is potentially of more use to more inexperienced ERG members.

For the website

• When categorising topics it may be appropriate to double list them in more than one category but this needs to be consistent to avoid confusion.

• Documents should be recorded with exact dates (day/month/year).

• The current search engine on the NICE website does not provide enough detail to find all relevant information. There needs to be a mapping tool for site visitors to tell them where to find documents for ongoing and completed projects. It is easy to find guidance but there are limited markers for the process documents (scopes, ACDs, ERG reports, etc.).

• As noted in other parts of this report, documentation for the STAs should be consistently placed on the NICE website.

• Unique identifiers for each topic on the website would make differentiation between topics easier, especially when a single intervention may be appraised for a number of different conditions.

• Standardising the referral date to final scope time period could be considered or making this process more transparent on the NICE website so that it is clear what activity is taking place during this time period.

General points

• There needs to be consideration of a requirement for a comprehensive review to identify studies to inform the conduct of indirect comparisons or MTCs and other additional aspects of the modelling exercise (e.g. utility data, cost data, etc.).

• Procedures to reduce the number of suspended, delayed and terminated STAs need to be considered.

1.1 Scope of the submission

1.2 Summary of submitted clinical effectiveness evidence

1.3 Summary of submitted cost-effectiveness evidence

1.4 Commentary on the robustness of submitted evidence

1.5 Key issues

For example, comment if there are major concerns with the comparators which have been identified in the submission.

2.1 Critique of manufacturer’s description of underlying health problem

For example, if the appraisal is about metastatic hormone-refractory prostate cancer, and the submission gives details predominantly about prostate cancer in general, are those details relevant to the appraisal?

2.2 Critique of manufacturer’s overview of current service provision

For example, does the submission concord with opinions of clinical and patient experts? Has sufficient backing evidence been given about how often the comparators and intervention are used? Are the constraints of UK market authorisations considered?

3.1 Population

Description of relevant patient population(s) and comment on whether they are appropriately defined in the submission.

3.2 Intervention

What is the technology and what is its relevant or proposed marketing authorisation/CE mark?

3.3 Comparators

Relevant comparators in an NHS context, justification for choice of comparators. For example, where hard evidence is not available, has the manufacturer asked an unbiased clinical panel, or done its own survey, and does it agree with what the clinical experts for the appraisal say?

3.4 Outcomes

Including clinical effectiveness, adverse events, QoL and health economic outcomes and a discussion of appropriate mechanisms for measuring these outcomes? Critique of whether focus of submission is on the appropriate outcomes. Comment on whether the analysis has been limited to non-ideal outcomes.

3.5 Time frame

3.6 Other relevant factors

4.1 Critique of manufacturer’s approach

4.2 Summary of submitted evidence

5.1 Overview of manufacturer’s economic evaluation

Including one-page summary of structure, assumptions and sources, with signposting to tables with numerical inputs and their distributions where appropriate.

5.2 Critique of approach used

5.3 Results included in manufacturer’s submission

Including one-page summary of sensitivity analyses and resulting range of ICERs: one-way, threshold and probabilistic.

5.4 Comment on validity of results presented with reference to methodology used

5.5 Summary of uncertainties and issues

7.1 Summary of clinical effectiveness issues

7.2 Summary of cost-effectiveness issues

7.3 Implications for research

1.1 Scope of the manufacturer submission

1.2 Summary of clinical effectiveness evidence submitted by the manufacturer

1.3 Summary of the ERG’s critique of clinical effectiveness evidence submitted

1.4 Summary of cost-effectiveness evidence submitted by the manufacturer

1.5 Summary of the ERG’s critique of cost-effectiveness evidence submitted

1.6 ERG commentary on the robustness of evidence submitted by the manufacturer

1.7 Summary of additional work undertaken by the ERG

2.1 Critique of manufacturer’s description of underlying health problem

Does the ERG believe that the manufacturer’s description of the underlying health problem is appropriate and relevant to the decision problem under consideration?

2.2 Critique of manufacturer’s overview of current service provision

Does the ERG believe that the manufacturer’s overview of current service provision is appropriate and relevant to the decision problem under consideration?

3.1 Population

To what extent does the clinical evidence submitted by the manufacturer match the patient population described in the final scope? Where there is a mismatch, provide further details. Does the clinical evidence submitted by the manufacturer reflect the characteristics of the patient population in England and Wales eligible for treatment? If not, provide further comment.

3.2 Intervention

Does the intervention described in the MS match the intervention described in the final scope? What is the technology and what is its relevant or proposed marketing authorisation/CE mark?

3.3 Comparators

Do the comparators described in the MS match the comparators described in the final scope? If not, provide further details. Where evidence is limited or not available for relevant comparators has the manufacturer asked an unbiased clinical panel, or carried out its own survey, and do the views elicited agree with what the clinical advisors to the ERG advocate?

3.4 Outcomes

Do the outcomes in the MS match the outcomes described in the final scope? If not, provide further details. Consider clinical effectiveness, adverse events, QoL and health economic outcomes and a discussion of appropriate mechanisms for measuring these outcomes. Is the focus of the submission on appropriate outcomes or has it been limited to non-ideal outcomes?

3.5 Other relevant factors

For example: Does the MS include a section on equity considerations? Is there an ongoing PAS application?

4.1 Critique of the methods used by the manufacturer to systematically review clinical effectiveness evidence

4.2 Summary and critique of submitted clinical effectiveness evidence

If there is more than one RCT described in the MS, it may be appropriate to discuss each trial individually using the headings described.

4.3 Conclusions

Describe the completeness of the MS with regard to relevant clinical studies and relevant data within those studies. Does the submission contain an unbiased estimate of the technology’s (relative and absolute) treatment effects in relation to relevant populations, interventions, comparators and outcomes? Are there any remaining uncertainties about the reliability of the clinical effectiveness evidence? Reference should also be made concerning the extent to which the submitted evidence reflects the decision problem defined in the final scope.

5.1 ERG comment on manufacturer’s review of cost-effectiveness evidence

5.2 Summary and critique of manufacturer’s submitted economic evaluation by the ERG

Summarise and critique the cost-effectiveness evidence submitted by the manufacturer (headings 5.2.1–5.2.11 are suggested headings). It is noted that the ERGs may prefer not to combine the summary and critique of the submitted economic evidence and instead report summary and critique sections separately.

5.3 Additional work undertaken by the ERG

Provide details of any additional work conducted by the ERG in relation to cost-effectiveness. If the results of any of the additional work affect the size of the ICER, refer the reader to the summary table in Section 6.

5.4 Conclusions

Describe the completeness of the MS with regard to relevant cost-effectiveness studies and data described in any de novo economic evaluations. Does the submission contain an unbiased estimate of the technology’s ICERs in relation to relevant populations, interventions comparators and outcomes? Are there any remaining uncertainties about the reliability of the cost-effectiveness evidence? Reference should also be made concerning the extent to which the submitted evidence reflects the decision problem defined in the final scope.

8.1 Implications for research