Informing a decision framework for when NICE should recommend the use of health technologies only in the context of an appropriately designed programme of evidence development

Traditional published literature

A traditional systematic search was undertaken in MEDLINE, MEDLINE In-Process & Other Non-Indexed Citations, EMBASE, The Cochrane Library, EconLit and Cumulative Index to Nursing and Allied Health Literature (CINAHL) based on appropriate search terms informed by key publications to identify relevant literature. 8 An iterative approach was used in the development of search terms and strategies incorporating ‘pearl-growing’ techniques through additional citation and reference searching of key publications and discussions with our Advisory Group. Searches were restricted to documents produced from 1999 onwards as the use of OIR and AWR policies is a relatively recent process and so it is not expected that any relevant references will be identified before this date.

To test the performance of the search strategy, records initially identified from the case for support and citations contained within those documents and articles were cross-referenced with the results identified from the searches (capture–recapture method). In doing this it was concluded that the final search strategy identified all of the records it would be likely to identify, that is, those accessible from traditional (non-grey literature) sources.

Grey literature searches

Additional searches of grey literature were also undertaken using a similar approach reported in the review by Stafinski et al. 8 The following grey literature databases were searched: the New York Academy of Medicine Grey Literature Report and the IDEAS database (Department of Economics, University of Connecticut). A similar set of key words as for the review of the traditional published literature was used for the grey literature searches, with some modifications because of the different recording mechanisms operated by the grey literature databases. Searches were again restricted to documents produced from 1999 onwards.

Policy and discussion documents

The third element of the search was based on policy and discussion documents. Although the focus of the search related to UK policy and discussion documents, documents from key international organisations were also included. Relevant international organisations and websites were identified in conjunction with our Advisory Group. A number of policy documents were also identified from those referenced in the case for support and from the reference lists of key papers. 9–12 A number of additional organisation websites were searched for relevant documents. Where possible, web searches were restricted to documents produced from 1999 onwards for consistency with the other search elements.

Other sources

The final element of the search considered a variety of other sources including relevant interest groups [e.g. the Health Technology Assessment international (HTAi) interest subgroup on conditional coverage and evidence development for promising technologies and the Programs for Assessment of Technology in Health (PATH) website containing publications relating to the programme of conditionally funded field evaluations (CFEEs) in Ontario], other reviews,8 references suggested by our Advisory Group and additional searches based on citations of papers identified from earlier stages.

Identified references

In total, 59 references were included in the review; 43 of these were journal articles,2–5,8,13–50 11 were policy documents (eight UK and three non-UK)9–12,22,51–56 and five were based on presentation slides or discussion documents. 56–60

Of the 43 journal articles, the majority are academic ‘think pieces’ on issues relating to OIR and AWR with additional responses to previous articles. 5,13–16 Although a number of the journal articles provide a discussion of many of the general issues relating to the use of OIR and AWR,2,3,17–24 a smaller proportion explicitly address the issue of terminology and/or provide a taxonomy of OIR/AWR types (n = 11). The majority of the journal articles discuss specific issues relating to OIR/AWR implementation, most commonly in relation to evidence collection and design (n = 41). These issues are often illustrated in the context of specific case study applications such as the positron emission tomography register or the National Emphysema Treatment Trial (NETT). A relatively large number of these articles also discuss ethical issues and social value judgements relating to the implementation of OIR/AWR policies (n = 20) and issues of investment and reversal costs (n = 21).

The 11 policy documents91–112,22,51–56 identified are from UK and non-UK sources. From the UK, relevant documents are produced by HM Treasury,9 the Department of Health,52 the House of Commons Health Committee,10 NICE and the Agency for Healthcare Research and Quality (AHRQ),51 NICE Citizens Council,11 NHS Scotland,53 the Office of Fair Trading (OFT)12 and the Office of Life Sciences. 54 Outside of the UK, relevant documents identified are all produced by US organisations: the Centers for Medicare & Medicaid Services (CMS),55 the Health Industry Forum56 and the National Health Policy Forum. 22 In many of these policy documents OIR and AWR are typically discussed as part of a wider general health policy theme as opposed to being the central issue under consideration. Only a small number22,51,55 offer any insights into terminology and/or taxonomies relating to OIR/AWR. Over half (n = 7) present general issues of OIR/AWR and specific issues most commonly related to evidence collection. Issues of investment and reversal costs, changing prices and ethical issues/social value judgements are covered to a lesser extent in the policy documents.

The final five references included are presentation slides or discussion documents. 56–60 These are all widely cited in the OIR and AWR literature. None of these references discusses terminology or taxonomies of OIR/AWR types. Instead they again focus on general themes of OIR/AWR, with some discussing specific issues such as evidence collection.

Summary of the key issues for practice and policies for ‘only in research’ and ‘approval with research’ schemes

The following sections discuss the main findings in line with the key objectives of the review.

Technologies without significant irrecoverable costs

Some element of cost that once committed by approval cannot be subsequently recovered is almost always present. However, the significance of these types of costs depends on their scale relative to expected population NHEs associated with the technology as well as the nature of subsequent events (see Technologies with significant irrecoverable costs and Chapter 5, Point 2: Are there significant irrecoverable costs?). 75 In this section we consider the relatively simple sequence of assessments and decisions which lead to guidance for those technologies that are not judged to have ‘significant’ irrecoverable costs associated with approval.

Technologies expected to be cost-effective

The sequence of assessments and decisions, which ultimately leads to guidance, starts with cost-effectiveness and the expected impact on population NHEs, that is, where existing NICE appraisal currently ends. This is an assessment of expected cost-effectiveness, that is, ‘on average’, based on the balance of the evidence and analyses currently available. It will include an assessment of effectiveness and potential for harms as well as NHS costs (see NICE’s Guide to Methods of Technology Appraisal1). Any assessment may be very uncertain with the scale and consequences of uncertainty assessed subsequently when considering the need for additional evidence. The sequence of assessments and decisions (judgements required) is illustrated in Figure 1. This demonstrates that an assessment of cost-effectiveness is only a first step and does not itself inevitably lead to a particular category of guidance. For example, a technology that might on balance be expected to be cost-effective might nevertheless receive OIR guidance if the additional evidence that is needed cannot be acquired if the technology is approved.

FIGURE 1.

Algorithm for technologies expected to be cost-effective.

Need for evidence

Some initial assessment of the need for further evidence and a decision about whether or not further research might be potentially worthwhile is important because a ‘no’ at this point can avoid further and complex assessments, for example a technology offering substantial and well-evidenced health benefits at modest additional cost is likely to exhibit little uncertainty about whether or not the expected population NHEs are positive. In these circumstances, further research may not even be potentially worthwhile (i.e. the opportunity costs of conducting this research exceed its potential value) and so guidance to approve could be issued on the basis of existing evidence and at the current price of the technology (e.g. ‘Approve⁴’ in Figure 1). If additional evidence is needed and further research might be worthwhile, then further assessments and decisions are required before guidance can be issued. Critically, some assessment is required of the type of evidence that is needed and whether or not the type of research required to provide it is likely to be conducted if approval is granted. 29

Research is possible with approval

If research is possible with approval, some further assessment of the long-term benefits of research is required, including (1) the likelihood that the type of research needed will be commissioned by research funders or conducted by manufacturers, (2) how long until such research will be commissioned, recruit and successfully report and (3) how much of the uncertainty might be resolved by the type of research that is likely to be undertaken. 70 An assessment of other sources of uncertainty that will resolve only over time is also needed, for example changes in prices or the launch of new technologies. 71 These sources of uncertainty will influence the future benefits of research that could be undertaken as part of AWR. For example, even if the current benefits of research, which might be very likely to be undertaken, are considerable, if the price of the technology is likely to fall significantly before or shortly after the research reports, or if future innovation makes the current technology obsolete (or more effective), then the future benefits, once the research reports, might be very limited. In these circumstances it might be better to approve (rather than use a policy of AWR) and reconsider whether and what type of research is needed at a later date once these uncertainties have resolved. The judgement of whether the long-term benefits of research are likely to exceed its expected costs determines guidance, with ‘AWR¹’ and ‘Approve¹’ in Figure 1 dependent on ‘yes’ and ‘no’, respectively.

Research is not possible with approval

The type of research needed, for example RCTs, may not be possible once a technology is approved for widespread NHS use (because of ethical concerns, recruitment problems and limited incentives for manufacturers). In these circumstances the expected benefits of approval for current patients must be balanced against the benefits to future patients of withholding approval to allow the research to be conducted. Initially, the same assessment of the long-term value of the type of research that might be conducted if approval is withheld is still required. Similarly, the impact of other sources of uncertainty on the longer-term benefits of research is also needed. If the benefits of research are judged to be less than the costs (i.e. research is not worthwhile anyway), the technology can be approved based on current evidence and prices (‘Approve³’ in Figure 1). However, judging that research is worthwhile at this point is not sufficient for OIR guidance. In addition, an assessment of whether the benefits of early approval (expected population net benefits for current patients) are greater than the opportunity costs (the net benefit of the evidence likely to be forgone for future patients as a consequence of approval) is required. If the expected benefits of early approval are judged to be less than the opportunity costs then OIR guidance would be appropriate (‘OIR¹’ in Figure 1). Alternatively, if the expected benefits of early access for current patients are judged to be greater than the opportunity costs for future patients then approval would be appropriate (‘Approve²’ in Figure 1). All of these assessments, including the benefits of early approval and the value of evidence, will change if the effective price of the technology is reduced (see Changes in effective prices).

Technologies not expected to be cost-effective

A technology that is not expected to be cost-effective will, on balance, impose negative population NHEs if it is approved. These negative NHEs can arise because the technology may not be effective, the potential for harm exceeds any benefits and/or the additional NHS costs are not justified by the magnitude of the expected health benefits offered. In these circumstances approval can be ruled out, but which of the other categories of guidance might be appropriate will depend on subsequent assessments and decisions (Figure 2).

FIGURE 2.

Algorithm for technologies not expected to be cost-effective.

Need for evidence

Any assessment will be uncertain, so it remains possible that a technology that is not expected to be cost-effective on the balance of existing evidence might offer positive NHEs. Therefore, the scale and consequences of this uncertainty must be considered to make an initial assessment of the need for additional evidence and whether additional research might, in principle, be worthwhile. If it is not then the technology can be rejected based on existing evidence and its current price (‘Reject⁴’ in Figure 2). Alternatively, if further research might be worthwhile then an additional assessment is required of whether the type of evidence and research that is needed can be conducted without approval.

Research is possible without approval

Generally, most types of research (including RCTs) will be possible without approval. Further assessment of the longer-term benefits of the type of research that is likely to be conducted, and when it might report, is required, including the impact of other sources of uncertainty that will resolve over time. If, following this reassessment, the expected benefits of research are judged to exceed the associated costs then OIR would be appropriate (‘OIR²’ in Figure 2). Alternatively, if the costs of research are likely to exceed the longer-term expected benefits then the technology should be rejected at this point (‘Reject¹’ in Figure 2).

Research is not possible without approval

In some circumstances it is possible that certain types of evidence might be acquired only, or be more easily acquired (more quickly and at lower cost), once a technology is in widespread use, for example linking surrogates (specific to the technology) to longer-term health outcomes, longer-term and/or rare adverse events, greater understanding of learning and incremental improvements in the use of a technology, or identifying the particular types of patients that might benefit most. 76 In this less common situation, in which the type of research needed is not possible (or is significantly more costly) without approval, the same assessment of the longer-term benefits of research is required. If further research is judged not to be worthwhile following this reassessment, the technology can be rejected (‘Reject²’ in Figure 2). Alternatively, if research is judged worthwhile, an additional assessment of whether the benefits of approval exceed the costs is required. In this case, approval of a cost-ineffective technology would make the research possible, but will impose opportunity costs (negative expected population NHEs). The key question is whether the net benefits of the research exceed these opportunity costs. If they do not then the technology should be rejected even though research, had it been possible without approval, would have been worthwhile (‘Reject³’ in Figure 2). Alternatively, if the net benefits of research more than offset the opportunity costs then AWR would be appropriate even though the technology is expected to be cost-ineffective (‘AWR²’ in Figure 2).

Therefore, AWR guidance for technologies not expected to be cost-effective is certainly possible but is appropriate only in certain circumstances: (1) the type of research needed is not possible without approval, (2) the long-term benefits of the research are likely to exceed the expected costs and (3) the additional net benefits of such research exceed the opportunity costs of approving a cost-ineffective technology. More commonly, research might be possible but more costly without approval. In these circumstances, AWR guidance could be considered only if the additional costs of research without approval exceed the opportunity costs of approving a cost-ineffective technology.

Technologies with significant irrecoverable costs

Irrecoverable costs are those that, once committed, cannot be recovered should guidance be revised at a later date. In most NICE appraisals these are included in the expected (per patient) cost of a technology. However, rarely is their potential additional impact explored when future events, such as research reporting or other sources of uncertainty resolving, might mean that guidance will be revised in the near or distant future. 6,72 These types of cost are commonly thought of as capital expenditure on equipment or facilities which have a life expectancy that extends beyond the current patient population. They might also include the resources required to implement guidance or to train staff to use a new health technology or a period of ‘learning’ during which NHEs are lower. Although these costs are incurred up front, they tend to be included in NICE assessments as if they are paid per patient treated over the lifetime of the equipment or facility. This common assumption will have no effect as long as guidance is certain not to change during this period. However, if it is possible that initial approval might be withdrawn at some point, then, although future patients will no longer use the technology, these upfront costs cannot be recovered (see Issues specific to ‘only in research’ recommendations). Therefore, in these circumstances it would be inappropriate to include these costs as if they were paid per patient treated (see Point 2: Are there significant irrecoverable costs?). The possibility that approve or AWR might be reconsidered after research reports, for example, and the impact that this would have on expected costs need to be considered before committing these types of capital costs, that is, it may be better to withhold approval and avoid commitment of resources until the uncertainty is resolved.

However, irrecoverable costs may be much more common. Even in the absence of capital investment in equipment and facilities, most new technologies offer a ‘risky investment profile’ for each patient treated. Generally they impose initial per patient treatment costs that exceed the immediate health benefits (see Point 1: Is it expected to be cost-effective?). These irrecoverable treatment costs are offset only by cost savings and health benefits in the longer run, that is, initially negative NHEs (losses) are only gradually compensated by later positive ones (gains). Therefore, a technology expected to be cost-effective may be expected to break-even, that is, when accumulated ‘gains’ compensate earlier ‘losses’, after some considerable time. If guidance is likely to change it is possible that initial losses will not be compensated by later gains and the expected additional NHEs will not be realised. 75 This type of investment profile becomes significant (has some influence on a decision to approve) if the decision to treat a presenting patient can be delayed until uncertainty is resolved (e.g. research reports or other events occur) because the commitment of irrecoverable opportunity costs (negative NHEs) can be avoided (see Point 2: Are there significant irrecoverable costs?). In these circumstances, OIR or reject avoids this commitment and preserves the option to approve the technology at a later date when its purchase by the NHS represents a ‘less risky investment’. 75,b

Although aspects of irrecoverable cost are almost always present, their potential significance also depends on their scale relative to expected population NHEs of the technology. Critically, their impact depends on the chance that guidance will be revised in the near or distant future as a result of new evidence becoming available or changes in prices and technologies. The full algorithm becomes more complex (see Figures 27 and 28 in Appendix 2) so here we focus on the key differences from the section on technologies without significant irrecoverable costs.

Different types of guidance

Each sequence of assessment and decision leads to different categories and ‘types’ of guidance for technologies with differing characteristics, indications and target populations. The different ‘types’ of guidance illustrate how similar guidance might be arrived at in different ways, helping to identify the particular combinations of considerations that might underpin guidance, contributing to the transparency of the appraisal process. The possible categories and types of guidance are summarised in Table 1; the numbers in the body of the table refer to the numbered guidance in Figures 1 and 2 and the algorithm in Appendix 2 (see Figures 26–28).

The categories of guidance available to NICE have wider application than is reflected in previous guidance (see Chapter 4). For example, there are five different types of OIR that may be appropriate when a technology is expected to be cost-effective. Indeed, OIR may be appropriate even when research is possible with approval if there are significant irrecoverable costs. AWR can be considered only when research is possible with approval, but reject remains a possibility even for a cost-effective technology if there are irrecoverable costs. Therefore, the full range of categories of guidance (OIR and reject as well as AWR and approve) ought to be considered for technologies that, on the balance of existing evidence and current prices, are expected to be cost-effective.

It is only approval that can be ruled out if a technology is not expected to be cost-effective, that is, cost-effectiveness is necessary but not sufficient for approval but lack of cost-effectiveness is neither necessary nor sufficient for rejection. Although likely to be uncommon, there are circumstances when AWR may be appropriate even when a technology is not expected to be cost-effective (see Technologies without significant irrecoverable costs and Technologies expected to be cost-effective). More commonly the choice of appropriate guidance will be either reject or OIR. Importantly, which category of guidance will be appropriate depends only partly on an assessment of expected cost-effectiveness and hence this assessment should be regarded only as an initial step in formulating guidance. Guidance will depend on a number of other key assessments including (1) the need for evidence, (2) whether or not the type of research required is possible with approval, (3) the expected longer-term benefits and costs of the type of research likely to be conducted, (4) the impact of other sources of uncertainty that will resolve over time and (5) the significance of any irrecoverable costs.

Changes in effective prices

Any change in the effective price of a technology, either through patient access schemes (which offer some form of discount that reduces NHS costs) or direct price changes (possibly negotiated though a future value-based pricing scheme), will affect key assessments and decisions, leading to different ‘paths’ through the algorithm, consequently changing the category of guidance that would be appropriate (Simon Walker, Centre for Health Economics, University of York, UK, July 2011, personal communication). 4,23 For example, provisional OIR guidance for a technology that is expected to be cost-effective might be revised to approve with a sufficient price reduction because the benefits of early approval will be greater and uncertainty about its cost-effectiveness and therefore the value of additional evidence will tend to be lower (e.g. from ‘OIR¹’ to ‘Approve²’ in Figure 1). c Similarly, AWR might be revised to approve if the benefits of early approval now exceed the value of additional evidence (e.g. from ‘AWR¹’ to ‘Approve²’ in Figure 1). d

Equally, provisional guidance to reject a technology that is not expected to be cost-effective might be revised to OIR if the reduction in price is not sufficient to make it cost-effective but makes the costs associated with a reject decision more uncertain and hence the value of research worthwhile (e.g. from ‘Reject¹’ to ‘OIR²’ in Figure 2). e If the reduction in price is greater and sufficient to make the technology cost-effective, then guidance might be revised to AWR if research remains worthwhile and possible with approval (e.g. from ‘Reject¹’ or ‘OIR²’ in Figure 2 to ‘AWR¹’ in Figure 1). Clearly, with an even greater reduction in price, it is possible that provisional guidance to reject could be altered to early approval (e.g. ‘Approve¹’ in Figure 1). Even if research is not possible with approval, a sufficient reduction in price could also lead to early approval (e.g. from ‘Reject¹’ or ‘OIR²’ in Figure 2 to ‘Approve^2,3,4’ in Figure 1). f

Therefore, consideration of the effect of price changes on OIR and AWR is needed when assessing the potential impact of patient access schemes and more direct price negotiation through value-based pricing. 65,77–79 It should be noted that, all other things being equal, the presence of significant irrecoverable costs will require greater reductions in effective price to achieve the same revision to a more permissive category of guidance.

Incentives for evaluative research

These threshold prices represent the maximum effective price at launch to achieve a particular category of guidance when the results of any subsequent research that might be undertaken are not yet known. This is different to the type of flexible pricing agreements described in the current Pharmaceutical Price Regulation Scheme (PPRS) in which price is revised once the research reports and the results are known, with prices increasing if the evidence suggests that benefits were originally underestimated and reducing if they were overestimated. 81 This means that manufacturers retain an incentive to conduct further evaluative research if they believe that there are additional benefits that could not be evidenced at launch. Publically funded evaluative research, however, will still be required when these incentives are insufficient and especially in those cases in which the original evidence is likely to have overestimated the benefits or underestimated the potential for harm. However, it should be noted that linking effective prices to the results of publically funded research means that the NHS will benefit (realise the value of evidence) only if the results lead to a lower price or more restrictive guidance because the technology is found not to be cost-effective (thus avoiding the losses associated with negative NHEs). Manufacturers will, however, be able to appropriate the value of evidence if research results suggest that NHEs were originally underestimated because prices will increase within the flexible pricing scheme (or a value-based pricing scheme that might replace it). Even under current arrangements this value can be appropriated when the technology is reappraised by NICE (e.g. any patient access scheme could be withdrawn or less restrictive positive guidance issued). Consideration of how the NHS and manufacturers are likely to share the value of evidence might inform whether manufacturers should be expected to conduct the research specified in AWR or OIR guidance. As long as incentive-consistent contractual arrangements can be set in place, that is, those that can be monitored and enforced with credible penalties, it should ensure that any agreed research is conducted in the way intended. Alternatively, manufacturers might be expected to make some contribution to the costs of publically funded research that may ultimately benefit their product (see Chapter 5, Point 7: Are the benefits of approval greater than the costs?).

It is important that policy provides (or at least does not undermine) appropriate incentives for manufacturers to conduct the type of research needed to support NICE guidance at launch. The use of OIR and AWR guidance, as described in the algorithm, could provide clear signals and incentives. For example, the threshold price for reject/OIR and reject/AWR will be higher than that for OIR/approve and AWR/approve. Guidance restricted to OIR also offers very limited NHS volumes and revenue to manufacturers. This provides a strong incentive to ensure that the type of evidence which would require research that cannot be conducted once a technology is approved for NHS use is available and is sufficient at launch (e.g. relative effectiveness and subtle but important differences in side-effect profiles). Therefore, a predictable OIR and AWR policy signals what type of evidence is likely to be most important at an early stage.

The use of OIR and AWR, as described in the algorithm, offers manufacturers a choice to (1) accept OIR and AWR guidance at a higher price, (2) reduce the effective price to achieve approval, when that is possible or (3) conduct the evaluative research at an earlier stage so that cost-effectiveness is not uncertain at launch. Other things being equal, those new technologies that are supported during the NICE appraisal process by more, better-quality and relevant evidence will be more likely to be approved (rather than receiving OIR or AWR guidance) and at higher prices than those that are not, because additional evidence is less likely to be needed. Therefore, greater consideration of OIR and AWR will tend to reward those manufacturers who have invested in good-quality and relevant evidence with earlier approval of their technology. In addition, the effect of price on OIR and AWR recommendations suggests that those technologies supported by better evidence will tend to get approval at higher effective prices, providing an incentive for manufacturers to invest in the type of evidence needed earlier in the development process.

Assessing the prospects of research

When considering OIR or AWR guidance there must be some assessment of (1) the type of research needed to address the key uncertainties, (2) whether or not this will be regarded as ethical and can be undertaken while the technology is approved for use, (3) whether or not it is likely to be a priority for public funding and be commissioned and (4) when it is likely to report.

Although the NICE appraisal process may be well suited to identifying the need for evidence when assessing cost-effectiveness, these other critical assessments are not necessarily ones for which NICE and its Advisory Committees, as currently constituted, have particular expertise, not least because they reflect the decisions of those responsible for research design, prioritisation and commissioning. 3,82 Without sufficient co-ordination between these communities there is a danger that OIR or AWR could be issued when the type of research required would not be regarded as either ethical or feasible or of sufficient priority compared with other competing research needs to be commissioned. Because publically funded research is also budget constrained, it is perfectly possible that research that might be valuable from a wider NHS perspective might nevertheless not be a priority if other more valuable research might be displaced. This might be a particular concern if there is a possibility that the research could be undertaken by the manufacturer rather than displacing other research without proprietary interest. Therefore, a decision of whether OIR or AWR research should be undertaken by the manufacturer or through publically funded research is one that NICE cannot properly take alone.

Although some judgement about how the research community might respond to OIR or AWR recommendations when NICE is formulating guidance is clearly possible, more informed judgements and better decisions might be possible through greater involvement of the research community. For example, a Research Advisory Committee could be constituted that could consider provisional OIR or AWR guidance, making recommendations during the consultation period about the type of research needed and its ethics, feasibility and likely priority before final appraisal and guidance. It might also make recommendations about whether research should be publically funded or undertaken by the manufacturer with appropriate contractual arrangements. There are of course many different ways in which greater co-ordination might be achieved. However, because some of the assessments that NICE must make in formulating OIR or AWR guidance are, in fact, research decisions that fall outside its remit, it would seem sensible to draw on the expertise of those involved in, and responsible for, these types of research decisions to help make these assessments.

Issues common to ‘only in research’ and ‘approval with research’ recommendations

Generally speaking, the benefit of attaching research conditions to NICE recommendations is an improved evidence base for resource allocation decisions in the future. The beneficiaries of the research are members of future populations who will profit from better informed allocation decisions. But achieving this benefit can impose significant opportunity costs on current patients. This is true for some OIR decisions and some AWR decisions.

To be clear about when a research condition does impose an opportunity cost on the present population, and how significant this is, two issues need to be carefully considered. The first is what is meant by present and future populations. For the purposes of this discussion the present population comprises people whose interests are directly affected by a NICE recommendation (e.g. they receive an innovative treatment approved by NICE, or benefit from resources made available because NICE rejects an innovative treatment). Future populations comprise people whose interests are indirectly affected by decisions, in particular by the subsequent research results that improve the evidence base for future NICE judgements. It is important to note that on these definitions some people – specifically patients with a chronic condition who live sufficiently long – will be members of both the present and the future populations. Thus, some individuals in the present population may benefit from the research condition because they will also be members of the future population. This will not be true of all, so the issue of balancing the interests of some individuals in the present population against some individuals in the future population remains.

The second issue is under what circumstances the present population is disadvantaged by the research condition compared with the alternative recommendation that NICE might make. This will depend both on what the alternative recommendation would be and on the level of current evidence about cost-effectiveness for the intervention. Three general types of situation might arise:

1. The evidence concerning the cost-effectiveness of a new treatment compared with standard treatment is genuinely balanced such that the likelihood of a new treatment turning out to be less cost-effective than standard treatment is the same as the likelihood of that new treatment turning out to be more cost-effective than standard treatment. Under these conditions it makes no difference to the present population whether a research-conditional recommendation is made or not. Indeed it makes no difference whether the new intervention is accepted or rejected. Future populations, however, will benefit from the research. Some current patients will be members of both present and future populations and will therefore benefit from research.

2. The balance of evidence as judged by NICE suggests that the new treatment is less cost-effective than standard treatment but NICE wishes strongly to encourage further research, hence it is considering a research-conditional recommendation. Assuming that NICE’s judgements on this issue are generally better than chance then the present population is disadvantaged by a research-conditional recommendation compared with rejection but advantaged by a research-conditional recommendation compared with acceptance. Future populations are always advantaged by effective research.

3. The balance of evidence is that the new treatment is more cost-effective than standard treatment. In this situation the present population is disadvantaged by a research-conditional recommendation compared with acceptance but advantaged by a research-conditional recommendation compared with rejection. Again, future populations will be advantaged by effective research.

There is currently little empirical evidence about whether or not NICE’s judgements about the relative cost-effectiveness of two interventions, in situations (2) and (3) above, are better than chance. If research-conditional recommendations are allowed then it will be important to collect data to examine whether or not further research, on average, confirms NICE’s initial judgement about expected cost-effectiveness. There is some evidence which suggests that relevant authorities do accurately judge when evidence favours neither one intervention nor the other. 84 This provides some grounds for expecting that when NICE judges that the evidence on cost-effectiveness favours an intervention, such a judgement will be correct more often than it is false.

In summary, in judging whether or not it is right to make a research-conditional recommendation there are four key issues to consider:

1. What is the likely effect on the current population of a research-conditional recommendation compared with whichever would be the alternative recommendation?

2. What is the likely benefit to the future population from the research?

3. What proportion of individuals in the present population is also likely to be in the future population?

4. How should we weigh up the disadvantages to individuals in the present population in relation to the advantages to individuals in the future population (some of whom will be in the present population)?

The first three of these considerations are essentially empirical issues and NICE will have to make judgements in the setting of uncertainty. The fourth issue is an ethical one. Do the interests of members of the future population count? If so, how are they to be weighed against the interests of members of the present population? Which set of interests should prevail?

One way of addressing this question is to consider how radical a departure from current NICE values, principles and practices it would be to accord weight to the interests of future populations. Arguably, doing so would not be much of a departure at all. For one thing, taking the interests of future populations into account is consistent with fundamental NICE assumptions about how to make allocation decisions. Specifically, NICE currently considers how the requirements of maximising health gain, need, equity and so on are to be balanced against each other; in doing so, NICE takes the identifiability of patients who benefit from an intervention to be irrelevant. So, the fact that beneficiaries of putting a research condition on approval are unidentifiable because they will exist only in the future does not seem to add anything to considerations that NICE currently recognises and weighs. Similarly, some consideration of future populations is implicit in standard NICE judgements. Consider, for example, new treatments for which evidence is poor but, nonetheless, what evidence there is suggests that they are cost-effective. If NICE took only the interests of the present population into account then these should be funded on the basis of the poor but positive evidence. In fact, such innovations are often rejected to ensure that the evidence attains the standard that NICE requires. The beneficiaries of this decision are not in the present population; what NICE has in mind is future populations of patients who will benefit from allocation decisions made on better evidence.

It would seem, then, that decisions that take into account the interests of future populations are consistent with NICE’s values, principles and practices. Nonetheless, there are some issues that require further thought. Although trading the interests of present and future populations coheres with the general NICE framework, quite how these sets of interests should be weighted will be important. Consider, for example, what might be called the ‘bird in the hand’ argument. Suppose that NICE is balancing the health benefit that will accrue from early approval against the value of further evidence. That the latter is considered greater than the former has to be offset against the fact that the former is more secure than the latter simply because one can be more confident about events in the near future (in this case that patients will enjoy health benefits from early approval) than about events in the further future (in this case that the research will report and improve the evidence base for allocation decisions). At this point, the discussion of the ethics of research-conditional judgements segues into practical considerations familiar from other sections of this report, such as the likelihood of research being conducted.

Issues specific to ‘only in research’ recommendations

To tease out the ethical issues raised by OIR it will be useful to construct an illustrative case. Suppose that NICE appraises a new treatment for which there is strong evidence of effectiveness – that is, the innovation is known to be clinically superior to existing treatments – but there is considerable uncertainty over its cost-effectiveness. Whether or not the new treatment would prove to be cost-effective depends not only on how expensive it is but also on how much health benefit it would produce – that is, quantification of the health benefit – compared with standard treatment. NICE might consider an OIR judgement in these circumstances to establish more exactly the size of benefit that, in turn, is deemed necessary to establish cost-effectiveness. For example, NICE could approve the new treatment only in the context of a RCT comprising two trial arms, the innovative treatment arm and the standard treatment arm. Crucially, on this decision, patients outside the trial, and participants randomly allocated to the standard arm of the RCT, would be denied what is almost certainly the better treatment for their condition. This scenario creates a number of important ethical issues.

Issues specific to ‘approval with research’ recommendations

‘Approval with research’ raises a problem about consent, which is related to the discussion of coercion above. AWR also raises a further distinctive issue centring on the likelihood of the research taking place.

Summary of ethical analysis

Although they create numerous ethical challenges, OIR and AWR judgements are ethically permissible given the values and principles that underpin standard NICE practices. Nonetheless, this ethical analysis has identified some issues requiring further thought, which are highlighted in this summary.

Inclusion and exclusion criteria

All NICE technology appraisal draft and final guidance up to January 2010 was considered for inclusion in the review. NICE guidance documents are published in a standardised format, with the guidance to the NHS presented in section 1. The rest of the document provides an overview of the evidence (sections 2 and 3), an explanation of how the evidence was interpreted by the committee (section 4), additional information to assist the implementation of the guidance (section 5) and a list of recommended related research (section 6). The research recommendations in section 6 do not form part of the formal guidance to the NHS. The committee’s final recommendations [Final Appraisal Determinations (FADs)] are made publicly available and can be appealed by specific stakeholders before becoming final guidance to the NHS. In 2002 the NICE process was amended to also publish draft guidance [Appraisal Consultation Documents (ACDs)] for public consultation. This review included both final published guidance and draft guidance. In some cases multiple ACDs or FADs were issued by NICE when guidance changed following consultation; in these cases all ACDs and FADs, along with the final guidance, were reviewed.

The criterion for inclusion was guidance from the Technology Appraisal Programme that recommends research, or refers to ongoing or planned research, in the guidance section of the documents. The research recommendations could be framed either as OIR or AWR based on the following definitions for this review:

• OIR – a recommendation which states in the guidance section that the technology should not be used routinely unless it is in the context of further research

• AWR – a recommendation which states in the guidance section that the technology should be used routinely and which recommends further research is conducted.

Only documents that have been made publicly available are included [specifically, ACDs for Technology Appraisals (TAs) TA1–43, except TA32, were not made publicly available]. Documents that have been publicly released but later removed from the NICE website are included in the review (e.g. guidance that has been replaced by a subsequent review) and have been obtained directly from NICE where appropriate. Draft recommendations that request further clarification or analysis from the sponsor of the technology (sometimes referred to as ‘minded no’ recommendations in the STA process) are excluded as they usually require the reanalysis of existing data rather than additional data collection. The documents containing OIR/AWR recommendations were cross-checked with a list of appraisals including OIR recommendations complied by NICE to check for potential omissions (Sarah Garner, Associate Director of Research and Development, 4 March 2010, personal communication).

Data extraction and analysis

Data from each draft and final guidance document that included OIR and/or AWR recommendations were extracted using a template developed for the project (see Appendix 3). The template was developed following the review of the literature described in Chapter 2, and iteratively alongside the development of the framework described in Chapter 3. Data extracted included information on the appraisal process, information on the technology, reported and accepted incremental cost-effectiveness ratios (ICERs) and the reasons for including an OIR or AWR recommendation in the guidance.

Data were extracted by one reviewer (JY) and a sample cross-checked by another reviewer (LL). The data were analysed to identify common characteristics of appraisals that included OIR and/or AWR recommendations. When OIR/AWR recommendations changed between draft and final guidance, explanations for the change were reviewed and assessed. In addition, the extracted data were analysed to assess the extent to which the assessments proposed in the framework in Chapter 3 were evident in informing the OIR/AWR recommendation in the draft or final guidance. Specifically, the following assessments were considered:

• significant irrecoverable costs

• cost-effectiveness of the technologies

• need for further research and the type of evidence requested

• possibility of conducting research with and without approval

• impact of price on the considerations

• resolution of uncertainties over time

• relative costs and benefits of research

• other considerations.

Review of implementation of the ‘only in research’/‘approval with research’ recommendations

Each piece of NICE guidance is considered for review at a specified length of time after publication (usually 3 years). To examine the impact of OIR/AWR recommendations on evidence collection and future guidance we considered guidance that included OIR/AWR recommendations and that had been updated by NICE. The documents were examined for changes in the evidence base available between original appraisal and review that were considered by the committee, and for changes to the decisions reflected in the guidance.

Use of ‘only in research’ and ‘approval with research’ recommendations in NICE technology appraisal guidance

Of the 184 appraisals conducted up to January 2010, 40 included OIR/AWR recommendations in the draft and/or final guidance. In total, 29 FADs and 31 ACDs were issued relating to these 40 appraisals and which included OIR/AWR recommendations. Multiple ACDs were released for some appraisals; the 31 ACDs with OIR/AWR recommendations arise from 25 appraisals. All of the 29 FADs relate to the final guidance to the NHS (i.e. there were no changes to the recommendations between publication of the FAD and it becoming guidance). Table 2 shows the frequency of OIR and AWR recommendations in the guidance documents. A list of all appraisals including OIR and AWR recommendations is provided in Appendix 3.

The research recommendations in the guidance most commonly took the form of OIR guidance. The terminology used in the OIR guidance differed between appraisals. Some guidance was specific about the type of research recommended whereas other guidance was much more general. For example, TA588 on the introduction of liquid-based cytology for cervical cancer screening is very specific in the research it recommends; it includes a whole host of recommendations and refers to ‘a programme of pilot implementation projects’ that should evaluate a range of impacts including ‘the effect on test results’, ‘the extent to which productivity improvements in cytology laboratories are realised in routine practice’ and ‘the impact in the primary care setting’. However, the guidance for a subgroup of patients in TA16389 simply states that ‘infliximab should only be used for the treatment of acute exacerbations of active ulcerative colitis in clinical trials’. It could be inferred that the more specific recommendations are recommending the research in a more positive way, whereas some of the more general recommendations could be seen to be akin to a diluted reject decision sometimes referred to as a ‘polite no’. Some guidance also refers to research ongoing at the time that the guidance was published. For example, TA890 on hearing aids states that ‘There is insufficient robust scientific evidence to support the nationwide introduction of digital hearing aids at present. Evidence regarding the benefits of digital devices as compared with the current NHS range and to more sophisticated analogue devices, is expected to be available after the completion of research projects currently being undertaken in the UK.’

Despite not being a formal category of guidance used by NICE, a handful of appraisals took the form of AWR guidance, that is, they recommended the technology but also recommended further research in the guidance section. One example of an AWR recommendation in final guidance comes from TA11391 on the use of inhaled insulin. The guidance section recommends the use of inhaled insulin for a specific subgroup of patients and also includes the statement that ‘Data on the use of inhaled insulin according to this guidance should be collected as part of a coordinated prospective observational study.’ Another example of AWR guidance included very specific recommendations for further research. TA3692 on treatments for rheumatoid arthritis recommended within the guidance that ‘All clinicians prescribing etanercept or infliximab should (with the patient’s consent) register the patient with the Biologics Registry established by the British Society for Rheumatology (BSR) and forward information on dosage, outcome and toxicity on a 6-monthly basis.’

Changes to the inclusion or exclusion of OIR/AWR recommendations between draft and final guidance were more common than suggested by the summary numbers in Table 2. Eleven appraisals included OIR/AWR recommendations in the draft guidance but not in the final guidance. Three appraisals included OIR/AWR recommendations in the final guidance but not in the draft guidance (ACDs were unavailable for a further 12 appraisals). The reported issues that led to the changes are explored further later in this chapter.

Most pieces of NICE guidance included several recommendations. These related to multiple technologies, multiple indications or different settings for the use of the technology. Over half of the OIR/AWR recommendations specified the need for further research in specified subgroups of patients (52% of OIR/AWR recommendations in final guidance documents). In approximately a quarter of cases, the OIR/AWR recommendations targeted a subset of the technologies included in the appraisal.

Table 3 shows the frequency of OIR/AWR recommendations by year since the beginning of the Technology Appraisal Programme, and the total number of final OIR/AWR recommendations as a proportion of the total number of pieces of final guidance that year. Sixteen per cent of all final guidance included an OIR/AWR recommendation. It appears that OIR/AWR guidance has been issued less frequently over the last few years. No final guidance included OIR/AWR recommendations in 2007, which is also the year that the STA process was introduced. Differences in the frequency of OIR/AWR recommendations were observed between the two NICE appraisal processes. Of appraisals issued through the multiple technology appraisal (MTA) process, OIR or AWR recommendations were included in draft guidance of 23 appraisals and final guidance of 28 appraisals. These 28 appraisals account for 19% of all final guidance issued within the MTA process. In the STA process only two ACDs and one piece of final guidance contained OIR/AWR recommendations. This accounts for just 2% of all final guidance issued through the STA process up to the time that the review was conducted.

The data were examined for differences in the use of OIR/AWR recommendations according to general disease area and the type of technology under appraisal. In absolute terms OIR/AWR recommendations were more common for cancer treatments (n = 10), accounting for over a third of all of the OIR/AWR recommendations in final guidance, followed by musculoskeletal conditions (n = 7), which accounted for almost a quarter of cases identified (Table 4). However, NICE has appraised a large number of treatments for cancer: 28% of all published appraisals over the review period. Only 7% of all NICE technology appraisal guidance has related to musculoskeletal conditions and so it appears that a disproportionate amount of this guidance has included OIR/AWR recommendations compared with appraisals for other conditions.

Just over half of the final guidance with OIR/AWR recommendations related to the appraisal of drugs (n = 16; 55%). However, taking into account the total number of pieces of drug guidance published, the use of OIR/AWR appears to be on average less common for drug appraisals: 11% of all drug appraisals within the period contained OIR/AWR compared with 47% of all guidance on therapeutic or surgical procedures and 27% of all guidance on devices.

Consideration of irrecoverable costs

Investment and irrecoverable costs were not explicitly quantified in any of the guidance documents and there were no documented considerations relating to these costs in the OIR/AWR guidance. This suggests either that the technologies reviewed by NICE do not have significant irrecoverable costs or that such costs do not explicitly influence decision-making. Three of the technology appraisals that included OIR/AWR guidance (TA60,93 TA68,94 TA9795) had their funding direction extended and, although the rationale for this is not explicit on the Department of Health website, the characteristics of the technologies suggest that it could be due, at least in part, to a need for significant additional training for staff. In addition, one appraisal explicitly noted some concerns around training although this was not directly cited as a reason for issuing the OIR guidance. TA5196 (an earlier version of TA9795) on computerised cognitive behavioural therapy (CCBT) stated that ‘Further information is required about the extent of training needed and circumstances under which different staff could provide support for users of CCBT.’ Thus, it would appear that some of the technologies considered in the Technology Appraisal Programme are associated with significant irrecoverable costs even if these have not previously been explicitly incorporated into the decision-making process.

Cost-effectiveness of technologies

As expected given the NICE process, all OIR/AWR appraisals included a consideration of the cost-effectiveness of the technologies. Most of the guidance documents reported several different estimates of incremental cost-effectiveness based on analyses submitted by different stakeholders relating to different uses of the technology or based on different sets of assumptions or evidence. However, a formal assessment of cost-effectiveness was not always conducted or reported in the ACD or FAD for the use of the technology specified in the OIR/AWR recommendation. Table 5 shows the ICERs (incremental cost per QALY gained) for the overall population and for the specific OIR/AWR indication where this differs. The table distinguishes between the base-case estimates of the ICER submitted by the AG/ERG in their original analysis and the estimate of the ICER considered most plausible by the Appraisal Committee following appraisal of all of the evidence submitted to NICE and noted in the guidance documents.

Most documents did not cite the ICER considered by the Appraisal Committee to be most realistic. The ERG/AG estimates were more frequently reported and it is likely that they were available in supporting documents and were not directly referred to in the ACD or FAD. ICERs for technologies with OIR/AWR recommendations were frequently > £30,000 (the upper bound of the NICE threshold range). The ‘other’ category includes ICERs that were reported using non-QALY-based outcome metrics and ICERs presented as a range that could not be classified into the categories. For example, TA588 on the use of liquid-based cytology reported ICERs of £1100 and £2500 per life-year gained depending on the length of the screening interval. When ICERs were not directly reported, there was usually an indication of whether the technology was considered to be cost-effective. For example, TA6597 states that ‘The clinical and cost effectiveness of rituximab in patients with localised disease has not been established’ and TA4498 states that ‘Appraisal Committee believed that metal on metal hip resurfacing arthroplasty was likely to be of similar cost effectiveness to conventional total hip replacements in people who were expected to outlive the device’.

Table 6 shows the numbers of technologies stated to be cost-effective when used in the context of the OIR/AWR recommendation according to the final guidance document. In most cases an OIR recommendation was issued and the technology was on average not cost-effective. There were two examples of an OIR recommendation being used when the technology was probably cost-effective based on the accepted analyses. Both of these appraisals (TA588 on liquid-based cytology and TA5196 on CCBT) requested that pilot implementation programmes be undertaken before routine introduction of the technologies in the NHS. The technology was considered likely to be cost-effective in three of the four cases in which an AWR recommendation was issued. In the remaining appraisal including an AWR recommendation [TA3692 – etanercept (Enbrel^®, Wyeth) and infliximab (Remicade^®, Schering-Plough) for rheumatoid arthritis], the ICERs were £27,000–35,000, close to, or above, the upper end of the cost-effectiveness threshold range.

Need for further evidence and type of evidence requested

The rationales for issuing the OIR/AWR recommendations as stated in the guidance documents are summarised in Table 7 (note that multiple reasons were cited in some cases). Three of the four appraisals that did not provide a rationale were issued before the committee’s considerations were routinely described in the documents. The OIR recommendation in the other appraisal (TA7599 on hepatitis C) related to three specific subgroups of patients; two subgroups were not referred to in the committee’s considerations at all and no evidence was noted for the third subgroup.

A need for further evidence on the relative effectiveness of the intervention in the overall population or the OIR/AWR subgroup was the most commonly cited reason for issuing the OIR/AWR recommendation. A need for longer-term data was also frequently cited. Uncertainty in the cost-effectiveness estimates was also a common consideration; however, in all cases this was coupled with a need for further clinical evidence. Concern about the budget impact of introducing the technology, investment and reversal costs and the potential impact on ongoing research did not lead to the OIR/AWR recommendation in any of the appraisals.

In some appraisals the stated rationale for the OIR/AWR recommendation changed between draft and final guidance. For example, the use of infliximab for the treatment of acute exacerbations of ulcerative colitis (TA16389) was recommended OIR in both the ACD and the FAD. However, the final guidance cited only poor relative effectiveness evidence as a rationale for the OIR recommendation whereas the draft guidance also cited a need for other types of evidence such as information on adverse effects and cost-effectiveness data.

Possibility of conducting research with and without approval

No appraisals cited a concern for the impact of recommendations on ongoing trials as a direct rationale for the OIR/AWR guidance. In addition, the possibility of conducting research was not explicitly noted in most appraisals. Decision-makers may be more likely to use OIR recommendations than AWR when data on relative effectiveness are required. A lack of sufficient evidence on relative effectiveness was cited in 19 pieces of final guidance identified in the review and most of these included OIR recommendations (i.e. including those that stated a need for more evidence on relative effectiveness for the whole population or for evidence on relative effectiveness for the target OIR population in Table 7). The exception to this was TA11391 (inhaled insulin), which included an AWR recommendation for a highly selective subgroup of patients, and which noted the committee consideration that data on relative effect would be most appropriately collected through a disease register.

Experimental research designs, such as RCTs, were more frequently referred to, and were slightly more common in draft guidance (Table 8). Two appraisals cited a need for further evidence on relative effectiveness in the final guidance but recommended observational studies because of anticipated difficulties in conducting RCTs in the specific OIR patient population (TA37100) or indication (TA167101). There were changes in the recommended type of research between draft and final guidance, which were mainly due to changes in the target OIR/AWR population (e.g. TA6894) or changes to be less specific about the research design (e.g. TA89102).

Overall, reference to the likelihood of research being conducted was rare in the guidance documents. However, there was one notable exception in which the likelihood of further research being conducted led to a change in the guidance: the ACD for TA129103 stated that bortezomib monotherapy was not recommended except for use in well-designed clinical studies; however, this recommendation was removed in the FAD, which noted that no further studies of bortezomib were planned and highlighted a difficulty of conducting further research in the group of patients to which the guidance related (this guidance was later further amended following the offer of a patient access scheme).

The impact of price on ‘only in research’/‘approval with research’ recommendations

Price changes or discounts featured in two of the appraisals including OIR/AWR recommendations. A patient access scheme was incorporated into TA129103 and the guidance changed from reject to approval. This appraisal had previously included an OIR recommendation in draft guidance but was changed to reject prior to the offer of the access scheme as described in the previous section. In the second appraisal, an OIR recommendation was revised to approval after the committee revised their estimates of cost-effectiveness based on discounted prices of the technology along with further information on quality-of-life improvements (TA166104 on cochlear implants).

Other considerations

Considerations around whether uncertainties in the evidence base would resolve over time were not explicitly mentioned as reasons for issuing OIR or AWR recommendations. In addition, the relative costs and benefits of conducting research were not reported as considerations of the committee when formulating its research recommendations.

Review of implementation of the ‘only in research’/‘approval with research’ recommendations

Among the appraisals with OIR/AWR recommendations in the final guidance, 10 were later reviewed by NICE, including two that were incorporated into NICE clinical guidelines. Table 9 provides details of the appraisals and whether or not additional evidence was provided and the change to the OIR/AWR recommendation (new evidence for other recommendations included within the guidance is not noted in the table).

In the majority of reviewed appraisals (n = 7), new evidence informing the OIR/AWR recommendation was available for the review. In four of these reviews, the OIR or AWR restriction was removed and the technology was recommended routinely. In two cases the additional evidence was considered insufficient to warrant a change in the OIR recommendation. In the remaining appraisal the OIR was revised so that some technologies within the class were recommended routinely whereas OIR recommendations were issued for others (TA5196 on CCBT). In all cases the changes in the guidance were owing to the new data relating to the evidence gap identified in the OIR/AWR recommendation.

In three cases no new evidence was provided on the OIR/AWR indication. For the review of TA6,105 no new RCT data were available for the OIR recommendation, which was made more restrictive in the review guidance. New evidence on clinical effectiveness was not available for the review of TA33,106 but further information on adverse effects was provided. This was considered inadequate and no change was made to the OIR recommendation. The OIR recommendation was removed from the review of TA37100 despite a lack of new evidence presented. The documents state that the reasons for this were a reduction in demand for the drug in this setting (it had since become licensed and NICE approved for treatment of an earlier stage of disease) and concerns about the feasibility of future data collection.

Selection of case studies

Case studies were selected to ensure that the full range of possible analysis was feasible within the time and resource constraints of this research project, while exploring situations in which OIR or AWR might be particularly relevant and challenging. Therefore, de novo analysis or substantial reanalysis of original assessments is not possible. Nor would it be necessary or informative as one of the objectives is to explore what additional information and analysis might be required. For this reason candidate case studies that met the following feasibility criteria were considered: (1) the economic analysis was regarded as a suitable basis for developing guidance, (2) an analysis of uncertainty in expected cost-effectiveness [probabilistic sensitivity analysis (PSA) as specified in the NICE reference case] was conducted and (3) ready access was available to the electronic versions of the models that informed guidance.

There are three groups of potential case studies in which the key principles and assessment described above might have influenced guidance: (1) when OIR or AWR was included in the FAD, (2) when OIR or AWR was considered during appraisal (e.g. included in the ACD or section 6 of the technology appraisal guidance) and (3) when OIR or AWR was not obviously considered at any stage. As well as examples of AWR for technologies expected to be cost-effective and OIR for those not, there are also a number of particularly interesting ways in which guidance might be influenced by these additional considerations. For technologies expected to be cost-effective these include (1) OIR rather than approve when research is not possible with approval and (2) OIR or even reject rather than AWR or approve even if research is possible with approval because there are significant irrecoverable costs.

To fully explore the implications of these principles and assessments it is useful to select case studies that reflect the range of possible and interesting characteristics. For example, technologies (1) that are and are not expected to be cost-effective, (2) with and without irrecoverable costs, (3) for which other sources of uncertainty are and are not present, (4) for which the research needed is and is not possible with approval, (5) that are non-pharmaceutical interventions and (6) that are appraised under the MTA and STA process. Four case studies will not be enough to demonstrate the full range of possible combinations of interesting characteristics or illustrate all of the potential impacts of interest. Therefore, in selecting case studies there was a need to balance feasibility and coverage of those characteristics of greatest interest.

Background to the case studies

The following four case studies were selected. A range of additional information was sought and further analysis conducted to inform the sequence of assessment and judgements required when completing the OIR/AWR checklist in Tables 10A and 10B.

Point 1: Is it expected to be cost-effective?

The sequence of assessments starts with cost-effectiveness and the expected impact on population NHEs, that is, at the following point in the algorithm:

This requires an assessment of expected cost-effectiveness based on the balance of the evidence and analysis currently available. Methods to estimate expected cost-effectiveness are well established within the NICE appraisal process and are extensively described in the Guide to Methods of Technology Appraisal. 1,b Commonly, expected cost-effectiveness is summarised and presented using ICERs. Equivalently, but more usefully in this context, cost-effectiveness can be expressed in terms of expected NHEs, which can be expressed per-patient treated or for a population of patients. This is especially important when later assessments require a comparison of benefits for current and future patient populations and when assessing the significance of irrecoverable costs (see Point 2: Are there significant irrecoverable cost?). All of the information required to express expected cost-effectiveness in these ways is available from the type of analysis already undertaken during appraisal.

Technologies expected to be cost-effective

The results for EECP are summarised in Table 11. There are only two alternatives (EECP and standard care) and so only one ICER. EECP is just expected to be cost-effective at a threshold of £20,000 per QALY. e Consequently, the NHEs of EECP are greater than those of standard care but the difference per patient treated (the incremental NHE) is small.

It is also important to consider how NHEs accumulate over time or the investment profile per patient treated with EECP. Figure 3 illustrates the cumulative incremental NHE over the patient time horizon. The initial per-patient costs of EECP are high and are far in excess of the immediate health benefits in the initial period of treatment. These negative NHEs are gradually offset by positive NHEs in later periods. In this case, it is only after 14 years that the initial losses are compensated by later gains, that is, EECP is not expected to break-even until 14 years from initial treatment. It is only beyond 30 years that the modest incremental NHEs reported in Table 11 are eventually achieved. f

FIGURE 3.

Cumulative incremental NHEs of EECP over the patient time horizon.

Multiple alternatives

Similar analysis can be conducted when there are more than two alternatives. For example, in the CLOP case study four treatment durations as well as current NHS treatment (aspirin alone) were considered at the time of TA80. 122 The results in Table 12 indicate that 12-month treatment with CLOP is expected to be cost-effective, although the difference in NHE between 12 and 6 months’ treatment is small.

TABLE 12 - Expected cost-effectiveness of CLOP per patient treated

Treatment	Cost (£)	QALYs	ICER (£/QALY)	Cost-effectiveness threshold £20,000 per QALY	Cost-effectiveness threshold £30,000 per QALY
				NHE, QALYs (£)	NHE, QALYs (£)
CLOP12	20,127	8.122	18,663	7.115 (142,307)	7.451 (223,525)
CLOP6	19,860	8.107	10,477	7.114 (142,288)	7.445 (223,362)
CLOP3	19,712	8.093	9396	7.108 (142,154)	7.436 (223,087)
CLOP1	19,598	8.081	4961	7.101 (142,025)	7.428 (222,837)
NHS	19,502	8.062	–	7.087 (141,734)	7.412 (222,353)

The investment profile of CLOP, per patient treated, is illustrated in Figure 4. The per-patient costs of CLOP are in excess of the health benefits during the period of treatment. These negative NHEs are eventually offset by positive NHEs in later periods. In this case, it is only after 5 years that 12 months of treatment with CLOP breaks-even against current NHS care and it is not until 21 years that it is better than a shorter treatment duration of 6 months. Notice that shorter treatment durations with CLOP offer a much less ‘risky profile’, for example the break-even point for 1 month of treatment is 2 years against current NHS care.

FIGURE 4.

Cumulative incremental NHEs of CLOP over the patient time horizon.

Technologies not expected to be cost-effective

The ICER for OMAL in Table 13 is greater than the threshold and so it is not expected to be cost-effective compared with standard care alone. Consequently, the incremental NHE of OMAL is negative.

The per-patient investment profile for OMAL is illustrated in Figure 5 and shows that it is always expected to offer negative NHEs compared with standard care over the entire patient time horizon, that is, the high costs of treatment are never compensated by future health gains. In this example, the initial treatment costs with OMAL continue for 10 years (10 years is assumed to represent the duration that a patient would continue to receive treatment with OMAL) with health effects predominately while on treatment. Therefore, OMAL is not so much a ‘risky purchase’ but one that is simply not cost-effective at its current price.

FIGURE 5.

Cumulative incremental NHEs of OMAL over the patient time horizon.

Multiple alternatives

Psoriatic arthritis offers an example in which the alternatives to the treatment already recommended by NICE (etanercept at the time of TA199125) are not expected to be cost-effective, that is, the results in Table 14 indicate that etanercept is expected to be cost-effective. Notice that although adalimumab is less effective than etanercept it is also cheaper; however, the resource savings it offers do not compensate for the reduction in health benefits.

TABLE 14 - Expected cost-effectiveness of PsA treatments per patient treated

Treatment	Cost (£)	QALYs	ICER (£/QALY)	Cost-effectiveness threshold £20,000 per QALY	Cost-effectiveness threshold £30,000 per QALY
				NHE, QALYs (£)	NHE, QALYs (£)
1: Infliximab	90,343	7.269	60,965	2.752 (5504)	4.258 (8516)
2: Etanercept	78,150	7.069	17,733	3.161 (6322)	4.464 (8928)
3: Adalimumab	72,972	6.777	14,622	3.129 (6258)	4.345 (8690)
4: Palliative care	51,800	5.329	–	2.739 (5478)	3.602 (7204)

Consequently, the investment profiles of the alternatives to etanercept, illustrated in Figure 6, differ in appearance. However, all of the biologic treatments for PsA have high initial costs, which are only gradually compensated for by later health benefits. Adalimumab, etanercept and infliximab all ultimately offer positive NHEs compared with palliative care but only break-even at 17, 17.5 and 34.5 years, respectively. Adalimumab offers a slightly less risky profile than etanercept and so it is only at 21.25 years that etanercept is expected to offer the highest NHE.

FIGURE 6.

Cumulative incremental NHEs in PsA over the patient time horizon.

Cost-effectiveness at the population level

Per-patient NHEs can also be expressed for the population of current and future patients. This requires information about prevalence and future incidence of the target population (already required in appraisal). It also requires a judgement about the time horizon over which the technology will be used. This ‘technology time horizon’ ought to reflect the period over which the technology is likely to be part of clinical practice and generate the expected NHEs. g An estimate of the scale of the total population NHEs and how they cumulate over time is important for subsequent assessments, including (1) when the NHE for current patient populations must be compared with the benefits to future patients and (2) when the treatment decision can be changed so that the irrecoverable opportunity costs of initially negative NHEs become significant, that is, might influence the category of guidance.

For example, there is a large prevalent population (n = 109,800) eligible for EECP relative to future incident populations (n = 9500 per annum) in this chronic condition. The total population NHEs, assuming that the technology will be used to treat prevalent and incident patients over 10 years, are reported in Table 15. The expected cost-effectiveness is unchanged (the ICER is the same as in Table 11) but the incremental NHE, although small per patient, is more significant at a population level because of the size of the target population.

TABLE 15 - Expected cost-effectiveness of EECP for the population

Treatment	Costs (£M)	QALYs	ICER (£/QALY)	Cost-effectiveness threshold £20,000 per QALY		Cost-effectiveness threshold £30,000 per QALY
				NHE, QALYs (£M)	Incremental NHE, QALYs (£M)	NHE, QALYs (£M)	Incremental NHE, QALYs (£M)
EECP	896	1,435,787	19,391	1,391,001 (27,820)	1405 (28)	1,405,930 (42,177)	16,334 (490)
Standard care	–	1,389,596	–	1,389,596 (27,792)		1,389,596 (41,688)

The investment profile for EECP when used to treat patients over 10 years is illustrated in Figure 7. At a population level it is not until 17 years (rather than 14 years at a patient level) that initial losses are compensated for by later gains and EECP breaks even. In other words, EECP appears a more risky investment when evaluated at a population rather than an individual level. This is because, although each patient treated with EECP is expected to offer the same profile of NHEs as shown in Figure 3, the negative NHEs associated with patients incident and treated in year 10 will not be offset by later gains until year 24. The population-level investment profile would exhibit greater risk (break-even later) if the prevalent population was smaller relative to the incident population and/or the technology time horizon was longer. For example, the break-even point extends to 23 years when the technology time horizon is increased to 20 years.

FIGURE 7.

Cumulative incremental NHEs of EECP for the population.

The effect on the other case studies of assessment at the population level is similar to the effect on the EECP case study. It simply increases the magnitude of differences in per-patient NHE (to a greater extent for longer technology time horizons) but leaves expected cost-effectiveness unchanged. However, the investment profiles at a population level also differ, exhibiting greater ‘risk’ indicated by later break-even points for the same reasons as for EECP. For example, the break-even points for CLOP when evaluated at a population level are reported in Table 16. At a technology time horizon of 10 years it is only at 11 years, rather than 5 years for a single patient, that 12 months of CLOP treatment breaks even against current NHS care and not until 27 years (rather than 21 years) that 12 months of CLOP treatment is better than a shorter treatment duration of 6 months. Even the shorter durations of treatment offer a ‘risky profile’, for example the break-even point for 1 month of treatment against current NHS care is 4 years (rather than 2 years).

TABLE 16 - Expected cost-effectiveness of CLOP for the population

Technology time horizon	Treatment	Incremental NHE, QALYs (£M)	Break-even point (years)
			12 months vs 6 months	12 months vs NHS	1 month vs NHS
5 years	CLOP12	269 (5.4)	24	8	4
	CLOP6	1881 (37.6)
	CLOP3	1804 (36.1)
	CLOP1	4073 (81.5)
	NHS	–
10 years	CLOP12	495 (9.9)	27	11	4
	CLOP6	3465 (69.3)
	CLOP3	3324 (66.5)
	CLOP1	7502 (150)
	NHS	–
15 years	CLOP12	686 (13.7)	30	12	4
	CLOP6	4799 (96)
	CLOP3	4603 (92.1)
	CLOP1	10,389 (207.8)
	NHS	–
20 years	CLOP12	846 (16.9)	33	12	4
	CLOP6	5921 (118.4)
	CLOP3	5680 (113.6)
	CLOP1	12,820 (256.4)
	NHS	–

Point 2: Are there significant irrecoverable costs?

The second point on the checklist requires (1) an assessment of whether there are irrecoverable costs and (2) a judgement of their potential significance, that is, at the following point in the algorithm:

Irrecoverable costs are those that once committed cannot be recovered if guidance is changed at a later date (see Appendix 6, Are there significant irrecoverable costs?). Irrecoverable costs are most commonly thought of as upfront or capital costs of new facilities or equipment with long life expectancy (they might also include any practitioner training and the costs of implementation efforts). In NICE appraisal these types of cost are first annuitisedh and then allocated pro rata to the number of patients likely to be treated during the lifetime of the equipment. That is, capital costs are treated as if they are paid per patient treated over the lifetime of the equipment. If guidance remains unchanged throughout this period (i.e. research does not report or other sources of uncertainty do not resolve) then this common assumption has no influence. However, should guidance change (initial approval is withdrawn) before the end of the lifetime of the equipment then, although future patents will no longer use the technology, the cost of the equipment allocated to them cannot be recovered. The possibility that initial guidance might change and its impact on expected costs needs to be considered before costs are made irrecoverable through approval or AWR. The impact of irrecoverable costs will tend to be greater if they represent a greater proportion of the total costs, if guidance is more likely to change and if guidance is more likely to change in the near rather than distant future.

Enhanced external counterpulsation is the only case study in which these types of cost are present to any great extent because treatment requires capital investment in the EECP machines themselves. The expected per-patient and population costs reported in Tables 11 and 15 allocated this capital cost in the usual way (i.e. annuitised over the 10-year lifetime of the machines and allocated to the number of patients treated each year). The irrecoverable costs are reported separately in Table 17 and represent 19% of the total. However, this will have no influence on expected cost-effectiveness as long as guidance does not change during the lifetime of the equipment. i

Point 3: Does more research seem worthwhile?

The third point on the checklist requires an assessment of the potential benefits of conducting further research, that is, at the following point in the algorithm:

This requires judgements about (1) how uncertain a decision to approve or reject a technology might be based on the estimates of expected cost-effectiveness and (2) whether or not the scale of the likely consequences of this uncertainty might justify further research. Some assessment of the potential consequences of uncertainty is important because it indicates the scale of the population NHEs that could be gained if the uncertainty surrounding this decision could be resolved immediately, that is, it represents an expected upper bound on the benefits of more research. m If the potential benefits of further research are unlikely to justify the costs, then a judgement that more research does not seem worthwhile will lead directly to guidance in the following circumstances (extracted from Table 32 in Appendix 4):

Assessing the consequences of uncertainty

Some assessment is required of (1) how uncertain a decision based on expected cost-effectiveness might be and (2) what the consequences, in terms of population NHEs, are likely to be if an incorrect decision is made.

Enhanced external counterpulsation is expected to be cost-effective compared with standard care (Table 18 and see Table 11), but the estimates of cost and QALYs are uncertain and so there is a chance that a decision to approve EECP based on existing evidence will be incorrect, that is, standard care might offer greater NHEs. Some assessment of the likely consequences of approving EECP when standard care might be better could be based on the difference in expected NHEs, that is, the expected incremental population NHEs reported in Tables 15 and 18). This is illustrated in Figure 8 in which a judgement about the probability that a decision based on expected cost-effectiveness is correct translates into expected consequences based on the expected incremental population NHE. For example, if the decision was judged to be 100% certain then there are no consequences and so there would be nothing to be gained by more research; however, as the probability that the decision is correct becomes less certain, the expected consequences (and hence potential value of more research) increase.

TABLE 18 - Expected consequences of uncertainty for EECP

Treatment	ICER (£/QALY)	Cost-effectiveness threshold £20,000 per QALY			Cost-effectiveness threshold £30,000 per QALY
		Incremental NHE, QALYs (£M)	Probability cost-effective	Expected consequences, QALYs (£M)	Incremental NHE, QALYs (£M)	Probability cost-effective	Expected consequences, QALYs (£M)
EECP	19,391	1405 (28.1)	0.428	9287 (185.7)	1,405,930 (490)	0.700	2774 (83.2)
Standard care		–	0.572		–	0.300

FIGURE 8.

Probability that EECP is cost-effective and the consequences of uncertainty.

This judgement of how uncertain a decision might be can be informed by the PSA already used to estimate costs and QALYs and required as part of the NICE reference case. 127,128 The probability that EECP is cost-effective is 0.428 (see Table 18),n which would translate into approximately 800 QALYs (see Figure 8) over the technology time horizon,o based on the expected or average difference between NHEs. However, the difference in NHEs when EECP is not the correct decision is not necessarily the average. In fact, it is very unlikely to be the average and such estimates may substantially under- or overestimate the expected consequences of uncertainty. p

The same probabilistic analysis can be used to record the difference between the NHEs of EECP and the NHEs of standard care and the frequency of such errors (i.e. the proportion of simulations in which standard care offers greater NHEs). The distribution of consequences associated with these errors is illustrated in Figure 9. Commonly there are no consequences, because EECP is the correct decision (i.e. there is a 42.8% chance of zero consequences). However, when EECP offers lower NHEs than standard care (a 57.2% chance) there are consequences in terms of NHEs forgone. Figure 9 illustrates that the consequences of errors may be relatively small, for example 9% of the simulations have consequences of < 5000 QALYs. However, consequences may also be very large, for example there is a small chance (5.7%) that they are > 30,000 QALYs. The average over this distribution provides the correct estimate of the expected consequences of uncertainty, which in this case is 9287 QALYs. q

FIGURE 9.

Distribution of the consequences of uncertainty for EECP.

These expected consequences can be interpreted as an estimate of the population NHEs over the technology time horizon that could be gained if the uncertainty surrounding this decision could be resolved immediately, that is, it indicates an expected upper bound on the benefits of more research. 127,129,r The consequences can also be expressed as the equivalent NHS resources required to generate the same population NHEs (£185.7M in Table 18). They will increase with the size of the patient population and the technology time horizon. In the case of EECP the consequences fall with the cost-effectiveness threshold because a decision to approve EECP will be less uncertain (see Table 18). A judgement at this point that more research might be worthwhile seems reasonable because the upper bound on its potential benefits exceeds the likely costs.

Multiple alternatives

Similar analysis can be conducted when there are more than two alternatives but greater difficulties are encountered unless the results of the PSA are used to assess both uncertainty and its consequences. For example, in the CLOP case study, 12 months’ treatment with CLOP is expected to be cost-effective but this is also uncertain. A judgement is required about the chance that 12 months of treatment is incorrect and if so which of the other four alternatives are likely to offer a higher NHE and how much higher. In other words, for decisions involving multiple alternatives, a judgement is required on the level of uncertainty surrounding the decision, how this uncertainty is distributed across the various alternatives and what the consequences are likely to be. The results of the PSA can inform this judgement. The probabilities that each of the five alternatives is cost-effective are reported in Table 19. This indicates that 12 months’ treatment is uncertain (probability that it is incorrect is 0.476). However, much of this probability of error is allocated to 6 months’ treatment with CLOP (0.18) for which the difference in NHEs is likely to be relatively modest.

TABLE 19 - Expected consequences of uncertainty for CLOP

The mean additional population NHE of moving from the least to the most effective alternative, that is, the incremental NHE of 12 months’ treatment compared with NHS care is the sum of these increments (14,786 QALY or £295.7M at £20,000 per QALY)

Treatment	ICER (£/QALY)	Cost-effectiveness threshold £20,000 per QALY			Cost-effectiveness threshold £30,000 per QALY
		Incremental NHE,a QALYs (£M)	Probability cost-effective	Expected consequences, QALYs (£M)	Incremental NHE,a QALYs (£M)	Probability cost-effective	Expected consequences, QALYs (£M)
CLOP12	18,663	495 (9.9)	0.524	5194 (103.9)	2798 (56.0)	0.677	3657 (109.7)
CLOP6	10,477	3465 (69.3)	0.180		4736 (94.7)	0.092
CLOP3	9396	3324 (66.5)	0.018		4305 (86.1)	0.009
CLOP1	4961	7502 (150.0)	0.075		8327 (166.5)	0.052
NHS	–	–	0.202		–	0.170

The distribution of consequences is illustrated in Figure 10. Most commonly (52.4%) there are no consequences because 12 months’ treatment with CLOP is the correct decision. When it is not, there is a greater chance of relatively small consequences of error (30% are < 10,000 QALYs), which occur predominantly when 6 months’ treatment duration offers the highest NHE (18% chance). There is a small chance of much larger consequences (< 5% chance that they are > 30,000 QALYs). These occur only when standard NHS treatment offers the highest NHE. Overall, there is an almost 20% chance that NHS treatment offers a higher NHE, that is, there remains important uncertainty about the cost-effectiveness of CLOP itself, not just the duration of treatment. The expected consequence of uncertainty (5194 QALYs) is simply the average over this distribution. Again, this can be interpreted as an estimate of the population NHE that could be gained over the time horizon of this technology if the uncertainty about treatment and its duration could be immediately resolved. Therefore, like EECP, a judgement at this point that more research might be worthwhile seems reasonable because the potential benefits exceed the likely costs.

FIGURE 10.

Distribution of the consequences of uncertainty for CLOP.

Psoriatic arthritis provides a similar picture to that of CLOP, in which approval of the alternative (etanercept), which is expected to be cost-effective, is uncertain (probability that approval is incorrect is 0.557), but in this case most of this probability of error is associated with palliative care (probability of 0.4 that it is cost-effective). Again, there is a greater chance of relatively small consequences (19% are < 28,000 QALYs), most of which occur when adalimumab has the highest NHE, and a smaller chance of very large consequences (4.7% chance that they are > 138,000 QALYs), which occur when palliative care offers the greatest NHE. The expected consequences of uncertainty and the upper bound on the population NHE that might be gained by immediately resolving uncertainty (35,342 QALYs or £707M over the technology time horizon) supports a judgement that more research may be worthwhile.

Alternative scenarios

There are often alternative views about the quality and relevance of evidence as well as other assumptions that might be made when estimating expected costs and QALYs. These are commonly presented as separate scenarios, with estimates of costs and QALY presented for each. Much of the deliberation by the Appraisal Committee often surrounds the scientific value judgements required to judge the credibility of the alternative assumptions represented by the scenarios. The type of probabilistic analysis reported represents the uncertainty within each scenario and will be sufficient to indicate the potential benefits of research when only one scenario is regarded as credible. However, when more than one scenario might be credible and carry some ‘weight’, there will be uncertainty between as well as within scenarios. The ‘weighting’ of scenarios can be made explicit by assigning probabilities to represent how credible each is believed to be. The weighted average of costs and QALYs across scenarios can easily be calculated (see Appendix 6, Is further research required?, Alternative scenarios). It is also tempting to take a simple weighted average of the expected consequences of uncertainty across these scenarios as well. However, a simple weighted average may under- or overestimate the combined consequences of uncertainty within and between scenarios (see Appendix 11, Why averaging scenarios may be misleading). 130 The correct estimate requires the probabilities (weights) to be applied directly to the simulated output from PSA rather than to the mean values. Although this does not require additional simulation and is quick and easy to implement, it does require either that the probabilities are made explicit in advance or that estimates be presented for a range of probabilities that might represent the judgement of the Appraisal Committee following deliberation.

For example, the CLOP analysis presented above assumes a constant relative treatment effect for different durations of treatment (scenario A). An alternative assumption (scenario B) was that the relative treatment effect also differed by duration based on the data reported in the Scottish Intercollegiate Guidelines Network (SIGN) guidelines. This alternative assumption made longer durations less cost-effective and reduced the expected consequences of uncertainty from 5195 to 3969 QALYs. Although scenario A was regarded as more credible by the Appraisal Committee, scenario B might nevertheless carry some weight or have some probability associated with it. In this case the simple weighted average of expected consequences (linear combination of mean estimates) is very similar to the correct estimate based on weighting the output of PSA in Figure 11. This also shows how these estimates can be presented for a range of probabilities.

FIGURE 11.

Expected consequences of uncertainty with alternative scenarios: CLOP case study.

An alternative assumption of a common class effect across the three biologics was considered in the PsA case study (scenario B) but was judged less credible than the analysis that allowed differential effects (scenario A). The alternative scenario made etanercept less likely to be cost-effective and increased the expected consequences of uncertainty from 34,930 to 38,521 QALYs (Figure 12). In this case a simple weighted average of expected consequences based on the probability assigned to each scenario is, in general, lower than the correct estimate of expected consequences based on the output from PSA.

FIGURE 12.

Expected consequences of uncertainty with alternative scenarios: PsA case study.

Elicitation

The single RCT of EECP showed evidence of improvements in quality of life at 12 months;121 however, the degree to which these are sustained in the long run is uncertain. Rather than make alternative assumptions and present extreme scenarios, formal elicitation of the judgement of clinical experts about the likelihood of QALY gains in subsequent years was undertaken. s The uncertainty in these elicited values is included in the estimates of the expected consequences of uncertainty reported in Table 18, which might otherwise have been represented by alternative scenarios, for example no QALY benefits beyond 12 months could be assumed for scenario A, benefits sustained for a patient’s lifetime for scenario B and benefits sustained for 4 years for scenario C. The results of elicitation implied probabilities of 0.243, 0.353 and 0.404 associated with each of these scenarios, respectively. A simple weighted average of the expected consequences within each scenario using these probabilities (1442 QALYs) significantly underestimates both the estimate of expected consequences based on all of the information from elicitation (9287 QAYs) and the estimate based on weighting scenarios using the simulated output rather than the mean estimates (13,081 QALYs). This illustrates that (1) a simple weighted average of expected consequences may be misleading and (2) elicitation may provide a richer characterisation of uncertainty as well the probabilities associated with alternative assumptions (see Appendix 11, Why averaging scenarios may be misleading for further discussion).

Point 4: Is research possible with approval?

The fourth point on the checklist requires an assessment of what type of evidence is needed and a judgement of whether the research required to generate it can be conducted while the technology is approved, that is, at the following point in the algorithm:

Although the decision at this point does not lead directly to guidance, it does determine whether AWR or OIR is a possibility. This judgement will depend, in part, on whether the type of evidence that is needed will require an experimental research design. For example, more precise estimates of relative treatment effect are likely to require a RCT if the dangers of selection bias are to be avoided; however, further RCTs for this particular indication and patient group are unlikely to be possible once a technology is approved for widespread NHS use.

This requires judgements about (1) how important particular types of parameters (inputs to the economic model) are to estimates of cost and QALYs, (2) what values these parameters would have to take to change a decision based on expected cost-effectiveness, (3) how likely is it that parameters might take such values and (4) what would be the consequences if they did, that is, what might be gained in terms of population NHE if the uncertainty in the values of these parameters could be immediately resolved (see Appendix 6, Is the research possible if the intervention is approved?).

Point 5: Will other sources of uncertainty resolve over time?

The fifth point on the checklist requires an assessment of whether or not changes are likely to occur in the future that will influence the cost-effectiveness of the alternative technologies and the potential benefits of research, that is, at the following point in the algorithm:

The judgement made at this point will influence the potential benefits of research and therefore subsequent decisions that lead directly to a particular category of guidance (see point 6 below). Even when research was not considered worthwhile (at point 3) the presence of other sources of uncertainty will determine whether or not significant irrecoverable costs are likely to influence the category of guidance. In some circumstances it can lead directly to guidance, that is, if there are no other sources of uncertainty even significant irrecoverable costs will have no influence and a technology that is expected to be cost-effective can be approved:

This assessment requires information about (1) changes in prices of the technology and its comparators, (2) the emergence of new technologies that might make existing ones obsolete or change their cost-effectiveness and (3) other relevant research reporting (see Appendix 6, Impact of other sources of uncertainty). A number of potential sources of information and evidence were examined to inform this assessment for each case study (see Appendix 4 for the full details of the sources and searches conducted). However, many potentially useful sources were proprietary or public access was restricted, making it surprisingly difficult to inform these assessments with publically available information. When information and estimates were available they were often not directly relevant to a UK context.

Point 6: Are the benefits of research greater than the costs?

The sixth point on the checklist requires a reassessment of the potential benefits of conducting further research that were initially considered at point 3, and a judgement of whether the benefits of research are likely to exceed the costs, that is, at the following point in the algorithm:

A judgement about whether or not the potential benefits of research identified earlier (see Is further research required?) will be realised requires an assessment of (1) whether or not the type of research that is required is likely to be conducted, (2) if conducted, when the results are likely to be available, (3) how much uncertainty is likely to be resolved and (4) the likely impact of the other sources of uncertainty identified in Point 5: Will other sources of uncertainty resolve over time? on the longer-term benefits of research (see Will the research be conducted?, How long until the research reports?, How much of the uncertainty will the research resolve? and How do other sources of uncertainty impact on the need for evidence?, Appendix 6, respectively).

The decision at this point may not necessarily lead directly to guidance, for example when the benefits of research exceed the costs but research is not possible with approval or there are significant irrecoverable costs. Which category of guidance will ultimately be appropriate will depend on whether or not the benefits of approval are judged to exceed the costs, that is, point 7 of the checklist (see next section). However, in many other circumstances the decision at this point will lead directly to a particular category of guidance. These circumstances or pathways through the algorithm are detailed below (extracted from Table 32 in Appendix 4):

The expected consequences of uncertainty reported in Point 3: Does more research seem worthwhile? represented the NHE that could be gained over the lifetime of the technology if the uncertainty surrounding the decision based on expected cost-effectiveness could be immediately and completely resolved. This represents an upper bound on the potential benefits of research for a number of reasons: (1) research, although recommended, might not be commissioned and/or recruit and report, (2) any research will take some time to complete before results are available and (3) not all of the uncertainty is likely to be resolved. In addition, future events (identified in Point 5: Will other sources of uncertainty resolve over time?) might change the NHE expected to be gained by future patient populations. Finally, the expected benefits of research once properly reassessed must be compared with the likely costs.

Will the research be conducted?

Even if research is recommended in OIR or AWR it might not be undertaken by manufacturers or commissioned by research funders. Even if undertaken or commissioned there is no guarantee that research will be able to recruit or it may not complete for other reasons (see Appendix 6, Will the research be conducted?). The expected consequences of uncertainty for CLOP and EECP reported in Point 3: Does more research seem worthwhile? are illustrated in Figures 13 and 14, respectively, for a range of probabilities that research will be successfully undertaken. This indicates that the potential gains depend on a judgement of whether the research recommended as part of OIR or AWR will be successfully completed. This also illustrates that the cost of research (in this case considered to be either £1.5M or £10M) can be compared directly with the potential benefits by either (1) expressing the potential gains in population NHE as the equivalent NHS resources (i.e. the resources that would be required to generate the same NHE) or (2) expressing the cost of research in terms of the QALYs that could be gained elsewhere in the NHS by using the same resources to provide access to health care.

FIGURE 13.

Expected potential benefits of research: CLOP case study.

FIGURE 14.

Expected potential benefits of research: EECP case study.

When will it be available?

Research, even if commissioned and successfully completed, will take time to commission, complete and report. Therefore, any assessment of the potential benefits should account for the fact that patient populations will not benefit from the results of research until they are available and change clinical practice. y Whether or not those patients who are prevalent while research is under way will be able to benefit from the results will depend on whether treatment decisions for presenting patients are irreversible or not (see Point 2: Are there significant irrecoverable costs?). If treatment decisions are irreversible, for example in the CLOP case study, it is only those patients incident after the research reports who will realise any of the potential benefits. In contrast, treatment decisions in EECP are not irreversible (it is a chronic condition) and so, although patients prevalent while research is undertaken will not benefit immediately, those who survive can benefit from the results once the research is completed. How long research might take to report will depend in part on the design (follow-up, sample size and end points), recruitment rate and size of the eligible patient population, as well as on how efficient the organisation and data collection might be. The potential value of research in CLOP and EECP over a range of possible time horizons is reported in Figures 15 and 16, respectively. In both cases the potential value of further research declines with increase in time to research reporting. This relationship gives some indication of the value of improving the timeliness of research through, for example, investment in research infrastructure or adopting a research design that, although offering less potential benefits, can be conducted more quickly. It also indicates the value of efforts to ensure that any revised NICE guidance following OIR or AWR is more quickly and more completely implemented.

FIGURE 15.

Potential value of research by time to report: CLOP case study.

FIGURE 16.

Potential benefits of research by time to report: EECP case study.

What is the impact of other sources of uncertainty?

In Point 5: Will other sources of uncertainty resolve over time? the information that was publically available identified that the patent for CLOP was due to expire 7 years after the appraisal. Based on the OFT estimate that generic prices tend to be 25% of the original brand price, this other source of uncertainty can be integrated quantitatively when estimating the potential value of research over the lifetime of the technology. In this case a significant fall in price in year 7 will substantially reduce the uncertainty surrounding 12 months of treatment with CLOP. Therefore, after year 7 there is less to be gained from resolving uncertainty and so the potential and value of research findings for patients incident after year 7 are reduced. The effect of a price change on research that could potentially resolve uncertainty in cost, natural history and utilities as well as relative effect is also illustrated in Figure 15. The potential value of the research is lower whenever the research reports because it includes the value to future as well as current patient populations. Nevertheless, even if research did not report until 7 years the potential value is likely to exceed the costs. The expected price reduction reduces the potential value of research at each time point, for example, from 174,519 QALYs when research is immediately available (see Figure 17) to 165,701 with a price change at year 7. z

FIGURE 17.

Potential value of research by time to report and other sources of uncertainty: CLOP case study.

There was some evidence of possible entry of a new technology (comparator) in the indication described in the CLOP case study. However, there was limited information on its characteristics. Therefore, two alternative but somewhat extreme scenarios are illustrated in Figure 17. In scenario A the new technology enters at year 5 and makes CLOP entirely obsolete, that is, not cost-effective and not uncertain (equivalent to a shorter technology time horizon of 5 years). At this point there is no value in the evidence generated by research about CLOP. aa In these circumstances research is likely to be worthwhile only if it reports quickly. In scenario B the new technology has a similar NHE to 12 months of treatment with CLOPab and the uncertainty surrounding its expected cost-effectiveness is also similar. Now research about CLOP has more potential value in the future because it will also help resolve some of the uncertainty in the choice between CLOP and the new technology for patients who become incident after that time. Although there was no evidence of new technologies emerging in EECP, the same scenarios are explored as the development and launch of new devices are more difficult to identify in advance. The impact on the potential value of research is illustrated in Figure 18, demonstrating similar qualitative effects as for CLOP. In scenario A (EECP becomes obsolete) the potential benefits of further research about EECP are likely to exceed the costs only if the research reports quickly. Nevertheless, even in this extreme scenario the benefits of research with only 1 year of follow-up are likely to exceed the costs as long as it reports before 4 years.

FIGURE 18.

Potential value of research by time to report and other sources of uncertainty: EECP case study. These potential values are based on a 1-year follow-up design.

The potential value of research presented in these figures, even after accounting for the type of evidence, follow-up and time until research reports, should still be regarded as an upper bound to the value that is likely to be realised by actual research for two reasons: (1) even well-designed research with large sample sizes will not fully resolve the uncertainty in the value that a parameter might take, especially in specific target populations and in particular (future) contexts and (2) insofar as implementation of NICE guidance is not ‘perfect’ and all clinical practice might not immediately respond to the results of research, the full benefits will be realised only over time or with additional implementation efforts. For these reasons a judgement of whether or not benefits of research are likely to exceed the costs might be made conservatively, requiring evidence that even in pessimistic scenarios the research would still be worthwhile.

Point 7: Are the benefits of approval greater than the costs?

The seventh and final point on the checklist requires an assessment and comparison of the benefits and costs of early approval. The costs of approval are not financial ones but opportunity costs and will include the potential value of any research that may be forgone as a consequence, for example if the research needed cannot be conducted once the technology is approved for use. They will also include any costs that are irrecoverably committed by approval. As well as the capital costs of equipment and facilities (or training and learning), they will also include the irrecoverable opportunity costs of initially negative NHEs (if treatment decisions are not irreversible – see the discussion in Chapter 3, Technologies with significant irrecoverable costs and Chapter 5, Point 1: Is it expected to be cost-effective? and Point 2: Are there significant irrecoverable costs?). A judgement of whether or not the benefits of approval and early access for current patients are likely to exceed the opportunity costs for future patients is required, that is, at the following point in the algorithm:

The decision at this point always leads directly to guidance, with all remaining possible pathways allocated to a particular type and category of guidance. These remaining (20) pathways through the algorithm are detailed below (extracted from Table 32 in Appendix 4):

Technologies without significant irrecoverable costs

Only 4 of the 20 possible pathways illustrated above are associated with technologies without significant irrecoverable costs. In these four pathways either (1) research was not considered possible with approval for those expected to be cost-effective (i.e. ‘Approve²’ or ‘OIR¹’) or (2) research was not possible without approval for those not expected to be cost-effective (i.e. ‘AWR²’ or ‘Reject²’). The CLOP case study provides an example of the former. It is research that would provide more precise estimates of the relative effect of CLOP and of shorter treatment durations that is potentially valuable (see Point 4: Is research possible without approval?). As a consequence, the type of experimental design that is likely to be needed is unlikely to be possible if 12 months of treatment with CLOP is already approved for widespread NHS use. Although treatment with CLOP does commit initially negative NHEs that are irrecoverable, these should not be regarded as significant as the treatment decision for a presenting patient is irreversible in relevant time frames (see Point 2: Are there significant irrecoverable costs?). Therefore, AWR may not be possible and so the benefits of early access to 12 months of treatment with CLOP (approval) must be compared with the potential value of OIR.

‘Only in research’ is more likely to offer greater expected NHEs than approve if the research can be conducted quickly and report sooner, as fewer patients forgo their access to CLOP and more can have their treatment choice informed by the research findings. This is illustrated in Figure 19, which reports the difference between approve and OIR in population NHEs over a range of times for when the research recommended in OIR might report. This takes account of both the expected changes in price at year 7 and research costs of £10M. It shows that OIR will be appropriate only if the research reports within 3 years of appraisal (T* = 3) because beyond this time the NHEs forgone by withholding access to CLOP will exceed the potential gains to future patients.

FIGURE 19.

Population NHEs of approve and OIR for time to research reporting: CLOP case study. T* is the time to research reporting at which the NHEs of OIR and approve are equal. That is, for a time to report of less than T* OIR offers higher NHEs than approve and for a time to report of more than T* approve offers higher NHEs than OIR.

The trade-off between NHEs for current patients and NHEs for future patients that lies behind Figure 19 is illustrated in Figure 20 using undiscounted values for ease of exposition. It illustrates the (per-period) population NHEs of approval and OIR if the research recommended as part of OIR reports at year 3. At this point the initial losses of NHEs caused by restricting access to CLOP (area A) start to be offset by the potential gains from the research findings (area B). The price change at year 7 increases the NHEs of approval (i.e. CLOP is more cost-effective) but on balance reduces the NHEs of OIR. In other words, because CLOP is more cost-effective and offers greater NHEs the evidence generated by the research is less valuable because the choice of treatment and duration is less uncertain (see Point 5: Will other sources of uncertainty resolve over time? and Point 6: Are the benefits of research greater than the costs?). With research reporting at 3 years the initial losses of OIR (area A) are just offset by the later gains (area B) so T* = 3. If research reported earlier than 3 years (area A < area B) OIR would be appropriate but if it reported later than 3 years (area A > area B) approve would be more appropriate.

FIGURE 20.

Population NHEs of approve and OIR at T*: CLOP case study.

However, there is no guarantee that the research recommended as part of OIR guidance will be conducted by manufacturers or commissioned by research funders. Even if it is, it is not certain that it will be successfully completed (see discussion in Point 6: Are the benefits of research greater than the costs?). Therefore, the probability that research will report at a particular time also needs to be considered. The implications of considering whether the recommended research will be conducted and when it might report are illustrated in Figure 21, which presents a boundary for when OIR might be appropriate or when approval should be granted. For example, if research is certain to report but will take 4 years, or when it will take only 1 year but has only a 50% chance of reporting, OIR would not be appropriate and 12 months of treatment with CLOP should be approved, that is, points that fall to the north-east of the boundary. Points to the south-west of the boundary indicate that OIR might be appropriate (see Appendix 6, When might it be commissioned and report).

FIGURE 21.

An OIR or approve boundary: CLOP case study.

However, the estimates of the potential value of additional evidence on which these boundaries are based are still likely to overestimate the value that will be realised by research (see discussion in Point 6: Are the benefits of research greater than the costs?); they represent a necessary condition for OIR. Therefore, OIR guidance should require a conservative judgement that the point is almost certain to be below the boundary rather than on balance close to it. For the same reason, points anywhere above the boundary represent a sufficient condition for approval. The boundary when the change in price is included is to the south-west, reflecting the lower potential value of research, and OIR guidance, once CLOP becomes more cost-effective. In this case it seems unlikely that the type of research required could report quickly enough and with sufficient confidence that OIR would be appropriate. Therefore, these assessments would support a judgement that the benefits of approval are likely to exceed the opportunity costs, and ‘Approve²’ (pathway 3 for CLOP in Table 32, Appendix 4) would seem more appropriate.

The assessments that have been undertaken for CLOP can be brought together to consider (1) what would be the value of being able to conduct research while CLOP is approved (value of AWR) and (2) what would be the value of making evidence that is needed by the NHS available at launch. These questions can be informed by the results already presented elsewhere but these results are also reported together in Table 25.

The difference in NHEs between AWR (if it was possible) and the next best feasible policy, which is OIR when T < T* and approve when T > T*, is £30M and £4M, respectively. This difference represents the value to the NHS of being able to conduct research while CLOP is approved for use, for example to inform whether investment in better data collection, registries or information systems is worthwhile that might make this possible. ac The difference in population NHEs if all uncertainty had been resolved before appraisal (at launch) and the next best available policy, OIR for T < T* and approve for T > T*, is £54M and £80M, respectively. This difference represents the value to the NHS of having access to the evidence needed at launch. This can also directly inform policies that might make better and more relevant evidence available.

It is also possible to consider the commercial as well as NHS value in each of the cells of this table. The value of early evidence at launch can then be considered from the perspective of the manufacturer (the expected revenue streams), taking account of prices (see discussion of price in Chapter 3, Changes in effective prices and Incentives for evaluative research) and expected volumes over the remaining patent life and technology time horizon. Together with estimates of the costs of conducting research by manufacturers or through public funding, this assessment might inform when manufacturers might be expected to conduct the research needed (e.g. high commercial value that exceeds the cost to manufacturers) or when the NHS might be expected to undertake it (e.g. low commercial value but high potential value to the NHS that exceeds the costs to the NHS). When the research is worthwhile from both a commercial and a NHS perspective the question of who should conduct, or pay for, the research might be informed by which sector has a comparative advantage, that is, which can gain the most net social value. Of course, the value to the NHS and to manufacturers will depend, to a large extent, on what type of flexible pricing arrangements and value-based pricing schemes might be in place. The question will also turn on how any agreements can be made and incentive consistent contracts written and enforced.

Technologies with significant irrecoverable costs

Most of the possible pathways illustrated above are associated with technologies with significant irrecoverable costs (16 out of the remaining 20). This is because, even when research is possible with approval (or even when not needed), the impact of committing irrecoverable costs through AWR (or approval) must be considered and so OIR (or reject) remains a possibility (see Appendix 6, What is the impact of irrecoverable costs?). The EECP case study provides an example of this, in which research that would provide more precise estimates of the effect of treatment on quality of life accounts for all of the potential value (see Point 4: Is research possible with approval?). EECP does commit both capital costs associated with long-lived equipment and initially negative per-patient NHEs. Unlike in the CLOP case study these irrecoverable opportunity costs at a patient level are significant because treatment choice for a presenting patient is not irreversible over relevant time frames (see Point 2: Are there significant irrecoverable costs?). As a consequence, even if research is possible with approval it is not clear that AWR would be appropriate, because OIR avoids the commitment of irrecoverable costs until research finding are available and a more informed decision can be made. ad

Research is possible with approval

Even when research is possible with approval OIR offers greater expected NHEs than AWR as long as research reports before 9 years (Figure 22). This is because the consequences (losses of population NHE) of committing both aspects of irrecoverable costs through AWR are greater than the NHEs forgone by restricting access to EECP through OIR. The costs of research have not been included because they are incurred with both AWR and OIR guidance. ae

FIGURE 22.

Population NHEs of AWR and OIR for time to research reporting: EECP case study.

As previously for the CLOP case study, there is no guarantee that the research recommended as part of OIR or AWR guidance will be conducted and research report. A boundary for when OIR rather than AWR might be appropriate is illustrated in Figure 23 for four research designs with differing times of follow-up. A 2-year follow-up will generate evidence with the lowest potential value (so the boundary is to the south-west) but it is likely to report sooner. Therefore, OIR might be appropriate even if the probability that the research will be conducted and report is relatively low. In this case it seems likely that the type of research required could report quickly enough and with sufficient confidence that OIR would be appropriate even though the research could be conducted while EECP is approved. Therefore, these assessments would support a judgement that the benefits of approval (through AWR) are unlikely to exceed the opportunity costs (the NHEs of OIR) and so ‘OIR⁴’ (pathway 18 in Table 32, Appendix 4) rather than ‘AWR⁴’ (pathway 17) would seem more appropriate.

FIGURE 23.

An OIR or AWR boundary: EECP case study.

Research is not possible with approval

For the general reasons discussed in Chapter 3 and those specific to EECP discussed in Point 4: Is research possible with approval?, the type of experimental research required to robustly estimate the effect of EECP on quality of life is unlikely to be possible once it is approved and in widespread use. Now approval (now through approve rather than AWR) not only commits the type of irrecoverable costs discussed above but also means that the potential value of evidence to future patients must be forgone. This is reflected in Figure 24 in which the difference between OIR and approve is always greater than the difference between OIR and AWR in Figure 22. It suggests that as long as the cost of the research exceeds the difference between OIR and approve, when it is expected to report, OIR rather than approve would be appropriate. This is also reflected in the boundaries for OIR and approve reported in Figure 25. These boundaries are always to the north-east of the OIR/AWR boundaries reported in Figure 23, again reflecting the fact that approval not only commits irrecoverable costs but also forgoes the potential value of evidence that might have been generated through an OIR recommendation. These assessments would support a judgement that the benefits of approval are unlikely to exceed the opportunity costs (the NHEs of OIR) and so ‘OIR⁶’ (pathway 25 in Table 32, Appendix 4) rather than ‘Approve⁹’ (pathway 24) would be more appropriate.

FIGURE 24.

Population NHEs of approve and OIR for time to research reporting: EECP case study.

FIGURE 25.

An OIR or approve boundary: EECP case study.

As with the CLOP case study, the assessments that have been undertaken for EECP are bought together in Table 26 and can help inform the same policy questions: (1) what would be the value of being able to conduct research while EECP is approved? and (2) what would be the value of making the evidence that is needed by the NHS available at launch? In this case, because of the irrecoverable costs associated with EECP, there is no value to the NHS of being able to conduct research while EECP is approved for use. In fact, these figures are negative, indicating that even if AWR was possible it would not be appropriate. However, as in the CLOP case study, there is value to the NHS of having the evidence needed before appraisal. This value, expressed in the equivalent NHS resources, depends on how long it would otherwise have taken for an OIR recommendation to deliver the same evidence, for example £62M if 3 years and £134M if 7 years. As with the CLOP case study, these assessments can also inform policies that might make better and more relevant evidence available and answer the questions of how to provide the evidence needed at the right time and who might contribute most to providing this evidence.

TABLE 26 - Population NHEs over the technology time horizon for different policies: EECP case study

AWR*, the expected population NHEs if AWR was to become a possibility; T, time to research reporting.

NHE for time to research reporting	Approve	OIR	AWR*	Reject	Value of AWR	Uncertainty resolved at launch	Value of evidence at launch
NHEs in QALYs
T = 3	1,391,001	1,397,192	1,393,578	1,389,596	–3614	1,400,288	3096
T = 7	1,391,001	1,393,608	1,392,030	1,389,596	–1578	1,400,288	6680
NHEs in £M
T = 3	27,820	27,944	27,872	27,792	–72	28,006	62
T = 7	27,820	27,872	27,841	27,792	–32	28,006	134

Other NICE programmes

The principles outlined in Chapter 3 and the checklist do not presuppose how the assessments required might be informed and judgements made. Distinguishing principles from methods of analysis in this way meant that there was a general consensus that the principles and the checklist might be useful in other NICE programmes, whilst recognising that how the assessment might be made is likely to differ. It was recognised that some amendments might be required when cost-effectiveness is not a consideration (e.g. in interventional procedures) or when only technologies that are expected to be cost saving to the NHS are considered (e.g. the Medical Technologies Evaluation Programme). Similarly, the complexity of multiple outcomes in public health and the greater scope and complexity of decision problems in clinical guidelines offer greater challenges for quantitative assessment but do not change the key principles and considerations. The relative paucity of evidence and the speed of innovation in devices are likely to influence how the assessment might be made. On the other hand, it was considered by some workshop participants that the more iterative process of appraisal in some other NICE programmes (e.g. clinical guidelines) might make it easier to integrate the assessments required in the checklist. Further research, using case studies from these other NICE programmes, would be required to identify how best the assessments might be informed given the programme-specific challenges, the existing methods of appraisal and the resource and time constraints of their current processes.

The checklist was subsequently applied to a series of four case studies (three pharmaceuticals from the Technology Appraisal Programme and one device from the HTA programme; see Chapter 5, Introduction to the case studies). These applications allow an examination of how each of the assessments might be made based on the type of evidence and analysis currently provided in NICE technology appraisal and how they might be better informed with a range of additional information and/or analyses. The possible implications for process and methods of appraisal are discussed below.

Categories and type of guidance

Each sequence of assessment and decision leads to different categories and ‘types’ of guidance for technologies with differing characteristics, indications and target populations, that is, each sequence of ‘yes’ or ‘no’ judgements in the checklist defines a single pathway leading to a particular type and category of guidance (see Table 32, Appendix 4). The different types of apparently similar guidance illustrate how the same category of guidance (e.g. approve, AWR, OIR or reject) might be arrived at in different ways, helping to identify the particular combinations of considerations that might underpin guidance, contributing to the transparency of the appraisal process. There was a general view that the application of the checklist would improve transparency in communicating the considerations that underpin guidance but that this alone would not be sufficient, especially in situations in which OIR guidance was made for a technology that was, on balance, expected to be cost-effective. Evidence of how, not just what, assessments and judgements were made would be required (see What additional information and analysis might be required?).

These principles suggest that the categories of guidance available to NICE have wider application than is reflected in previous guidance (see the review of NICE guidance in Chapter 4). For example, there are five different types of OIR guidance that may be appropriate when a technology is expected to be cost-effective (see Table 1). Indeed, OIR may be appropriate even when research is possible with approval if there are significant irrecoverable costs. AWR can be considered only when research is possible with approval but reject remains a possibility even for a cost-effective technology if there are irrecoverable costs. Therefore, the full range of categories of guidance (OIR and reject as well as AWR and approve) ought to be considered for technologies that, on the balance of existing evidence and current prices, are expected to be cost-effective. It is only approval that can be ruled out if a technology is not expected to be cost-effective, that is, cost-effectiveness is necessary but not sufficient for approval but lack of cost-effectiveness is neither necessary nor sufficient for rejection.

Scope of ‘only in research’ recommendations

Importantly, which category of guidance will be appropriate depends only partly on an assessment of expected cost-effectiveness and hence this assessment should be regarded only as an initial step in formulating guidance. Guidance will depend on a number of other key assessments, which include (1) the need for evidence, (2) whether or not the type of research required is possible with approval, (3) the expected longer-term benefits and costs of the type of research likely to be conducted, (4) the impact of other sources of uncertainty that will resolve over time and (5) the significance of any irrecoverable costs.

In general, it was accepted that as well as AWR for technologies expected to be cost-effective and OIR for those not, there are other important circumstances when OIR should be considered. In particular, for technologies expected to be cost-effective, OIR rather than approve may be appropriate when research is not possible with approval and OIR or even reject rather than AWR or approve may be appropriate even if research is possible with approval when there are significant irrecoverable costs. The ethical implications of such decisions were set out in Chapter 3 (see Social value judgements and ethical principles) and are discussed below.

‘Only in research’, ‘approval with research’ and effective price

Any change in the effective price of the technology, either through patient access schemes (which offer some form of discount that reduces NHS costs) or through direct price changes (possibly negotiated though a future value-based pricing scheme), will affect the key assessments leading to different categories of guidance. For example, provisional OIR guidance for a technology that is expected to be cost-effective might be revised to approve with a sufficient price reduction because the benefits of early approval will be greater and uncertainty about its cost-effectiveness and the value of additional evidence will tend to be lower. Similarly, AWR might be revised to approve if the consequences of uncertainty and what might be gained from additional evidence is sufficiently reduced. Equally, provisional guidance to reject a technology that is not expected to be cost-effective might be revised to OIR if the reduction in price (although not sufficient to make it cost-effective) made a reject decision more uncertain and hence research worthwhile.

Therefore, consideration of the effect of price changes on OIR and AWR is needed when assessing the potential impact of patient access schemes and more direct price negotiation through value-based pricing. However, there are limits to the effects of price reductions as even at a zero price the technology might not be cost-effective and/or further research may still be required if there is a lack of confidence that the technology is effective (harms may not be compensated for by benefits) and/or it imposes other non-acquisition costs on the NHS.

Implications for value-based pricing

The price at which the technology would just be expected to be cost-effective is commonly regarded as the value-based price for the technology. It is the maximum price that the NHS can afford to pay without imposing negative health effects. This describes the threshold price below which approve rather than reject would be appropriate when (1) OIR or AWR guidance is not available to the decision-maker, (2) there is no uncertainty over cost-effectiveness or (3) the research, if needed, can be conducted with approval and there are no irrecoverable costs. In all other circumstances there are a number of other value-based prices, each representing the threshold price below which guidance would change. Importantly, once uncertainty and the need for evidence as well as the impact of irrecoverable costs are recognised, the threshold price that would lead to approval will always be lower than or equal to a single value-based price based on expected cost-effectiveness alone, that is, disregarding uncertainty over costs and effects.

For example, when research could be conducted without approval but not with it, there will be a price threshold above which reject rather than OIR would be appropriate and a lower price threshold below which approve rather than OIR would be appropriate. This approve/OIR threshold price will always be lower than a value-based price based on expected cost-effectiveness. If a technology also imposes significant irrecoverable costs then this threshold price will be lower still. There may also be up to three threshold prices as all four categories of guidance might become relevant at different prices.

Value-based pricing and irrecoverable costs

Health technologies with patent protection are more likely to be priced close to the point at which the ICER is close to or equal to the threshold, that is, expected incremental NHEs are close to zero. A value-based pricing scheme would formalise these existing incentives so technologies will tend to impose greater irrecoverable opportunity costs (initially negative NHEs). A technology that is only just expected to be cost-effective will not break-even until close to the end of the patient time horizon and not for much longer for the population of patients likely to benefit from its use. Therefore, those technologies already priced close to the threshold, and all new technologies considered in a value-based pricing scheme, will tend to increase the scale of irrecoverable costs committed by approval. If these irrecoverable costs are significant (because treatment decisions are not irreversible) then the price threshold for unrestricted access (i.e. approve or AWR) will be lower than the price at which the technology is just expected to be cost-effective, even if research can be conducted with approval (see Chapter 3, Issues specific to ‘only in research’ recommendations). Further research is required to establish how these considerations could be integrated within a practical value-based pricing scheme. In addition, irrecoverable opportunity costs are commonly associated with many health technologies that offer future benefits following treatment. The significance of these types of irrecoverable costs is not widely recognised and further research to demonstrate their potential impact more generally is needed.

Retaining ‘only in research’ as part of value-based pricing

Even in circumstances in which price negotiation becomes possible alongside NICE appraisal, it will be important to retain OIR and AWR as available categories of guidance for two reasons. First, there is no guarantee that manufacturers will always agree to the lower price below which approval rather than OIR or AWR would be appropriate. For example, manufacturers will be unwilling to agree prices that are below marginal production costs. They will also be unwilling to lower UK prices if they can be referenced by other countries and have an impact on global revenues. Second, and possibly more importantly, there may be many circumstances when there is no effective price reduction that would make approval appropriate. For example, reject or OIR guidance may still be appropriate even if the effective price of a technology is zero if there is substantial uncertainty about its effectiveness (and/or potential for harms) or if the technology imposes other non-acquisition costs on the NHS.

Flexible pricing agreements

Consideration of OIR and AWR recommendations provides a link between uncertainty, evidence and price which might appropriately align incentives for manufacturers conducting the type of evaluative research that would be most valuable for the NHS. The current PPRS provides for flexible pricing agreements in which price is revised once the research reports and the results are known, increasing prices if the evidence suggests that benefits were originally underestimated or reducing prices if benefits were overestimated. This means that manufacturers retain an incentive to conduct further evaluative research if they believe that there are additional benefits that could not be evidenced at launch. Publically funded evaluative research, however, will still be required when these incentives are insufficient and especially in those cases in which the original evidence is likely to have overestimated the benefits or underestimated the potential for harm. However, it should be noted that linking effective prices to the results of publically funded research means that the NHS will benefit (realise the value of evidence) only if the results lead to a lower price or more restrictive guidance because the technology is found not to be cost-effective (thus avoiding the losses associated with negative NHEs). Manufacturers will, however, be able to appropriate the value of evidence through higher prices when it suggests that NHEs were originally underestimated. Even under current arrangements this value can be appropriated when the technology is reappraised by NICE. For example, patient access schemes, which offer an effective discount to the NHS, can be withdrawn or approval might be made less restrictive when a technology is reappraised following further research.

Incentives and early advice

It is important that policy provides (or at least does not undermine) appropriate incentives for manufacturers to conduct the type of research needed to support NICE guidance at launch. The use of OIR and AWR guidance, and its link to price, provides a clear signal and an incentive for manufacturers to ensure that the type of evidence that cannot be acquired following approval is available and sufficient at launch (e.g. relative effectiveness and subtle but important differences in side effect profiles). Therefore, a predictable OIR and AWR policy signals what type of evidence is likely to be most important at an early stage. It offers manufacturers a choice, to (1) accept OIR guidance at a higher price but restricted volume, (2) reduce the effective price to achieve approval, or AWR when that is possible, or (3) conduct the evaluative research at an earlier stage so that additional evidence is not required at launch.

Other things being equal, those new technologies that are supported by more, better-quality and relevant evidence will be more likely to be approved and at higher prices than those that are not. Therefore, greater consideration of OIR and AWR guidance will tend to reward those manufacturers who have invested in good-quality and relevant evidence with earlier approval of their technology. It was also suggested that the checklist of assessments could be used within the NICE Scientific Advice Programme, providing advice to manufacturers at an early stage before technology appraisal.

Who should conduct ‘approval with research’ and ‘only in research’ research?

Consideration of how the NHS and manufacturers are likely to share the value of evidence might inform whether manufacturers should be expected to conduct the research specified in AWR or OIR guidance. Alternatively, manufacturers might be expected to make some contribution to the costs of publically funded research that may ultimately benefit their product. As well as an assessment of how the value of additional evidence is likely to be shared between the NHS and the manufacturers, it was widely recognised that two other issues need to be considered. First, the resource constraints on publically funded research may mean that other research priorities (often without commercial interest) may be more valuable to the NHS (see How should the assessments be undertaken?). Second, the success of AWR recommendations when manufacturers are asked to conduct the research will depend on whether or not NICE and/or the Department of Health are able to establish incentive-consistent contractual arrangements as part of an AWR recommendation, that is, arrangements that can be monitored and enforced with credible penalties to ensure that agreed research is conducted and in the way intended. It was widely recognised that, at present, NICE does not have a credible mechanism to ensure that the type of research recommended in AWR guidance would actually be undertaken by manufacturers. Removing approval of a technology simply because recommended research has not been conducted was not considered an ethical or a credible threat. Further research could consider what type of feasible contractual arrangements would offer credible threats that would make AWR more likely to be successful. This work could include more detailed examination of linking an AWR agreement to price penalties within a value-based pricing scheme.

Although OIR provides a greater incentive to undertake research, it was recognised that there may be circumstances when manufacturers would nonetheless choose not to undertake it, that is, accepting an effective reject. If the research is also not a sufficient priority to secure public funding, then approval rather than OIR (an effective reject in these circumstances) would be appropriate only at the lower approve/OIR price threshold. Many workshop participants took the view that it would be better if all AWR and OIR research was publically funded rather than undertaken by manufacturers to ensure wide availability of research findings and provide confidence that commercial interests do not influence its design, conduct and reporting. It was suggested that the costs of publically funded research might be recovered directly from manufacturers or indirectly through other price discounts. Such arrangements might be mutually beneficial in some circumstances (e.g. if the costs of publically funded research are lower), while allowing wider access to the data generated and more transparency and accountability in the conduct of the research.

Further research could examine the relative costs of conducting this type of evaluative research and explore feasible mechanisms for recovering the costs of publically funded research from manufacturers. It should also include an examination of how the value of research is likely to be shared between manufacturers and the NHS in different circumstances and under different price renegotiation rules. The pay-offs for the different policies reported in Tables 25 and 26 are from an NHS perspective only. It would be interesting to estimate the commercial pay-offs associated with each, which could start to inform when manufacturers or public bodies might be expected to conduct or pay for the research recommended as part of AWR or OIR guidance.

Value of ‘approval with research’ and evidence at launch

The assessments that need to be made (especially in Chapter 5, Point 6: Are the benefits of research greater than the costs? and Point 7: Are the benefits of approval greater than the costs?) can be used to consider (1) what would be the value of being able to conduct research while a technology is approved (value of AWR), (2) what would be the value of making evidence that is needed by the NHS available at launch and (3) what is the value of being able to acquire evidence more quickly.

The difference in population NHEs between AWR and the next best feasible policy represents the value to the NHS of being able to conduct research while a technology is approved for use. This can inform investment decisions in better data collection, registries or information systems that might make AWR possible. Importantly, this value will differ by technology and will depend on the scale and significance of irrecoverable costs (if they are sufficiently high there will be no value in AWR – see Chapter 5, Technologies with significant irrecoverable costs). The difference in population NHEs between resolving all uncertainty before appraisal and the next best available policy represents the value to the NHS of having access to the evidence needed at launch. This might inform a range of policies, such as early advice, public investment in transitional and evaluative research earlier in the development process and other incentives for research and development. Understanding the relationship between the time taken for research to report and the value of the evidence to future populations can help to inform (1) investments that might make research findings more quickly available, (2) the trade-off implicit in the choice of alternative research designs (i.e. greater precision or timeliness) and (3) research prioritisation (identifying those areas in which if research is to be undertaken there must be confidence that it can report quickly).

The value of early evidence at launch and AWR can also be considered from the perspective of the manufacturer (the expected revenue streams), taking account of prices and expected volumes over the remaining patent life and technology time horizon. Together with estimates of the costs of conducting research by manufacturers or through public funding, this assessment might inform when manufacturers might be expected to conduct the research needed (e.g. high commercial value that exceeds the costs to manufacturers) or when the NHS might be expected to undertake it (e.g. low commercial value but high potential value to the NHS that exceeds the costs to the NHS). In some circumstances, the value to manufacturers and to the NHS will exceed their respective costs. The question of who should conduct, or pay for, the research might then be informed by which sector has the comparative advantage, that is, which sector can offer greater gains in net social value. The value to the NHS and to manufacturers will depend, to a large extent, on the type of pricing arrangements that might be in place (see Implications of patient access schemes and value-based pricing) and how any agreements that might be made can be enforced with incentive-consistent contracts. As well as understanding the pay-offs from an NHS perspective, these policy questions could be better informed with further research to estimate the likely commercial pay-offs and possible contractual arrangements that might align incentives and make AWR possible.

Order and sequence of assessments

The order of the assessments in the checklist relates to the sequence of decision nodes that fully describe the algorithm in Appendix 2. This order of considerations means that all seven assessments do not necessarily need to be made when an earlier judgement (e.g. at points 3, 5 and 6) can lead directly to guidance. In addition, the early assessment of the scale and potential significance of irrecoverable costs (at point 2) can avoid further and more complex assessment later.

There was a general consensus that the order was appropriate and that all of the assessments do not need to be routinely made for every appraisal, especially at points 6 and 7, which would need to be made only when the Appraisal Committee was considering OIR or AWR as a possibility. However, the assessment of whether, in principle, further research might be worthwhile and what type of evidence might be required would need to be undertaken routinely (additional information or analysis that might inform these assessments is discussed in What additional information and analysis might be required?). For example, at point 3 a judgement that more research does not seem worthwhile will lead directly to guidance (see Point 3: Does more research seem worthwhile?). However, if research may be worthwhile, some indication of the type of evidence needed would also be useful for those making an assessment of the prospects of research (see next section) and whether or not the type of research required to generate the evidence would be possible with approval. Therefore, routine assessment up to point 4 of the checklist (see Point 4: Is research possible with approval?) would seem appropriate before others with expertise and responsibility for research design and commissioning consider the prospects of the type of research needed. In fact, some assessment of other sources of uncertainty, such as future price changes, entry of new technologies and research reporting, which influence the future value of research, probably ought to be undertaken routinely as well to inform these deliberations. A number of participants suggested that this type of information, required to make an assessment at point 5 of the checklist, ought to be collected at a much earlier stage in the appraisal process – ideally at scoping.

One model for an efficient order of assessment would be to consider points 1–5 routinely (possibly with information required for point 5 collected at scoping). The Appraisal Committee would then be in a position to either rule out OIR or AWR and issue guidance in the usual way or indicate in the ACD that OIR or AWR was provisionally recommended subject to advice from a Research Advisory Committee and subsequent analysis to support an assessment of points 6 and 7 of the checklist at FAD. This model would avoid unnecessary analysis and assessment and incorporate the judgements of the research community without necessarily adding to the time that an appraisal might take. However, it does pose some questions about who will conduct the additional work between ACD and FAD and whether this is consistent with current contractual arrangements with the TAR groups or whether this work might be undertaken by the Decision Support Unit. This is a matter that NICE and NIHR are best placed to resolve.

Assessing the prospects of research

When considering OIR or AWR guidance there must be some assessment of (1) the type of research needed to address the key uncertainties, (2) whether or not this will be regarded as ethical and can be undertaken while the technology is approved for use, (3) whether or not it is likely to be a priority for public funding and be commissioned and (4) when it is likely to report.

Although the NICE appraisal process may be well suited to identifying the need for evidence when assessing cost-effectiveness, these other critical assessments (the type of research and its priority) are not necessarily ones for which NICE and its Advisory Committees, as currently constituted, have particular expertise, not least because they reflect the decisions of those responsible for research design, prioritisation and commissioning. Without sufficient co-ordination between these communities there is a danger that OIR or AWR guidance could be issued when the type of research required would either not be regarded as ethical or feasible or not of sufficient priority compared with other competing research needs to be commissioned. Because publically funded research is also budget constrained, it is perfectly possible that research that might be valuable from a wider NHS perspective might nevertheless not be a priority if other more valuable research might be displaced. This might be a particular concern if there is a possibility that the research could be undertaken by the manufacturer rather than displacing other research without a commercial interest. Therefore, a decision of whether OIR or AWR research should be undertaken by the manufacturer or through publically funded research is one that NICE cannot properly take alone (see Incentives for evaluative research).

Some judgement about how the research community might respond to OIR or AWR recommendations when NICE is formulating guidance is clearly possible, but more informed judgements and better decisions are likely to be possible through greater involvement of the research community. For example, a Research Advisory Committee could be constituted that could consider provisional OIR or AWR guidance (at ACD), making recommendations about the type of research needed and its ethics, feasibility and likely priority during the consultation period before final appraisal and guidance. It might also make recommendations about whether research should be publically funded or undertaken by the manufacturer with appropriate contractual arrangements (which may require the involvement of the Department of Health at some stage). Such recommendations might be informed by the type of analysis described in Chapter 5, Point 6: Are the benefits of research greater than the costs? and Point 7: Are the benefits of approval greater than the costs? and discussed in Incentives for evaluative research above. There are of course many different ways in which greater co-ordination might be achieved. However, as some of the assessments that must be made in formulating OIR or AWR guidance are, in fact, research decisions that fall outside the NICE remit, it would seem sensible to draw on the expertise of those involved in, and responsible for, these types of research decisions to help make these assessments, for example translating the need for particular types of evidence into a need for particular types of research, its cost, relative priority, likelihood of success and when it is likely to report. There was a general consensus among workshop participants that greater involvement of research commissioners when NICE is considering AWR or OIR recommendations would be useful. However, the finer detail of how the process of appraisal might need to be amended to accommodate this is for NICE to consider and may require an update to the methods and process of technology appraisal.

Reappraisal

A number of participants pointed out that the triggers for reappraisal need to be linked to OIR and AWR guidance in two respects: (1) to ensure that guidance is reconsidered once research findings are available and (2) to reconsider guidance if the type of research anticipated as part of AWR or OIR guidance is not undertaken and there is little prospect that it will be, for example when there is no agreement with a manufacturer or publically funded research was not prioritised or commissioned. In addition, other changes (prices, entry of new technologies and new evidence) that might mean that research is no longer required ought to be included as criteria for reappraisal, to be considered by NICE when prioritising reappraisals. This suggests that the criteria for reappraisal and the process by which they are judged by NICE might need to be amended. Some consideration also needs to be given to the unintended consequences of reconsidering OIR before sufficient time has elapsed to encourage manufacturers to undertake research or publically funded research to be prioritised and commissioned.

Population net health effects

In current NICE appraisal cost-effectiveness is most commonly summarised in terms of ICERs. However, such summary measures give little indication of the scale of the expected NHEs for current and future patient populations, which is a key assessment when considering OIR and AWR. For this reason cost-effectiveness was presented in terms of NHEs per patient treated and for the population of patients over time, which provides information in a way that is directly relevant to the assessments that need to be made (especially at points 2 and 7 for the checklist – see Chapter 5, Point 1: Is it expected to be cost-effective?). All of the information required to express expected cost-effectiveness in these ways is already available during appraisal and they are entirely equivalent to the more familiar ICER.

Irrecoverable costs and time horizons

Amending how irrecoverable capital costs are incorporated into the estimates of expected costs poses few technical difficulties. Some of the judgements required are already made in current methods of appraisal, for example the patient time horizon is specified in existing economic models and the expected lifetime of equipment is assumed when annuitising capital costs. The technology time horizon is not currently explicitly considered and some judgement by the Appraisal Committee supported by sensitivity analysis as well as expert clinical views is likely to be required. Considering the significance of the irrecoverable opportunity costs of initially negative NHEs of a technology (the investment profile) is more nuanced (see Chapter 3, Issues specific to OIR recommendations) and their precise effect will depend on assessments made at points 5 and 6, which can be supported by additional analysis using the type of economic models developed during appraisal. However, some early indication of their potential importance can be based on their scale relative to expected NHEs, the point at which initial losses are expected to be compensated by later gains, whether or not treatment decisions are reversible and what opportunities to improve health might be forgone by a delay to initiating treatment.

Assessing the consequences of uncertainty

The judgement at point 3 of the checklist requires some assessment of (1) how uncertain a decision based on expected cost-effectiveness might be and (2) what the consequences, in terms of population NHEs, are likely to be if an incorrect decision is made. The methods of analysis presented in Chapter 5, Point 3: Does more research seem worthwhile? attempt to decompose this assessment into a series of steps, each presenting what is already available within current methods of appraisal but in ways that can more directly inform the assessment required. Unfortunately, a simple proxy measure of the expected consequences based on expected costs and effects is not useful because it provides neither an upper or a lower bound, that is, it may seriously under- or overestimate what might be gained through further research. However, the information required to estimate the expected consequences of uncertainty and decompose them (see Figure 10) is already available in existing probabilistic analysis but is generally unused. Such estimates do require some judgement (implicit in all research decisions) about the period over which research findings will continue to be useful. Again, such judgements can be supported by sensitivity analysis as well as expert views, for example from a Research Advisory Committee. The implications of scenarios and other sources of uncertainty are discussed below.

What type of evidence is required?

This assessment at point 4 of the checklist requires judgements about (1) how important particular types of parameters (inputs to the economic model) are to estimates of cost and QALYs, (2) what values these parameters would have to take to change a decision based on expected cost-effectiveness, (3) how likely is it that parameters might take such values and (4) what would be the consequences if they did, that is, what might be gained in terms of population NHEs if the uncertainty in the values of these parameters could be immediately resolved. The methods of analysis presented in Chapter 5, Point 4: Is research possible without approval? decompose this assessment into this series of steps, presenting in turn what is already available within current methods of appraisal but in ways that more directly inform the assessment required. It is only when assessing the consequences of uncertainty associated with particular parameters that additional analysis (using the results of existing probabilistic analysis) is required to provide quantitative estimates. There are circumstances when this additional computation is a significant burden although, increasingly, suitable simplifications and approximations are available.

Uncertainty between scenarios

It was recognised by participants that uncertainty in the parameters included in probabilistic analysis generally do not represent all sources of uncertainty. Commonly there is also uncertainty between alternative assumptions or judgements that might be made, often represented by alternative scenarios. How the consequences of uncertainty between as well as within scenarios can be presented was explored in Chapter 5, Point 3: Does more research seem worthwhile? and Point 4: Is research possible without approval?. Properly accounting for this source of uncertainty requires ‘weights’ (probabilities) representing the credibility of each scenario to be applied directly to the simulated output from probabilistic analysis. Although this does not require additional simulation and is quick and easy to implement, it does require that either the probabilities are made explicit in advance or estimates are presented for a range of probabilities that might represent the judgement of the Appraisal Committee following deliberation. Rather than make alternative assumptions and present extreme scenarios, formal elicitation of the judgement of clinical experts about the unknown parameters for which assumptions are required is possible. Such elicitation may provide a richer characterisation of uncertainty than weighting alternative scenarios but would require relevant experts to be identified in advance and the Appraisal Committee to accept their judgements. Alternatively, this type of analysis might be possible in real time during Appraisal Committee meetings using recent initiatives such as the Transparent Interactive Decision Interrogator, so the quantitative implications of the judgements of the Appraisal Committee could be represented. 132

Additional information

The current appraisal process generally already provides the information and much of the analysis required to complete all of the analysis reported in Chapter 5 and as a consequence requires little additional resource. However, the information required to assess whether or not other sources of uncertainty will resolve over time (point 5 on the checklist) is not commonly sought as part of NICE appraisal. This information includes (1) likely changes in prices of the technology and its comparators (e.g. patent expiry and likely generic prices), (2) the emergence of new technologies that might make existing ones obsolete or change their cost-effectiveness and (3) other relevant research reporting. A number of potential sources of information and evidence were examined to inform this assessment for each case study (see Chapter 5, Point 5: Will other sources of uncertainty resolve over time? and Appendix 4 for full details of the sources and searches conducted). However, many potentially useful sources were proprietary or public access was restricted, making it surprisingly difficult to inform these assessments from the information that is currently publically available. When information and estimates were available they were often not complete or directly relevant to a UK context. NICE may need to consider how ERGs, TAR groups and manufacturers can be provided with access to this type of information or whether the Institute should extract this type of information itself at an early stage of appraisal, for example at scoping or topic selection, as suggested by a number of participants.

Who should conduct the analysis?

Whether or not the ERG/TAR group or the manufacturers should be primarily responsible for conducting any additional analysis to inform these assessments was a question raised by a number of participants. Some were concerned that this analysis should be undertaken independently and others that ERGs might not have sufficient resources and time to properly interrogate submissions that include such analysis. Because the STA process bases appraisal primarily on manufacturers’ submissions, any additional analysis would need to be included in the submissions and be reviewed by the ERG. Although the time and resource implications of the additional analysis are limited, even if all of the quantitative analysis in Chapter 5 was undertaken (most is already required but is sometimes presented in different forms), more explicit consideration of OIR and AWR and the link to price would make the critique of how uncertainty and its consequences have been characterised more important. An assessment of whether or not the point estimate of cost-effectiveness is reasonable is inevitably a more limited task than also assessing whether or not the uncertainty surrounding that assessment is credible. Any additional burden on TAR groups, ERGs and manufacturers might be eased with clear guidance on the details of how analysis should be conducted and presented and what common assumptions are deemed reasonable (e.g. time horizons) and the provision of additional information by NICE. Others also suggested that greater opportunities for iteration between ERGs and manufacturers might be helpful as well as considering only points 6 and 7 on the checklist when NICE has recommended AWR or OIR at ACD and after advice from a Research Advisory Committee (see Implications of patient access schemes and value-based pricing).

Implementing changes informing assessments

Although it was recognised that most of the analysis presented in Chapter 5 was appropriate and by and large simply used existing analysis to more directly inform the assessment required (consequently with limited resource implications), a number of participants felt that this might not be self-evident to Appraisal Committee members, ERGs/TAR groups and manufacturers. They suggested that some ‘training’ activities might be considered to ensure that appropriate skills were in place and that any additional analysis or change in the way that cost-effectiveness is presented is properly conducted and interpreted appropriately.

Policy papers

1. 1. Cooksey D. A review of UK health research funding. London: The Stationery Office; 2006.

2. 2. Centers for Medicare & Medicaid Services. National coverage determinations with data collection as a condition of coverage: coverage with evidence development. 2006. URL: www.cms.gov/Medicare/Coverage/Coverage-with-Evidence-Development/index.html.

3. 3. House of Commons Health Committee, National Institute for Health and Clinical Excellence. Report no. HC 27–1. London: The Stationary Office; 2008. Chapter 6.

4. 4. Mechanic R, Zinner D. Optimizing information under coverage with evidence development. Health Industry Forum Policy Brief. 24 October 2007. URL: http://healthforum.brandeis.edu/meetings/materials/2007-24-Oct./Policy-Brief.pdf.

5. 5. NICE Citizens Council. Only in research. London: NICE; 2007.

6. 6. NHS Quality Improvement Scotland. Coverage with evidence development in NHS Scotland. Discussion paper. 2008. URL: www.Healthcareimprovementscotland.org/previous_resources/hta_report/evidence_development.aspx.

7. 7. Office of Fair Trading. The pharmaceutical price regulation scheme. An OFT market study. London: OFT; 2007. Chapters 4 and 5.

8. 8. Office for Life Sciences, HM Government. Life sciences blueprint. A statement from the Office for Life Sciences. July 2009. URL: www.medilink.co.uk/Libraries/connect/Life-sciences-2009-blueprint.sflb.ashx.

Discussion and application papers

1. 9. Medicare’s coverage with evidence development: a policy-making tool in evolution. J Oncol Pract 2007;3:296–301.

2. 10. Briggs A, Ritchie K, Fenwick E, Chalkidou K, Littlejohns P. Access with evidence development in the UK: past experience, current initiatives and future potential. Pharmacoeconomics 2010;28:163–70.

3. 11. Carino T, Sheingold S, Tunis S. Using clinical trials as a condition of coverage: lessons from the National Emphysema Treatment Trial. Clin Trials 2004;1:108–21.

4. 12. Chalmers R. Addressing uncertainties about the effects of treatments offered to NHS patients: whose responsibility? J R Soc Med 2007;100:440–1

5. 13. Chalkidou K. ‘Only in research’: a polite ‘no’ or a valuable policy option? Birmingham: NICE; 2006.

6. 14. Chalkidou K, Hoy A, Littlejohns P. Making a decision to wait for more evidence: when the National Institute for Health and Clinical Excellence recommends a technology only in the context of research. J R Soc Med 2007;100:453–60.

7. 15. Chalkidou K, Lord J, Fischer A, Littlejohns P. Evidence-based decision making: when should we wait for more information? Health Aff 2008;27:1642–53.

8. 16. Dhalla IF, Garner S. Perspectives on the National Institute for Health and Clinical Excellence’s recommendations to use technologies only in research. Int J Technol Assess Health Care 2009;25:272–80.

9. 17. Garrison L, Sullivan S, Carlson J, Bresnahan B, Carlson R, Neumann P, et al. Performance-based risk-sharing reimbursement agreements for new medical products. Washington, DC: University of Washington; 2008.

10. 18. Kary W, Bergthold L, Talbott M, Aubry W. Coverage for evidence development: a conceptual frame. Baltimore, MD: Center for Medical Technology Policy; 2009.

11. 19. Macdonald F. Coverage with evidence development: a personal, industry view. Presentation slides. 19 September 2008. URL: www.docstoc.com/docs/105704904/Coverage-With-Evidence-Development.

12. 20. McCabe CJ, Stafinski T, Edlin R, Menon D. Access with evidence development schemes: a framework for description and evaluation. Pharmacoeconomics 2010;28:143–52.

13. 21. Menon D, McCabe CJ, Stafinski T, Edlin R, signatories to the Consensus Statement. Principles of design of access with evidence development approaches: a consensus statement from the Banff Summit. Pharmacoeconomics 2010;28:109–11.

14. 22. Miller FG, Pearson SD. Coverage with evidence development: ethical issues and policy implications. Med Care 2008;46:746–51.

15. 23. Mohr P, Tunis SR. Access with evidence development: the US experience. Pharmacoeconomics 2010;28:153–62.

16. 24. Hutton J, Trueman P, Henshall C. Coverage with evidence development: an examination of conceptual and policy issues. Int J Technol Assess Health Care 2007;23:425–35.

17. 25. Pearson SD, Miller FG, Emanuel EJ. Medicare’s requirement for research participation as a condition of coverage: is it ethical? JAMA 2006;296:988–91.

18. 26. Stafinski T, McCabe C, Menon D. Funding the unfundable: mechanisms for managing uncertainty in decisions on the introduction of new and innovative technologies into healthcare systems. Pharmacoeconomics 2010;28:113–42.

19. 27. Towse A, Garrison LP. Can’t get no satisfaction? Will pay for performance help?: toward an economic framework for understanding performance-based risk-sharing agreements for innovative medical products. Pharmacoeconomics 2010;28:93–102.

20. 28. Trueman P. Coverage with evidence development: applications and issues. Int J Technol Assess Health Care 2010;26:79–85.

21. 29. Tunis SR, Pearson SD. Coverage options for promising technologies: Medicare’s ‘coverage with evidence development’. Health Aff 2006;25:1218–30.

22. 30. Tunis SR, Chalkidou K. Coverage with evidence development: a very good beginning, but much to be done. Commentary to Hutton et al. Int J Technol Assess Health Care 2007;23:432–5.

23. 31. Tunis S, Whicher D. The National Oncologic PET Registry: lessons learned for coverage with evidence development. J Am Coll Radiol 2009;6:360–5.

Traditional literature searches

Grey literature searches

Results

Websites searched for policy documents