Better guidelines for better care: accounting for multimorbidity in clinical guidelines structured examination of exemplar guidelines and health economic modelling

Systematic examination of how guidelines account for multimorbidity

The Boyd et al. 2 and Hughes et al. 3 studies referred to above are based on carefully selected vignettes, chosen to emphasise the more extreme implications of following guideline recommendations in people with multimorbidity. However, more systematic examinations of published CGs also show they at best only partly account for multimorbidity. Lugtenberg et al. 37 examined 20 published guidelines for four common conditions (COPD, depressive disorder, type 2 diabetes and osteoarthritis). The study found that, although 85% made some mention of comorbidity and there was a mean of three treatment recommendations with a specific variation in the presence of a comorbidity, most such variations were in relation to single concordant15 comorbidities (e.g. a glycaemic control treatment recommendation in people with acute MI). Vitry and Zhang6 examined 17 Australian CGs and found that 53% made at least one specific recommendation for people with one comorbid condition, but only 12% did the same for people with several. 6 The time required to benefit from treatment in the context of limited life expectancy was discussed in only 18%, and treatment burden in 29%. A study of 16 Canadian guidelines using the same methods had similar findings, although more guidelines (94%) made at least one specific recommendation for people with one comorbid condition. 7 Cox et al. 38 examined 14 Canadian guidelines with similar results in terms of the proportion of guidelines making at least one ‘comorbidity’ recommendation. Twelve of the 14 guidelines also provided one or more age-varied recommendations for those aged 65–79 years, and five for those aged ≥ 80 years. However, of the total 1189 recommendations made, only 33 (2.8%) and 7 (0.6%) were actually age varied for these two age groups, consistent with a rather more limited accounting for multimorbidity than implied by counting any mention of comorbidity in an entire guideline.

Single-disease guidelines explicitly designed to account for multimorbidity

A small number of guidelines have been specifically created to account for multimorbidity, with two broad types. The first type provides general guidance on how to manage people with multimorbidity. The American Geriatrics Society (AGS) guiding principles for the care of older people with multimorbidity is the most comprehensive published example,39–41 and the NICE multimorbidity CG that is in development also takes this approach. 42 However, the focus of this project is accounting for multimorbidity in single-disease guidelines, so these are not directly relevant. The second type is single-disease guidelines that explicitly aim to account for comorbidity, notably the California HealthCare Foundation and International Diabetes Federation guidelines on diabetes care in older people,43,44 and the NICE guideline on the management of depression in adults with a chronic physical health problem. 17

The California HealthCare Foundation guideline for diabetes care in older people has several features that distinguish it from whole-population diabetes guidelines. It more explicitly considers the applicability of trial evidence to older people, and clearly states that many recommendations should be applied to all or most older people. However, it specifically varies some recommendations in people with significant comorbidity, frailty or reduced life expectancy because of consensus in the GDG that benefits are less likely to be realised and harms are more likely to happen. Finally, it also includes specific reference to geriatric syndromes such as falls and urinary incontinence because these are common in older people and may be precipitated or exacerbated by diabetes treatment. 43 For example, based on consensus, the recommendation in relation to the treatment target for glycated haemoglobin is:

For older persons, target hemoglobin A1c (A1C) should be individualized . . . For frail older adults, persons with life expectancy of less than 5 years, and others in whom the risks of intensive glycaemic control appear to outweigh the benefits, a less stringent target such as 8% is appropriate.

California HealthCare Foundation43

The International Diabetes Federation guideline takes this stratification of recommendations one step further, by making distinct recommendations for older people who are functionally independent, who are functionally dependent (with impairments in activities of daily living, with further variation for two subgroups of the functionally dependent: people who are frail and people with dementia) and who are at the end of life. Using the example of glycated haemoglobin again, the recommended target levels are 7.0–7.5% for the functionally independent, 7.0–8.0% for the functionally dependent without frailty or dementia, ≤ 8.5% for people with frailty or dementia, and treating only if symptomatic at end of life. In addition, recommended treatments are varied, with increasing emphasis on avoiding hypoglycaemia as frailty increases and life expectancy decreases. 44

In contrast, the NICE guideline on depression in adults with a chronic physical health problem17 largely copies recommendations from the single-disease depression guideline, which was developed in parallel, although with several important modifications. 45 Recommendations are varied where the evidence in people with a chronic physical problem is different (e.g. some psychological interventions are not recommended) and physical condition treatment recommendations are added where there is some evidence of impact on depression (e.g. peer support focused on physical condition). The guideline also provides much more detailed advice on drug–drug interactions, reflecting the fact that coprescribing of physical health drugs will be much more common, with risks of adverse drug events (ADEs).

A feature of all three of the modified single-disease guidelines is that general-population recommendations are varied for important subgroups (older people, people with physical comorbidities). This is either because there was an available evidence base in the subgroup, which informed variation in recommendations, or more commonly because it was the considered consensus judgement of the GDG that a variation was appropriate. However, although these guidelines provide models for how recommendations might be varied, they do not clearly state a more general process by which a GDG could decide when and how it might be appropriate to vary or qualify recommendations.

Published guidance on how to account for multimorbidity in guidelines

Since the original proposal was written, two papers have been published addressing how single-disease guidelines could account for multimorbidity. In the context of respiratory disease, Fabbri et al. 46 suggested that guideline developers needed to:

1. explicitly consider whether or not the trials underpinning treatment recommendations included people with the most common comorbidities, and if there is usable information from subgroup analyses

2. use expected absolute risk reduction (ARR) to inform treatment decisions, or at a minimum consider the range of variation in absolute benefit based on variation in baseline risk and competing risk of death from other causes

3. specify the actual outcomes that each therapy improves, since individual patients may vary in their preferences for which outcome matters most to them

4. present data on the average and extremes of length of therapy required to achieve expected absolute benefit, because ‘time to benefit from therapy’46 is essential in decision-making in patients with competing risks, who may have reduced life expectancy

5. address interactions that are common or important given the prevalence of specific comorbidities. 46

More recently, Uhlig et al. 47 proposed a more comprehensive framework for guideline development to account for multimorbidity, which specifically identified when during guideline development these issues should be addressed and how the strength of recommendations might vary as a result. 47

Of interest is the considerable overlap between both of these frameworks and the work that was proposed for this study, including in relation to applicability of evidence, interactions, the importance of absolute benefit to compare across guidelines, and the temporal dimension of benefit in the context of short life expectancy. A key difference is that both papers take the form of general guidance and do not examine how these elements could be feasibly integrated into guideline development. For example, time to benefit (TTB) is a very appealing idea, but is difficult to operationalise meaningfully. The focus of the Better Guidelines project was therefore on how existing guideline development could be practically altered to address the problems identified by previous research.

The National Institute for Health and Care Excellence guideline development process

Although GDGs have considerable discretion in how they interpret evidence and what recommendations they write, NICE requires them to work according to a well-defined process specified in the Guidelines Manual:48

1. Topics are referred to NICE by stakeholders such as the English Department of Health, or may be identified internally in terms of existing guidelines requiring updating.

2. A draft guideline scope is created and finalised after stakeholder consultation. For guideline updates, the scope may cover the whole guideline or only selected aspects of it. The populations and settings that will be covered are defined by the scope, which also identifies the key issues and questions that will be considered.

3. A suitably representative GDG is convened, which agrees the clinical research questions (CRQs) required to cover the scope and which CRQs will be prioritised for new economic modelling to examine cost-effectiveness. Relevant evidence for these questions is systematically searched for and synthesised, and additional evidence from stakeholders is sought if required. Evidence syntheses are discussed by the GDG, which uses a deliberative and consensus process to formulate recommendations that are structured and worded based on a common framework. Recommendations can be both for care (diagnosis, treatment, follow-up, service organisation) and for future research to inform future guidelines.

4. A draft guideline is published for stakeholder consultation and amended as required before final publication. There are a number of versions of guidelines produced, and the format of these has changed over time. Currently, NICE publishes both a NICE guideline and a full guideline. The NICE guideline essentially lists the recommendations and priorities for implementation. The full guideline is a much larger document and details the evidence and the GDG discussion that connects evidence to the recommendations made. Historically, a quick reference guide was also usually published, but this has since been replaced with a web-based NICE pathway. The NICE guideline and the quick reference guide or NICE pathway are the documents that clinicians and patients are most likely to use, with the full guideline serving as a reference document detailing how the guideline was created.

Three exemplar guidelines

Analysis focuses on exemplar NICE guidelines for type 2 diabetes (CG6649 and its partial update CG8750), depression in adults (CG9045 and the accompanying guideline for depression in adults with a chronic physical health problem CG9117) and chronic heart failure (CG10851). Where necessary, other guidelines are also examined, as for the interactions work below, which required a broader perspective, and for some of the economic modelling, where we were constrained by model availability. We chose these three conditions because they:

• are individually important because they are common and are associated with a large burden of population morbidity and health service resource use

• are commonly comorbid with each other, and people with each condition commonly have a range of other comorbidities in addition (Figure 4)

• had a recent NICE guideline with economic modelling carried out for at least some of the individual treatment recommendations

• included both physical and mental health conditions where co-occurrence is known to worsen outcomes of both conditions,52 and where there are some published trials of the effectiveness of treatment of physical and mental health conditions in the presence of the other53,54

• included a physical condition where treatment is primarily long-term and preventative (type 2 diabetes) and one where life expectancy is much reduced but treatment has a major benefit over short periods [heart failure, particularly moderate/severe left ventricular systolic dysfunction (LVSD)].

FIGURE 4.

Common comorbidities of the three index conditions compared with other conditions. Source: based on data from the study reported by Barnett et al. 13 TIA, transient ischaemic attack.

Structure of this chapter

This chapter describes initial work examining the NICE guidelines for the three exemplar conditions, in terms of:

• examining and cross-referencing CRQs and treatment recommendations

• quantifying the extent to which guideline development is informed by any economic evidence, and specifically by new economic analysis

• quantifying the potential for drug–drug and drug–disease interactions in people with multimorbidity.

The background, methods and findings for each of these is presented in turn with a short summary, and the implications of all the findings are discussed at the end of the chapter.

Background

As described in Chapter 1, a number of commentators have identified that multimorbidity poses challenges to guideline developers,4,5 and several studies have described situations where cumulative guideline recommendations might be irrational or troublesome. 2,3 More systematic examination of guidelines from a range of international sources has shown that, although most guidelines do qualify at least some recommendations for people with comorbidity, this is relatively infrequent and usually relates to concordant comorbidity, although some guidelines account at least partly for comorbidity by making recommendations for older people, in whom comorbidity will be more common. 6,7,37,38 However, guideline developers may be reluctant to routinely make age-qualified recommendations in order to minimise unconsidered non-treatment of older people, particularly given that age is now a protected characteristic in the UK under the Equality Act 2010. 55

We therefore systematically examined treatment recommendations made by guidelines for the three exemplar conditions, specifically in relation to:

• the extent to which comorbidity or specific age groups were accounted for in the CRQs

• the extent to which statements about the recommended treatments accounted for comorbidity or specific age groups

• the extent to which the research recommendations that were made accounted for comorbidity or specific age groups.

Methods

We examined relevant guideline documents for our three exemplar conditions as previously described, including those publicly available on the NICE website and (in the case of the CRQs) internal NICE documents listing these, since they are not explicit in all published guideline documents. Although in principle it is straightforward to count how often CRQs or recommendations accounted for comorbidity, several problems were encountered in practice. First, guidelines are evolving documents, which are often partially rather than fully updated. This means that at any one moment a guideline may contain unchanged recommendations copied in from a previous version and recommendations amended or created in the light of new evidence. Mapping which CRQs are relevant to any one guideline is therefore not always straightforward. For example, the CG87 type 2 diabetes guideline was a partial update of CG66, meaning that many recommendations in it are copied in from CG66. The CRQs for six guidelines were therefore examined (CG66 and CG87 for type 2 diabetes; CG90 and CG91 for depression; and CG05 and CG108 for heart failure). Of note is that the two depression guidelines were unusual in that they were commissioned at the same time and developed in parallel. It therefore makes no real sense to consider them separately because CG90 (depression in adults) effectively accounts for comorbid physical health problems even if it never explicitly mentions them because it cross-references CG91 (Depression in Adults with a Chronic Physical Health Problem,17 which to our knowledge is the only published NICE guideline specifically addressing commonly occurring comorbidity). Second, counting how often guidelines account for comorbidity is complicated because it is not always clear what constitutes a recommendation. NICE ‘recommendations’ are numbered, but any single numbered piece of text may contain a number of separate statements recommending different aspects of care, and recommendations for the use of any single treatment (e.g. metformin in type 2 diabetes) may be defined across multiple numbered recommendations. For the purposes of this analysis, we therefore examined recommended treatments defined as pharmacological and non-pharmacological interventions of various kinds, and considered whether or not comorbidity was accounted for in these treatments across all relevant numbered recommendations.

Similarly to previous studies,6,7,37,38 three reviewers first examined the three most recent guidelines for the exemplar conditions (CG87, CG108, CG90) to classify text into discrete recommendations, and classified these recommendations in terms of whether or not they related to treatment. For each recommendation, the same reviewers then made a judgement about:

• whether or not comorbidity was addressed in the way the recommendation was worded:

  • without cross-reference to other NICE guidance and unlinked to a recommended treatment

  • by cross-reference to other NICE guidance and unlinked to a recommended treatment

  • in relation to a recommended treatment

• when comorbidity was addressed in the way the recommendation was worded, the proportion of times that this was in relation to a concordant comorbidity as defined by Piette and Kerr15

• whether or not use of a recommended treatment in older people was addressed. Although age and comorbidity are not the same constructs, comorbidity is much commoner in older people, so we believed it relevant to consider qualifications by age as at least partial proxies for comorbidity38

• whether or not use of a recommended treatment was addressed in relation to life expectancy. Although life expectancy can be shortened by single conditions, multimorbidity is commonly perceived to be important partly because it is associated with reduced life expectancy independent of single-condition severity46,47

• explicitly contradictory treatment recommendations, which we defined as the same treatment stated to be indicated in one guideline and contraindicated in another, and explicitly synergistic treatment recommendations, which we defined as the same treatment stated to be indicated in more than one of the three guidelines. This is examined across more guidelines in the interactions analysis later in this chapter

• synergistic treatment recommendations.

Findings

Summary

All three guidelines had a number of specific qualifications in treatment recommendations for people with comorbidity. There were eight such examples in the type 2 diabetes guideline, of which four were for concordant conditions, and seven in the heart failure guideline, of which five were for concordant conditions. All of the treatment recommendations in the main depression guideline (CG90) were effectively qualified by the presence of a chronic physical health problem by cross-reference to CG91, and there were seven other specific examples, of which only one was for a concordant condition (see Table 1).

The diabetes guideline had three treatments where the recommendation to use was qualified for older people (all in relation to the risk of hypoglycaemia with some drugs). The heart failure guideline had two qualified treatment recommendations (one providing general advice about dosing and adverse effects of recommended drugs in older people, the other emphasising that age was not a restriction to beta-blocker use). The depression guideline had two qualified treatment recommendations (one providing general advice about dosing of recommended drugs and interactions related to physical health conditions in older people, the other relating to risks of electroconvulsive therapy and associated anaesthesia). None of the recommended treatments was qualified in relation to life expectancy (although the heart failure guideline stated that palliative care was central to the care of many people with heart failure). There were no explicitly contradictory recommendations across the three guidelines. There were two recommended treatments where the diabetes and heart failure guidelines were considered to be synergistic in that both recommended the same drugs, with closely related beneficial outcomes in at least a subset of people with each condition.

Background

Methods

The use of economic evidence within guideline development was evaluated by one researcher (AT), who searched systematically through the published full guideline versions of CG66 (type 2 diabetes),49 CG90 and CG91 (depression)17,45 and CG108 (heart failure)51 to identify CRQs and match them to relevant sections of the full guidelines and economic plans. Three strategies were used to establish the extent of the economic evidence used. This was necessary because, although CRQs are central to guideline development, the published full guidelines are not necessarily explicitly structured in the same way. The full guideline layout has also changed over time and varies between the national collaborating centres that produce them:

1. For each CRQ, key terms contained in the CRQ were used to search within the full document. Once the key term had highlighted the relevant section within the guideline discussing the relevant CRQ, the associated section within the guideline was then screened manually and the nature of the economic evidence available was recorded and categorised.

2. Each guideline was examined to identify use of variants of the term ‘cost-effectiveness’ or ‘health economics’ until all the findings within the guideline had been exhaustively identified. Classifications generated from this approach were based around guideline chapters.

3. The referenced list of economic evidence was retrospectively manually mapped to the CRQs.

The lack of consistency between the CGs and the need for three strategies to extract the relevant data means that the findings are only indicative in terms of how quantifiable they are, since it was not possible to fully recreate the process by which economic evidence was used.

Findings

The number of CRQs ranged from 11 in CG91 to 40 in CG66 (Table 2). In the two depression guidelines, 26% and 27% of CRQs were supported by at least some economic evidence, compared with 53% and 92% of CRQs in the type 2 diabetes and chronic heart failure guidelines. A much smaller proportion of CRQs were supported by producing a de novo economic model. This at least partly reflects the resources available to NICE in terms of health economics capacity and expertise, in the context of having to generate answers to the substantial number of CRQs underlying most CGs. De novo modelling was available for 9% and 11% of depression CRQs, compared with 20% of diabetes CRQs and 8% of heart failure CRQs.

Summary

The extent and type of economic evidence used were hard to quantify systematically because of the variability between how a CRQ was specified and how it was reported or made explicit in the full guideline reporting economic evidence. However, systematically identifying when and how economic evidence was used was helpful for generating a qualitative understanding of the process. Half (51%) of all the CRQs examined were not informed by any economic evidence, although this varied considerably across guidelines (range 8–74%), and only a minority (14.6%) of CRQs were informed by de novo economic modelling, again with variation between guidelines (range 8–20%). This will partly reflect the quality and scope of the clinical evidence, and partly the resources available to the GDG to create de novo CEA models.

With regard to multimorbidity, the development of CGs follows the way that health care and health-care research are currently organised in relatively discrete disease-specific silos. 13 Economic evaluation, and methods of model-based CEA, as subdisciplines of health economics, have also evolved within the biomedical model of disease and thus share preferences for probabilistic evidence sourced from systematic reviews, meta-analyses and RCTs as well as a focus on populations with a single index condition. Therefore, model-based CEAs commonly address one particular physiological condition rather than a decision problem that involves a patient with multiple morbidities. This review has confirmed that current uses of economic evidence are unlikely to routinely support guideline development that accounts for multimorbidity.

Background

As described in Chapter 1, the prevalence of polypharmacy and potentially serious drug–drug interactions has risen considerably since 1995,32 with evidence that ADEs have also become more common. 34 The harm from ADEs is considerable, with 6.5% of emergency hospital admissions in the UK being attributable to ADEs, of which about half are judged preventable. 70 A study of older people in Italy found that 31.5% of reported ADEs in older people were judged to be caused by drug interactions, with 35.9% of ADEs being severe enough to require hospitalisation. 71 The drugs associated with ADEs causing hospital admission are typically those that guidelines recommend for common conditions such as hypertension, diabetes and CHD, reflecting the fact that most harm is caused by commonly prescribed drugs with relatively low risk rather than rarely prescribed drugs with high risk. 72 However, the observed increases in interactions and ADEs are not necessarily inappropriate, since not all ADEs are predictable (e.g. an anaphylactic reaction in a patient not known to be allergic to a drug), and increases in drug-related harm could in principle be more than balanced by the benefit of prescribing that drug. 32 The appropriate balancing of benefits and harms including consideration of drug–disease and drug–drug interactions has been identified as a key element of optimal care for older adults with multimorbidity. 40

We consider that it would be difficult for CGs to list all possible drug–drug interactions in a helpful way, and, in the UK at least, such a list would also duplicate information available elsewhere or embedded in GP electronic prescribing systems. NICE guidelines usually make this clear; for example, the heart failure guideline says: ‘It is not possible in the development of a CG to complete extensive systematic literature reviews of all pharmacological toxicity. NICE expect the guidelines to be read alongside the Summaries of Product Characteristics’ (p. 24). 51 However, most guidelines do contain comment about some interactions, although it is not always clear why these have been selected.

The study reported in this section therefore aimed to quantify how often the drugs recommended by NICE guidelines for the three exemplar conditions have drug–disease interactions in the presence of other commonly comorbid conditions, or have potentially serious drug–drug interactions with drugs recommended by guidelines for these conditions. The findings of the study were published in the British Medical Journal in 2015, and the findings and figures are reported here under the terms of the Creative Commons Attribution (CC-BY) open access licence that applies, which allows free sharing and adaptation of material for any purpose. 73

Methods

Our starting point was single-disease guidelines for heart failure,56 type 2 diabetes74 and depression. 75 We then selected nine other NICE guidelines to examine, choosing common and chronic conditions that were commonly comorbid with the exemplar conditions, and with recently published NICE guidelines that included recommendations for the initiation of a drug treatment (Figure 5). After discussion with the PRG, the nine selected other guidelines were for atrial fibrillation,76 osteoarthritis,77 COPD,78 hypertension,79 secondary prevention following MI,80 dementia,81 rheumatoid arthritis,82 chronic kidney disease (CKD)83 and neuropathic pain. 84 These were selected on the basis of being common and important problems, for which there was recent NICE guidance available, and with varying prevalence of comorbidity for the three exemplar conditions (see Figure 5).

FIGURE 5.

Percentage of people with the three index conditions who have each of the other conditions. Morbidity data were not available for osteoarthritis or neuropathic pain; the ‘painful condition’ data shown are defined by receipt of four or more prescriptions for non-over-the-counter analgesics in the previous 12 months. Reproduced from Dumbreck et al. 73 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.

A panel of three clinicians reviewed all 12 guidelines to identify and classify recommendations about initiation of chronic drug treatments. Drugs were defined as ‘first-line’ if they were recommended as a treatment for all or nearly all people with the condition [e.g. angiotensin-converting enzyme (ACE) inhibitors for people with heart failure]. Drugs that were recommended for only some patients with the condition under some circumstances were defined as ‘second-line’ (e.g. spironolactone for people with heart failure and high levels of symptoms despite first-line treatment).

Drug–disease and drug–drug interactions were then identified and classified, and the published guidelines examined to identify if these were explicitly discussed. For each of the three exemplar index guidelines, the BNF was systematically searched to identify drug–disease warnings for guideline-recommended drugs, taking account of the pre-defined 11 conditions (the other two index conditions and the nine others). 85 Drug–disease interactions were defined as being significant if a disease was stated to be a contraindication in relation to all or most people with the condition, or if the BNF stated that drugs should be used only with caution accompanied by a clear statement to avoid in all or most people with the condition. For CKD but not for other conditions, BNF warnings frequently recommended dose adjustment, and this was additionally counted for CKD.

For drug–drug interactions, we counted those that the BNF categorised as ‘potentially serious’, which is when ‘concomitant administration of the drugs involved should be avoided (or only undertaken with caution and appropriate monitoring)’ (p. 852). 85 These interactions are defined as potentially serious based on the severity of the potential harm, not the likelihood of the interaction. Potentially serious interactions were identified between drugs recommended by each of the three index guidelines and drugs recommended by any of the 12 guidelines (since two drugs recommended in the same guideline can interact), and the nature of the harm caused was classified, with any disagreement between expert panel members resolved by discussion. The nature of harm was categorised as bleeding risk, central nervous system toxicity, cardiovascular adverse effect (including change in blood pressure, or effect on heart rate or rhythm), effect on renal function or serum potassium, or other (which included harms associated with changes in level of narrow therapeutic index drugs such as lithium carbonate, digoxin and theophylline). 86 These classification categories were chosen to reflect the types of ADEs associated with emergency hospital admission. 70

Interaction findings

Table 3 shows the number of drugs recommended as first or second line for each of the 12 guidelines, which varied from 1 to 9 and from 1 to 19, respectively. Table 4 shows how often drugs recommended for each of the three index conditions would be contraindicated by, or should be avoided in the presence of, any of the other 11 conditions. Drug–disease interactions were not common, with the exception of those related to CKD, which affected type 2 diabetes in particular. CKD was involved in 27 of the identified 32 drug–disease interactions for drugs recommended in the type 2 diabetes CG and all of the 6 and 10 drug–disease interactions for the depression and heart failure guidelines, respectively. The guidelines for type 2 diabetes and heart failure each specifically discussed just one of these identified drug–disease interactions. For type 2 diabetes, this recommendation was regarding the need to avoid thiazolidinedione treatment in people with comorbid heart failure. For heart failure, it was identified that amlodipine should be considered for the treatment of comorbid hypertension and/or angina in patients with heart failure, but verapamil, diltiazem or short-acting dihydropyridine agents should be avoided. The depression guideline did not discuss any of the identified drug–disease interactions.

Potentially serious drug–drug interactions were common (Figure 6). There were 133 potentially serious drug–drug interaction pairs identified for the type 2 diabetes guideline, of which 25 (19%) involved one of the four drugs recommended as first-line treatments (the diabetes guideline had a total of four drugs, or classes of drug, recommended as first-line treatments and 19 second-line treatments; see Appendix 1). Nine of the recommended drugs for diabetes did not have any potentially serious drug–drug interactions. For the depression guideline, 89 potentially serious drug–drug interaction pairs were identified, of which 19 (21%) involved the one drug class recommended as first line (SSRI antidepressants) rather than the 12 drugs or drug classes recommended as second-line treatments for depression. For heart failure, 111 potentially serious drug–drug interaction pairs were identified, of which 21 (19%) involved the two drug classes recommended as first line rather than the nine drugs (or drug classes) recommended as second line for heart failure.

FIGURE 6.

Number of potentially serious drug–drug interactions for each of the three exemplar guidelines. This figure has been reproduced from Dumbreck et al. 73 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.

Figure 7 summarises the types of harm expected from potentially serious drug–drug interactions by index condition (see Appendix 2 for further details). For type 2 diabetes, cardiovascular-related harm such as significant hypotension or bradycardia was the most frequent category, followed by ‘other’ (which includes increased lithium or digoxin levels causing risk of toxicity, and myopathy with statin therapy), and harms associated with renal or serum potassium. For depression, bleeding risks were the most commonly identified harm, particularly involving SSRI antidepressants recommended as first line, followed by ‘other’ harms (most commonly relating to lithium toxicity), and cardiovascular and central nervous system toxicity. The majority of cardiovascular adverse effects in the depression guideline were related to increased risk of ventricular arrhythmias. The most common potentially serious drug interactions for the heart failure guideline were for bleeding events, but also drug interactions causing severe hypotension or related to increased digoxin or lithium levels causing risk of toxicity.

FIGURE 7.

Type of harm of potentially serious drug–drug interactions for each of the three exemplar guidelines. This figure has been reproduced from Dumbreck et al. 73 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.

A very limited number of the identified drug–drug interactions were highlighted in the index guidelines. In the guideline for type 2 diabetes, only two interactions were mentioned: potassium-sparing diuretics with ACE inhibitors, and potassium-sparing diuretics with angiotensin receptor blockers (ARBs). The depression guideline highlighted only the increased risk of bleeding with SSRIs plus non-steroidal anti-inflammatory drugs (NSAIDs) or aspirin. None of the recommendations in the heart failure guideline contained an explicit discussion of the 111 potentially serious drug–drug interactions identified.

Summary

Potentially serious drug–drug interactions were found to be relatively common among guideline recommendations for each of the three index conditions and 11 other common conditions. In contrast, drug–disease interactions were found to be relatively uncommon, with the exception of interactions when an individual has comorbid CKD. The types of harm potentially introduced by coprescription of drugs varied by CG and was most commonly cardiovascular and ‘other’ for diabetes; bleeding and ‘other’ for depression; and bleeding and cardiovascular for heart failure.

Previous studies of the implications of following single-disease guidelines in people with multimorbidity have usually considered single, hypothetical patients with carefully selected multiple conditions, an approach that is likely to overstate the scale of the problem. 2,3 Using US population survey data, Lorgunpai et al. 87 estimated a much higher rate of drug–disease interactions (which they termed ‘therapeutic competition’), with one-fifth of older American adults being prescribed drugs for one condition that have the potential to worsen another. 87 However, their study included interactions that did not reach our threshold of being recommended to avoid in all or most patients. For example, the use of beta-blockers for CHD in people with COPD was common in their study, but, although it carries some risk, it is not stated as a contraindication or recommendation to avoid in the BNF, because the benefits outweigh the harms in most patients.

One key potential limitation is the use of a selection of CGs as exemplar case studies, and some other guidelines do discuss interactions in more detail. For example, NICE has produced a guideline for depression in people with a chronic physical health problem,17 which includes extensive discussion about drug interactions (although in a full guideline appendix, which will not be commonly read by clinicians), and a guideline on management of bipolar disorder, which includes detailed recommendations about safe use of lithium. However, we would not expect the pattern of findings to be substantially different for other guidelines that include a reasonable number of recommendations for chronic drug treatment. Any recommendations for commencing drugs for acute conditions were excluded from this analysis, but it should be noted that interactions with drugs such as antibiotics and NSAIDs used for short-term intercurrent illness are common and important. 33 The inclusion of additional guidelines would have further increased the number of potential interactions identified. Both of these exclusions imply that our findings are likely to be conservative.

This study systematically examined recent national guidelines produced by NICE for important and common clinical conditions, using data on interactions drawn from a single, authoritative UK source. Defining contraindications and potentially serious interactions was not straightforward, reflecting the facts that the risk of such events is often poorly quantified and information sources vary in what they rate to be significant. 88 This study used the BNF because it is the reference source used by most UK-based clinicians. The BNF draws on data from manufacturer summaries of product characteristics, NICE guidance, the medical literature and expert opinion, but other reference sources might not be consistent with this, and a databases of listed potentially serious drug interactions might have yielded different results. For example, a summary of product characteristics for amitriptyline from the online electronic medicines compendium of up-to-date, approved and regulated prescribing information for licensed medicines89 includes cardiac arrhythmias and history of MI as contraindications but these are not listed in the BNF. Of note is that the focus of this study has been negative interactions, and there will of course be situations where drug–condition interactions are positive in that one drug has benefits for more than one condition. In addition, the way in which we defined both drug–drug and drug–condition interactions prioritises interactions more strongly associated with harm. Less severe interactions will sometimes cause harm but are also much more common. 87 Our judgement was that focusing on interactions more strongly associated with harm was appropriate to help ensure feasibility in a guidelines context.

Summary of the three studies

Interpretation and implications of the three studies

How commonly is applicability of evidence a potential problem?

Several studies have shown that many or most people with a condition would not be eligible for the RCTs providing evidence of treatment effectiveness. Of a sample of 283 trials published in high-impact general medical journals, 81% excluded people with common comorbidities, 54% excluded people taking other commonly prescribed drugs and 72% excluded people on the basis of age. 94 When considering single diseases, fewer than half of people in Scotland with newly diagnosed type 2 diabetes would have been eligible for the landmark UK Prospective Diabetes Study (UKPDS) 33 and UKPDS34 studies of initial hypoglycaemic treatment. 97 Only 13–25% of older people discharged from hospital with heart failure in the USA would have been eligible for inclusion in three seminal trials of heart failure treatment. 98 A maximum of 20% of people with COPD in the community would have been eligible for the 18 key trials underlying the Global Initiative for Chronic Obstructive Lung Disease guidelines for COPD (median 5%). 99 A maximum of 43% of people with asthma would have been eligible for the Global Initiative for Asthma guidelines (median 6%). 100 Depending on the trial, only 17–71% of women with breast cancer would have been eligible for 12 major breast cancer trials that changed clinical practice. 101 Evidence of treatment benefit from RCTs is therefore often based on highly selected subgroups of people with the target condition.

Of note is that, in most such studies, what is being evaluated is exclusion due to easily measured factors such as age, and that there will be additional exclusions from less well specified criteria such as ‘any life-threatening disease (other than heart failure)’102 or ‘any factors likely to limit adherence to interventions’. 103 Such criteria are open to the recruiting researcher’s judgement and would be expected to exclude people of all ages who are sicker or less likely to adhere to treatment, or perceived to be at increased risk of harm. For example, patients in carotid endarterectomy trials who were judged unfit for surgery (i.e. were believed to have very high risk from surgery) had much worse outcomes in terms of stroke risk and death even though their baseline characteristics as measured by trial variables were not different from those judged fit for surgery. 96 Excluded patients will therefore plausibly have different baseline risks of the outcomes targeted by the trial, different risks of harm and potentially different likelihoods of benefit (although the standard assumption is that relative risk is constant across populations). However, although older and sicker people, who are often excluded from trials, are more likely to be harmed by treatment (e.g. the Action to Control Cardiovascular Risk in Diabetes study excluded those aged ≥ 80 years because the pilot study showed higher risk of serious hypoglycaemia in older people103), it is often difficult to predict the extent to which net benefit is likely to vary.

Extrapolation of evidence into recommendations in guideline development

Guideline recommendations for whole populations of people with a particular condition will therefore often require extrapolation from the available evidence, and GDGs use their judgement to make such extrapolations. The 2009 NICE Guidelines Manual identifies a number of challenges when writing recommendations, one of which relates to applicability and extrapolation (Box 1).

The 2014 update to the NICE Guidelines Manual also discusses this issue, but using the example of patients with different conditions in the same care setting.

. . . a review of systems for managing medicines in care homes for people with dementia may identify good practice that is relevant in other care home settings . . . extrapolation must be considered carefully by the Committee, with explicit consideration of the features of the condition or interventions that allow extrapolation.

p. 16748

Of note is the assumption in the 2014 guidelines manual that the use of GRADE working group profiles will account for the problem of applicability.

How does Grading of Recommendations Assessment, Development and Evaluation deal with applicability?

The GRADE working group has produced detailed recommendations for guideline development, and current NICE guideline development draws on these with some modifications. 105 Applicability is dealt with as part of whether evidence is direct or indirect, and judgement of directness is used to assess the strength of the evidence underlying recommendations. 106 In GRADE, indirectness can arise from one or more of four mechanisms:

1. differences between trial populations and the population specified as being the focus of the guideline (as defined in the PICO question), which GRADE considers a problem of applicability

2. differences between the intervention used in the trial and the intervention that is the focus of the guideline, which GRADE considers a problem of applicability

3. differences between the outcomes used in trials and outcomes that matter to patients, which GRADE discusses in terms of surrogate outcomes

4. the use of indirect comparisons, where choices are being made about two treatments for the same condition that have been evaluated in trials against placebo, but not against each other.

The presence of serious indirectness leads to the quality of the evidence in the GRADE profile being downgraded, but this in itself may not alter the actual recommendation made. Where trial populations are completely or substantially different from the population for which recommendations are being made, then it is fairly easy to say that the evidence is of low quality for applying to the latter (e.g. evidence from adults with one type of lymphoma applied to children with another). Much more common, however, is that trial populations are a subset of the population for which recommendations are being made. In that sense, the trial evidence is directly applicable to some of the real-world population, but is indirect for another part of the population. In this situation, downgrading the quality of the evidence is unreasonable for people who would have been eligible for the trial, but not downgrading for people who were not eligible may be problematic. For this reason, we believe that the GRADE application of directness does not adequately deal with the problem of applicability, since it is applied only at the evidence synthesis stage, whereas it also applies when GDGs create recommendations based on the evidence. Whether or not applicability is a problem will depend on the extent to which trial populations are very different in their expected response to the intervention from the populations for which GDGs make recommendations, or are very different from a substantial subset of the guideline population.

The aim of this element of the project was therefore to examine the applicability of trial evidence by comparing the populations included in trials underlying selected treatment recommendations for the three exemplar conditions with the actual population for which recommendations were being made.

Characteristics of the trial populations

The UKPDS was the only trial that assessed long-term patient-centred outcomes for initial treatment with sulphonylureas or insulin107 and, in overweight people, initial treatment with metformin. 108 It was carried out in the UK for people with newly diagnosed type 2 diabetes aged 25–65 years. Exclusion criteria included ketonuria, creatinine > 175 µmol/l (at least moderate CKD), recent or multiple vascular events, and ‘severe concurrent illness that would limit life or require extensive systemic treatment’. 107

One-third of patients screened were excluded,108 and patients were randomised to active treatment intended to maintain fasting plasma glucose < 6 mmol/l or to diet with drugs, only if symptomatic or fasting plasma glucose > 15 mmol/l (i.e. what would now be considered poor or very poor control). Patients who were overweight (> 120% ideal bodyweight) were randomly assigned to either metformin or diet. Patients who were not overweight were randomly assigned to sulphonylurea, insulin or diet [although analysis was of ‘intensive’ treatment (i.e. sulphonylurea or insulin) vs. diet]. The mean age of patients in all arms was 53 years.

Comparison of trial population and the population for which recommendations are being made

Using data for all newly diagnosed people in Scotland in 2008, Saunders et al. 97 found that 49.3% of patients would have been excluded from UKPDS33 and 68.2% from UKPDS34. Using our data, 44% of 6647 people newly diagnosed with type 2 diabetes in 2006/7 (approximately one-third of the Scottish population) would not have been eligible for UKPDS based on age alone, with some younger people excluded by other criteria. The remainder of the analysis focuses on age, since this is the main reason for exclusion.

Table 5 shows rates of comorbidity by age group (the UKPDS population were all aged < 65 years). Not surprisingly, older people newly diagnosed with diabetes who were not eligible for the trial were substantially more likely to have other comorbidities, including vascular and non-vascular conditions, notably CKD (2.8% of those < 65 years compared with 25.5% of those > 75 years), stroke/transient ischaemic attack (TIA) (3.2% vs. 15.5%), atrial fibrillation (1.4% vs. 14.9%), dementia (0.2% vs. 5.7%), recent cancer (3.3% vs. 10.6%), CHD (10.6% vs. 33.1%), heart failure (1.7% vs. 10.7%) and COPD (5.2% vs. 11.4%). The main exceptions to this were depression and pain, which were present in between one-sixth and one-fifth of patients at all ages, with depression a little more common in those aged < 65 years.

Drug–disease and drug–drug interactions

For common comorbidities (> 5% prevalence) the only stated drug–disease interaction in the BNF was renal impairment for metformin (dose reduction recommended if estimated glomerular filtration rate < 45 ml/min and avoidance if estimated glomerular filtration rate < 30 ml/min) and sulphonylureas (use ‘with care in mild to moderate renal impairment’ and avoid in more severe renal impairment).

Table 6 lists drugs with potentially severe interactions with sulphonylureas (metformin does not have any such interactions), and the percentage of the population with type 2 diabetes who are dispensed them. There are relatively few interactions and, with the exception of coumarin anticoagulants and NSAIDs, most are with drugs that are infrequently used.

Are there applicability issues with metformin and sulphonylureas for the initial management of hyperglycaemia in people with newly diagnosed type 2 diabetes?

Characteristics of the trial populations

Four of the five large ACE inhibitor trials were carried out in people with significant LVSD [ejection fraction (EF) ≤ 40% and often lower] although with varying levels of symptoms, and one in people with normal EF (Table 7). In principle, the Survival and Ventricular Enlargement study (SAVE) and the two Studies of Left Ventricular Dysfunction (SOLVD) trials recruited patients aged up to 80 years, but the mean age of those actually participating in these three trials was 59–61 years. The Trandolapril Cardiac Evaluation (TRACE) study recruited only adults, with a mean age of 67 years. All trials excluded people with varying degrees of CKD (although typically at least moderately severe with creatinine more than ≈ 170 mmol/l), and several stated that they excluded people with limited life expectancy from causes other than heart failure.

The beta-blocker trials were also largely carried out in people with significant LVSD and significant symptoms. Six of the seven trials recruited patients with mean ages of 58–64 years, and one [Study of the Effects of Nebivolol Intervention on Outcomes and Rehospitalisation in Seniors with Heart Failure (SENIORS)] recruited patients aged ≥ 70 years with a mean age of 76 years, although not all participants in SENIORS had LVSD. All trials excluded people with a range of physical comorbidities (most commonly CKD, although again usually moderate to severe rather than mild) and coprescribing (most commonly calcium-channel blockers and beta-agonists), and several explicitly excluded people with limited life expectancy.

Comparison of trial population and the population for which recommendations are being made

The majority (52.4%) of people with heart failure recorded on GP registers are aged ≥ 75 years, and it is unlikely that trials that were not focused on those aged > 70 years recruited many patients in this age range. Trials largely recruited patients with more severe heart failure assessed by left ventricular EF and/or symptoms. Trials additionally excluded people with a range of comorbid conditions, notably moderate to severe CKD and life-limiting conditions (the latter was explicitly stated by only a minority of trials, although conversely no study explicitly includes such people).

Table 8 shows rates of comorbidity by age group. Comorbidity is common at all ages in people with heart failure, but substantially more common in older people, where a majority of those aged > 65 years with heart failure had at least four other conditions. CKD was much more common in those aged > 75 years than < 65 years (30.6% vs. 7.5%), as were peripheral vascular disease (11.5% vs. 5.4%), recent cancer (11.2% vs. 3.9%), dementia (6.7% vs. 0.5%), stroke/TIA (20.0% vs. 7.4%), atrial fibrillation (32.4% vs. 11.7%) and COPD (19.3% vs. 10.6%). As with diabetes, pain affected approximately one in five people and depression affected approximately one in six people with heart failure, irrespective of age.

Drug–disease and drug–drug interactions

The BNF states that ACE inhibitors should be used with caution in people who may have undiagnosed renovascular disease (those with peripheral vascular disease and widespread atherosclerotic disease) and those with renal impairment. Older people, who were usually excluded from trials, were much more likely to have peripheral vascular disease or other atherosclerotic disease (although these were all common in younger people as well) and CKD.]

The BNF states that beta-blockers are relatively contraindicated in asthma and COPD, as well as in heart block, unstable heart failure and diabetes. COPD was twice as common in those aged > 75 years with heart failure as in those < 65 years, although there is good evidence that beta-blockers improve outcomes in this population. Asthma was less common in the older age group (3.9% vs. 5.9%), and diabetes had a similar prevalence.

Table 9 lists drugs with potentially severe interactions with ACE inhibitors and beta-blockers, and the percentage of the population with hospital-admitted heart failure who are currently dispensed them. The most commonly prescribed drugs that potentially interacted with ACE inhibitors were all ones that are commonly used to treat heart failure – ARBs (which are usually used in people intolerant of ACE inhibitors), diuretics and aldosterone antagonists in particular – although coprescription of these may increase the risk of renal adverse effects. Drugs that interacted with beta-blockers were rarely used and most have effective alternatives.

Are there applicability issues with angiotensin-converting enzyme inhibitors and beta-blockers for the management of heart failure due to left ventricular systolic dysfunction?

Characteristics of the trial populations

There were a large number of relatively small trials of SSRI antidepressants in people with depression (CG9045) and in people with depression with a chronic physical health problem (CG9117). Not all trials reported detailed inclusion or exclusion criteria, or the age range within which patients were eligible, or the mean age of those included. We therefore focus on age in this section. Trials included in CG90 usually had lower age limits of 18–21 years, and the majority (23 of 39 trials that reported age range for eligibility) had upper age limits between 56 and 65 years. However, as Figure 8 shows, the mean age of recruited patients was well below this, being between 36 and 45 years for 28 of the 31 trials reporting mean age. Among trials included in CG91, the majority selected patients based on them having one condition, most commonly stroke (seven trials), cancer (four trials), diabetes (four trials) and Parkinson’s disease (three trials). The mean age of included patients was higher than for the depression alone trials, but patients were more middle aged than older (e.g. the four depression with cancer trials had mean participant ages of 53, 54, 56 and 60 years).

FIGURE 8.

Mean age of patients included in studies of SSRI antidepressants for the management of depression.

Comparison of trial population and the population for which recommendations are being made

Using epidemiological data to make this comparison is more problematic than for type 2 diabetes and heart failure because depression is not well coded in clinical records, so the epidemiological data are for people treated with antidepressants. We therefore measured depression in a large data set derived from GP clinical records, defining it as ‘depression Read Code recorded in the last year OR receipt of four or more antidepressants (excluding low dose tricyclic antidepressants) in the last year’. 13 The population being compared with the trial population is therefore people with ‘recently recorded or currently treated depression’. 13 Of note is that clinically diagnosed or drug-treated depression is likely to include people with persistent mild to moderate symptoms of depression, who would not have been eligible for trials. Using this definition, 75.0% of people with depression were aged < 65 years, and 12.6% aged ≥ 75 years (Table 10).

Comorbidity was common at all ages, including physical comorbidity, but was much commoner in older people. Of people with depression aged < 65 years, 59.7% had at least one of the 33 physical conditions counted, with almost one-quarter having a painful condition and 14.2% hypertension. More than 90% of those aged > 65 years had a physical condition. In those aged > 75 years, 84.4% had two or more comorbid physical conditions; 28.5% had CHD and 16.1% diabetes, which are the two conditions in which comorbid physical disease and depression have been most studied.

Drug–disease and drug–drug interactions

The BNF states only acute mania as a disease contraindication for a SSRI, but this could not be reliably examined using GP clinical data. To examine drug–drug interactions, we used dispensing data for all patients resident in the NHS Tayside region of Scotland to define a population of people with treated depression in terms of ‘current dispensing’ of SSRIs (drugs in BNF chapter 4.3.3) or MAOIs (BNF chapter 4.3.2) or ‘other’ antidepressants (BNF chapter 4.3.4). 85 Current dispensing was defined as the dispensing of one or more of these drugs in the 84 days before 31 March 2010.

Table 11 lists drugs with potentially severe interactions with SSRI antidepressants, and the percentage of the population with depression who are currently dispensed them. Strikingly, more than one-third of older people with depression treatment were prescribed aspirin, with significant percentages prescribed oral anticoagulants, NSAIDs and clopidogrel. SSRIs increase bleeding risk, with cumulative increases in risk when coprescribed with other drugs causing bleeding, and further increases in risk with increasing age. 16 Other commonly coprescribed drugs were tramadol, TCAs and antiepileptics (although in practice the interaction – lowering of seizure threshold – does not apply to most people taking such drugs because they are prescribed them for pain modification).

Are there applicability issues with selective serotonin reuptake inhibitor treatment for depression?

Summary of findings

The analysis shows that it is feasible to use a combination of trial inclusion and exclusion criteria and epidemiological data to examine more systematically whether or not applicability of evidence is likely to be a problem, and whether or not there are important drug–disease or drug–drug interactions. An important limitation is that the study was not funded to carry out de novo data extraction and analysis. Our ability to completely match guideline populations was therefore incomplete. For example, for the examination of drug–drug interactions, heart failure was defined as hospital-admitted heart failure, in which category people are likely to have more severe symptoms, and depression was defined as drug treatment with selected antidepressants. In addition, it is likely that epidemiological data will be of higher quality for some conditions (e.g. type 2 diabetes, where diagnostic criteria are clear) than others (e.g. depression, where diagnosis and recording in routine records are more variable). Strikingly, the three exemplar conditions have distinct patterns of applicability and interaction problems.

Making recommendations for the initial treatment of hyperglycaemia in all people newly diagnosed with type 2 diabetes requires an extrapolation to older people of evidence derived from much younger people. Older people have many more comorbidities and are prescribed more drugs. They are at a higher baseline risk of CVD, meaning that in principle they would benefit more from effective treatment, but there is little evidence that tight glycaemic control per se improves cardiovascular outcomes in the first 10 years of treatment. Metformin treatment does improve morbidity and mortality, but the effect sizes are relatively small over a relatively long time. In this situation, where benefit accrues over a long time and where adverse events from treatment are probably common, our interpretation is that consideration of the potential importance of reduced life expectancy due to comorbidity/competing risks is indicated. Drug–disease interactions are common for CKD, which is common in people with type 2 diabetes at diagnosis (9.1% overall, 25.5% in those aged > 75 years), and is relevant to metformin and sulphonylurea prescribing. Our interpretation is that consideration of whether or not this requires noting in recommendations is indicated. There are two fairly common potentially serious drug–drug interactions to consider between sulphonylureas and coumarins, and between sulphonylureas and NSAIDs.

Making recommendations for the use of ACE inhibitors and beta-blockers for all people with heart failure requires an extrapolation to older people of evidence derived in younger people. Although there have been large trials of both ACE inhibitors and beta-blockers in older people (aged ≥ 70 years) with heart failure, those aged > 75 years are a very large proportion of the population with heart failure and so are significantly under-represented overall in the body of evidence informing guideline recommendations. In addition, trials are largely in people with significant (and often severe) LVSD and at least moderate symptoms despite best treatment, and exclude people with significant comorbidities. However, given evidence of substantial benefit in trial populations, it is unlikely that older populations would not benefit, although the risks of treatment will be greater, at a minimum because of greater comorbidity and coprescribing. Drug–disease interactions for ACE inhibitors are common for comorbid CKD and conditions associated with significant undiagnosed renovascular disease, both of which are common in people with heart failure, and much more common in older people with heart failure. Our interpretation is that consideration of whether or not this requires noting in recommendations or inclusion of recommendations to mitigate risk is indicated. Coprescribing of drugs that have renal adverse effects in addition to ACE inhibitors is common. Our interpretation is that consideration of whether or not explicit accounting for renal function requires noting in relevant recommendations or inclusion of recommendations to mitigate risk is indicated.

Making recommendations for SSRI treatment of depression requires less extrapolation in the sense that current evidence is derived from the main population of people with treated depression: those aged < 65 years without and with chronic physical disease. However, although older people are only a relatively small proportion of people with depression, they are a population who almost all have chronic physical disease and who have very high rates of coprescribing of drugs with potentially serious interactions. Extrapolation to this group is therefore potentially problematic because harms are likely to be much more common than in trial populations, and our interpretation is that this should be explicitly considered in recommendations. There are no significant drug–disease interactions that need considering. There is a high prevalence of significant drug–drug interactions, particularly those associated with GI bleeding, and our interpretation is that these require consideration when writing recommendations.

Of note is that the heart failure and depression (including depression with a physical condition) guidelines did address some of the identified issues, although the type 2 diabetes guideline did not (beyond a general statement to agree individualised glycaemic control targets where appropriate). However, a formal consideration of the epidemiology of the population for which recommendations are being made at scoping would be potentially useful to ensure that applicability is explicitly considered by GDGs and problems of applicability are responded to appropriately. Such formal consideration will always be limited by the data available, in terms of both the detail that trial reports provide on participants (which was poorer for depression trials than for the much larger trials considered for the two other conditions) and the quality of the epidemiological data on the characteristics of the treated population. In terms of the latter, guideline producers such as NICE or SIGN would ideally create a single data set describing patterns of morbidity and prescribing, which could then be used in the development of multiple guidelines.

Implications of findings

Given access to large and representative epidemiological data sets that characterise the population for which guideline recommendations are being made, it is feasible to examine more systematically the extent of the extrapolations being made in making treatment recommendations to inform GDG decisions about whether or not and how these recommendations should be qualified. It is worth noting that GDGs do already do this, but, as we understand it, this is largely driven by the knowledge and expertise of the individual members who happen to have been recruited. However, we believe that applicability and interactions are sufficiently important that they should be addressed more systematically, in the same way that evidence synthesis is, rather than being left to being judged in the context of the informal knowledge and expertise of GDG members.

Of note is that we do not believe that applicability is sufficiently dealt with if GRADE methods are used. GRADE consideration of applicability takes place during evidence synthesis both for individual trials and for the body of evidence in any meta-analysis. Applicability is one of four criteria used to judge whether evidence is direct or indirect. As defined by GRADE:

We are more confident in the results when we have direct evidence. By direct evidence, we mean research that directly compares the interventions in which we are interested delivered to the populations in which we are interested and measures the outcomes important to patients.

p. 1304106

However, the consequence of indirectness in GRADE is that the quality of the evidence that may support a recommendation is downgraded, which influences the strength with which recommendations are made. Our conclusion is that applicability either needs explicitly accounting for by defining pre-specified subgroups to examine during evidence synthesis, if there is enough prior evidence to support this approach, or needs explicitly re-examining after evidence synthesis and drawing on the findings of all included studies, in order to inform the writing of recommendations (Figure 9). This is because there may be direct evidence for some people in the guideline population but not for others, and therefore it may be appropriate to make different recommendations for subgroups based on different quality of evidence in those subgroups.

FIGURE 9.

Directness and applicability at different stages of guideline development.

Then GDGs will have to consider whether they wish to make a single recommendation for all people with a condition, or stratified or otherwise qualified recommendations for different subgroups. Such judgements already happen but we believe that systematic use of epidemiological data to inform them is required. Based on discussion with the PRG, factors which GDGs might consider when making such judgements include:

• the nature of the treatment, in terms of whether or not its mechanism of effect is likely to apply across all patients, and its potential for harm

• the duration over which the treatment will be used, which is relevant when extrapolating to populations with limited life expectancy due to other conditions, age or general frailty

• the absolute size of the observed benefit in the trials, which is relevant, since large benefits are less likely to be sensitive to small variations in benefit or harm when used in people not eligible for trials

• the nature of the differences between trial and non-trial populations, including age, comorbidity, coprescribing and likely life expectancy, in terms of whether or not these are large enough to matter in the context of the previous factors.

Such considerations are particularly likely to apply when the outcomes being improved by treatment are not observable by clinicians or patients. For example, clinicians and patients can observe change in pain during treatment with an analgesic. In contrast, most preventative treatments require clinicians and patients to take it on trust that meaningful outcomes are better, because a prevented heart attack or other prevented future event is not observable in an individual.

Using absolute benefit expressed in terms of trial outcomes

Historically, binary outcome treatment effects measured in RCTs were usually only reported in terms of relative risks or risk ratios (RRs), odds ratios, relative risk reductions (RRRs) or hazard ratios. Not all patient-centred outcomes are binary, but such outcomes are common, including mortality, serious events such as stroke or heart attack, and admission to hospital or a care home. The evidence-based medicine movement highlighted that reporting binary outcomes in terms of relative benefit was often misleading, since the absolute benefit to which this translates often varies widely depending on the baseline risk of the outcome. 122 For example, a treatment that reduces death by 60% (RR 0.4, RRR 0.6) sounds very impressive, but does not actually prevent many deaths if only 10 in 1000 people not treated die compared with 4 in 1000 people who are treated. Instead the recommendation was that clinicians focus on absolute benefit, which in the example above would be six avoided deaths per 1000 people treated (ARR of 0.006 deaths per patient treated). However, since ARR is not always easy to grasp, it was proposed that this could be usefully expressed as a ‘number needed to treat’ (NNT), which is defined as 1/ARR and conventionally rounded up to the nearest whole number. In the hypothetical example given, the NNT would be 1/0.006 = 167 (i.e. 167 patients would need to receive the treatment to avoid one death).

One important rationale of the NNT was to create a measure of treatment effect that was more meaningful to clinicians and patients, and therefore better able to support decision-making. There is some evidence that clinicians feel that NNTs are meaningful123 and that they vary their treatment recommendations in response to vignettes more appropriately when presented with NNTs than RRRs. 124 However, although there is evidence that both clinicians and patients are both less likely to choose a theoretical treatment if benefit is presented as a NNT or ARR than when it is presented as a RR or RRR, it is unclear whether or not the use of NNTs actually influences practice, and NNT does not appear to be as helpful to patients. 125,126 Stovring et al. 127 suggest that all numbers may be confusing to patients, but that ARR is marginally easier for them to interpret.

Of note, however, is that, despite the appealing face validity of the example above, such simple statements of NNTs are in practice misleading. For example, ‘NNT of 167 to avoid one death’ leaves much unsaid that has to be explicitly considered, including the nature of the treatment and what it is being compared with, the duration of treatment and the length of follow-up. The interpretation of that NNT would be very different if the treatment were a tablet with no side effects, taken once, with the death avoided in the next 24 hours, from if it were lifelong treatment with a drug that caused persistent low-level nausea in everyone who took it, with the death avoided after 30 years of treatment. 128,129

Overall, there is general approval of the idea of reporting treatment effects in terms of absolute risk; for example, the Consolidated Standards of Reporting Trials statement for trial reporting states ‘For binary outcomes, presentation of both absolute and relative effect sizes is recommended’. 130 However, a number of important problems have been identified in using absolute benefit and NNT. In practice, the NNT in particular is not routinely calculated or reported in trials, systematic reviews or guidelines (although the increasing use of the GRADE approach means that guidelines increasingly report net benefit of treatments in a standardised way105,131,132).

Using estimated lifetime quality-adjusted life-year gain as a single metric for comparison

The QALY presents an attractive solution to at least one of the three problems highlighted by McAlister128 in his critique of the NNT: that of disparate outcome measures. In principle, the QALY provides a standardised outcome by which treatments can be benchmarked and prioritised for those with multimorbidity based on absolute health gains.

The QALY has become the cornerstone of CEA used to inform resource allocation decisions because it provides a generic measure of patient benefit that, when combined with costs, allows mutually exclusive alternative interventions to be compared and prioritised. The QALY calculation has two dimensions that are combined: survival multiplied by HRQoL. Generating values for the survival dimension can be argued to involve few value judgements. In contrast, generating values for the HRQoL dimension requires a fundamental judgement regarding what constitutes ‘quality’. The NICE methods guide for technology appraisal recommends using the European Quality of Life-5 Dimensions (EQ-5D) as the generic HRQoL measure. 59 The EQ-5D is a multiattribute measure of HRQoL comprising five domains with three levels (a newer five-level version is now available), which generates a total of 243 health states. 143 To use it in the context of economic evaluations, it is necessary to understand the preference weight attached to each health state. The commonly agreed scale used is anchored at 1 for full health and 0 for health states equivalent to dead. NICE recommends that that the values attached to the HRQoL dimension should be informed by societal preferences. The preference weights currently used in the UK were collected as part of a study called the Social Tariff survey. 143 The preference weights were estimated using a time trade-off exercise with a representative set of the UK population.

Traditionally the QALY has been limited to the domain of economic evaluation. In this domain, the benefits of the QALY are clear, as it provides a common currency by which scarce health-care resources can be allocated in the most efficient manner. The characteristics of the QALY, calculated as a combination of morbidity and mortality into a single measure of absolute benefit, make it attractive outside this domain. 144

If communicated using clear and explicit language suitable for the audience, the use of the absolute QALY could potentially provide a useful measure of clinical benefit to stakeholders, such as patients and clinicians, as an aid to their treatment decisions. In addition, guideline developers focusing on multimorbidity could use estimates of the absolute QALY to complement current cost-effectiveness evidence as an additional source of evidence. The absolute QALY, once a set of interventions have been found to be individually cost-effective, could be used to prioritise treatments within bundles of interventions.

For the purposes of this work, the absolute QALY is defined as the number of QALYs gained from receiving intervention A minus the number of QALYs gained without receiving intervention A. The ‘absolute’ part of the absolute QALY is reliant on the comparator being defined as the ‘do-nothing’ option.

Figure 10 is adapted from a NICE Decision Support Unit (DSU) document that was used as supportive evidence to inform discussions on whether or not to preferentially value interventions that treat conditions in which the ‘burden of illness’ is large. 145 It can be used to illustrate the absolute QALY concept. As part of this evidence base, the absolute QALY shortfall was put forward as a potential proxy measure of ‘burden of illness’. When showing the importance of burden of illness, the starting point is X in Figure 10, where patients are already getting treatment for their condition. The burden of illness is then defined as the difference in the area that lies between O₁ and O₂ and between O₁ and X (labelled areas D, E and F in Figure 10). Each of these areas, with its own interpretation, represents the absolute QALYs a patient would lose on an intervention compared with the total possible number of QALYs if in, age-adjusted, full health. The QALY shortfall has two dimensions: a HRQoL impact (measured by D) and a survival impact (measured by F). The area E represents the consequential impact of both HRQoL and survival impact together. In contrast, the starting point for the absolute QALY calculation starts at the value of QALYs assuming patients are not on treatment (point Y on Figure 10). At point Y the baseline QALYs for no treatment are represented by the area Z. The absolute QALY gain is calculated as the difference between X and Y (areas A, B and C). Survival gain is measured by the change in S and HRQoL gain is measured by the change in H. The combined impact of the improvement in survival (area F) and improvement in HRQoL (C and A) is represented by area B.

FIGURE 10.

The use of absolute QALY shortfall as a proxy for burden of illness. Source: adapted from the NICE DSU document. 145

Calculating absolute quality-adjusted life-years

Quality-adjusted life-years were calculated using the same data inputs as in the original model used to inform the NICE CGs. All QALYs were discounted at a rate of 3.5%. To make the models comparable for the purpose of estimating absolute QALYs, some of the baseline assumptions that were used to inform the original guideline recommendations were adapted. The base-case results in the antihypertensives model were for a 65-year-old with a 2% annual risk of CVD. This risk was lowered to a 1% annual risk to bring it in line with the lipid modification guideline, which used a 10% 10-year risk of CVD. Subgroup analyses for different starting ages were also conducted.

Absolute QALYs were calculated by subtracting the expected QALYs for no treatment from the expected QALYs of treatment with the cost-effective intervention. It was not feasible to produce a measure of the uncertainty around the mean absolute QALYs because the model-based CEA used as exemplars did not include the necessary type of PSA as part of the original evidence submitted to inform the NICE CGs. The PSA used included only a selection of parameter values.

Comparing absolute benefit using trial outcomes

Tables 14 and 15 show a version of the GRADE profile used by NICE modified for our purposes to focus on relative and absolute benefits and omitting assessment of quality of evidence for reasons of space. Stang et al. 129 recommend that a NNT should never be quoted without clarity about which treatments are being compared, the period of treatment and follow-up, and the direction of the effect towards benefit or harm.

Table 14 shows the relative risk and ARR for multiple outcomes for five different treatment/condition combinations: ACE inhibitors and beta-blockers in heart failure due to left systolic dysfunction, metformin and sulphonylureas/insulin in newly diagnosed type 2 diabetes, and statins in people with type 2 diabetes. The size of the table highlights that providing clinicians and patients with comparative data for treatments for different conditions is likely to fairly rapidly require decisions about which outcomes to compare, and paper-based ways of delivering this information such as NICE already produce153 are unlikely to be feasible. Of note is that, as McAlister128 points out, comparing across multiple outcomes is challenging. Table 15 shows the same data but only for the total mortality outcome, which makes direct comparisons more feasible. The NNTs are estimated from the trial data (i.e. by applying the estimated trial RRR to the baseline risk in the control group), and where the relative risk is not significantly different from 1 then NNT has not been calculated. Of note is that all NNTs are of a similar order (most are 15–30, the largest is ≈ 60), but this cannot be easily interpreted because trial duration varies more than 10-fold, and the NNT in itself is not meaningfully interpretable without simultaneous consideration of the duration of treatment. In addition, extrapolating these NNTs to other populations is straightforward only if the baseline risk of the outcome is the same as or very similar to that observed in the trial population. 133

Using more realistic estimates of baseline risk

An important limitation of absolute risk estimates calculated using the baseline rate of the outcome in trial control arms is that these may not be very representative, and therefore may over- or underestimate likely absolute benefits in the treated population or in individuals. Depending on how the trial population has been selected, control-arm baseline risk may be higher or lower than baseline risk in the real-world population, and baseline risks may vary considerably between different groups of the real-world population.

From a guideline development perspective, for conditions where there are published data on the distribution of baseline risk across the population, it would be feasible to estimate the likely range of absolute benefit across the actual range of baseline risk in the population for which recommendations are being written. This would potentially allow a GDG to define if there are subgroups for whom recommendations could or should be varied. For individual patient decision-making, baseline risk can be estimated by a suitable prediction tool (if available). Such tools would ideally have been developed using a representative population cohort and validated in an independent cohort. Commonly used examples of such calculators include the Framingham and QRisk scores for risk of CVD, and the FRAX™ (Sheffield, UK version, World Health Organization Collaborating Centre for Metabolic Bone Diseases) and QFracture (version 2012-1, ClinRisk Ltd, Nottingham) scores for risk of fracture. For heart failure, there are two risk prediction tools available for estimating total mortality risk. The Seattle Heart Failure model is derived from a single trial population and validated in five other trial populations. 155 The Meta-Analysis Global Group in Chronic heart failure (MAGGIC) model156 is derived from six trial and 24 observational populations and has been externally validated using the Swedish national heart failure registry. 157 For type 2 diabetes, the NICE lipid modification guideline recommended using QRisk2 to estimate cardiovascular risk in people with type 2 diabetes, on the basis that it was derived from a much larger and more recent cohort than the UKPDS Risk Engine or the Framingham study. 148 In practice, baseline risk estimates in real-world populations are often not available, and, for the illustration here, we chose to use MAGGIC estimates of 1-year survival in heart failure, and QRisk2 estimates of cardiovascular risk in type 2 diabetes.

Comparing treatments using annualised number needed to treat across plausible ranges of baseline risk

For heart failure, we used published data on the distribution of predicted risk score in the Swedish national heart failure register (which have a near normal distribution) to estimate average, lower and higher baseline risks for mortality at 1 year defined as MAGGIC scores of 25, 17 and 32 (approximately encompassing the interquartile range of risk). 157 Using these baseline risk estimates and estimates of relative risk from meta-analyses, we estimated 1-year ARR and NNT for ACE inhibitors versus placebo and for beta-blockers versus placebo.

For type 2 diabetes, we are not aware of any published data on the individual distribution of cardiovascular risk. We therefore chose to model 10-year baseline risks of CVD at 10%, 15% and 20%, representing the range between current (10%) and previously recommended (20%) thresholds for use of statins for primary prevention of CVD. The published guidelines do not provide a relative risk for reduction in total CVD events, and we therefore applied the estimated RR for preventing MI using metformin versus diet (plus drugs, only if glycaemic control were very poor) from UKPDS34108 and using statins versus placebo. 148 Since statins are less effective at preventing stroke, this is likely to overestimate the benefits of statins on total CVD events. We then annualised the NNT to account for the 10-year follow-up implied by the QRisk baseline risk estimate.

Table 18 shows the estimated ARR and annualised NNT. Annualised NNTs across the range of estimated baseline risk examined were 9–34 for beta-blockers versus placebo and 25–93 for ACE inhibitors versus placebo to prevent one death in people with LVSD. NNTs ranged from 139 to 278 for metformin versus diet and from 114 to 228 for statins versus placebo to prevent one cardiovascular event in people with type 2 diabetes.

Comparing treatments using absolute quality-adjusted life-year gain

Table 19 presents the estimated absolute QALY gains from pharmacological treatment of hypertension and lipid management based on the model-based CEAs used to inform the relevant NICE guidelines. The results show that the lifetime absolute QALY gain from the use of calcium-channel blockers to treat hypertension is much larger than the lifetime gain for high-intensity statin treatment to manage a similar level of cardiovascular risk.

Comparing the absolute benefit of treatments using trial clinical outcomes

There are no major technical barriers to using a consistent method to produce different estimates of the absolute benefit of treatments for different conditions, but all such estimates rely on making a number of significant assumptions, namely:

1. Relative risk is assumed to be constant across all populations. This is a standard assumption and it is difficult to extrapolate any trial evidence without making it. There is evidence that it is usually but not always true. 138

2. Competing risks of death are assumed to be insignificant. This is also a standard assumption that is rarely examined, and is unlikely to be true in the presence of other conditions with a high risk of death. For example, mortality in heart failure trial populations is likely to be more driven by heart failure than by other conditions, whereas mortality in real-world populations with higher rates of comorbidity and greater age-related frailty than trial populations is likely to have a larger component that is not amenable to heart failure treatment. As an example, a young patient without comorbidity who is in the highest risk group of MAGGIC scores is likely to have severe, symptomatic LVSD and this will drive their mortality risk. An older patient with comorbidity may be in the same risk group despite having asymptomatic mild LVSD, and a larger proportion of their mortality will be from causes that will not be affected by heart failure treatment. In populations with significant competing risks of death, benefits are therefore likely to be overestimated by the method applied.

3. Baseline risk is assumed to be measured without error, at both population level (e.g. median baseline risk in a trial) and individual level if a risk calculator is used in individual decision-making. 159 The confidence intervals (CIs) around ARR and NNT estimates are therefore likely to be falsely narrow. Although there are methods available to account for this, baseline risk calculators do not normally provide CIs around baseline risk estimates. 160 A broader concern is that all baseline risk estimates are at risk of a number of forms of bias. For example, observational data in principle provide better estimates of baseline risk in the population who will actually receive the treatment than trial estimates. However, observational data are more likely to underascertain outcomes because they usually rely on routine clinical recording of incident conditions. In addition, both trial and observational data may not reflect changes due to changes over time in baseline risk, for example because treatment with other effective interventions has become routine. 159

4. Estimates of relative risk and estimates of baseline risk may not be available for the same outcome. Here, they were not both available for metformin and statins; the relative risk used is for the outcome of MI, whereas the baseline risk estimate is for total cardiovascular risk, which includes fatal and non-fatal stroke. Since the estimated relative risk for stroke reduction is somewhat smaller in both cases, the actual absolute benefit is also likely to be smaller and so the NNT somewhat larger.

5. Harm is assumed to be constant across populations, which is unlikely given that older people and those taking multiple drugs are known to be more likely to experience ADEs. 34 For metformin, there is evidence that age, coprescribing and genetic factors all contribute to metformin discontinuation in the absence of treatment failure (as a proxy for adverse effects), with age dominating. 161 For people on ACE inhibitors, risk of acute kidney injury in the face of coprescribing of other nephrotoxic drugs is strongly associated with increasing age. 162 Net benefit may therefore be reduced in some populations because of increased harm.

Comparing absolute benefit using quality-adjusted life-years

Absolute QALYs gained are a potentially attractive concept that could potentially inform prioritisation between two treatments. In the example given, where key principles were satisfied, then antihypertensive treatment on average leads to three- to four-fold QALY gain compared with statin treatment. For example, there is a QALY gain of 0.85 in men aged 60 years with antihypertensive treatment compared with 0.20 with statin treatment (although, even in this case, the models do not entirely match, as the starting age is not identical). This approach specifically addresses one of McAlister’s128 three critiques relating to the problem of comparing the benefit of treatments when they affect different outcomes. However, making such comparisons is not as straightforward as it might first appear, and we propose that it should be underpinned by five guiding principles that help make assumptions explicit, and that can be reported and provided to decision-makers wishing to use the information to guide prioritisation.

There are a number of important considerations to take account of when using absolute QALY gains to compare between interventions. The decision-analytic models used in this analysis were similar in terms of a number of key attributes, but differed in the time horizon for the analysis, as the antihypertensive model ran for 35 years compared with 40 years for the lipid modification model. The length of time in which absolute QALYs have had chance to accrue is a key variable that may drive the results of an analysis estimating absolute QALYs. Economic evaluations are disparate in terms of the assumed time horizon for the analysis. 163 The relevant time horizon will be guided by the relevant patient population and the need to capture the key differences between intervention and comparators during the time period for the analysis. For the selected case studies, there was a relatively small difference of 5 years. This difference in time horizon is unlikely to change the findings substantially but analysts will have to be transparent and inform decision-makers when the difference in time horizon between models is likely to bias the ability to compare between interventions using absolute QALYs. They also differed in the characteristics of the starting cohort. Two model-based CEAs could have the same time horizon (lifetime) but, if the starting age of the original cohort is different, then the length of time in which the QALYs have to accrue will clearly be different. Similarly, even for treatments affecting the same outcome (in our example, CVD), then the way in which each model defines the population studied in terms of baseline risk may have large effects on the conclusions drawn.

Following the approach recommended in the NICE reference case, a de novo model-based CEA created to inform guideline development usually reports the incremental QALY comparing the intervention with the next best alternative. This approach is underpinned by the logic that NICE is assessing the cost-effectiveness of mutually exclusive, non-interacting interventions. However, incremental comparisons against the next best alternative are not the same as absolute gains against a do-nothing approach. In the analysis estimating absolute QALYs gained, we assumed a do-nothing alternative to allow direct comparison between interventions. This approach is not new. The recognition that this standard method of generating incremental QALYs from CEA may not be not appropriate when interactions between interventions are important has been recognised by the World Health Organization in its generalised cost-effectiveness framework. 164

The generalisability of findings from model-based CEA has been the topic of ongoing debates between health economists and users of economic evidence. 163,165 Some health economists argue that the limited generalisability of economic evaluations means that producing an assimilated evidence base is not useful. 166 A review of cost-effectiveness studies found some 26 factors that influenced how results from studies would vary from location to location and from time to time. 163 In the analysis presented here, the issue of poor generalisability, when generating absolute QALYs for comparison between interventions, was taken into account by a number of key assumptions. Costs were not included and, therefore, many of the socioeconomic and organisational factors that vary between organisations affecting resource use and cost were eliminated. Most of the jurisdiction-specific factors were also limited by adopting a standardised approach, such as the NICE reference case, and comparing only outcomes that have been produced in agreement with this approach.

The QALY has become a key input into model-based CEA, and the current NICE reference case recommends using published public preference weights for health states generated using the EQ-5D. 59 This approach has become accepted practice when using model-based CEA to inform guidance produced by NICE as part of its technology appraisal or guideline development processes. The estimate of an absolute QALY within either a trial- or model-based framework does not imply that all patients can be expected to receive such an outcome, in the same way that estimates of life expectancy do not mean all individuals will live for the same length. The absolute QALY is an aggregate measure calculated either within a hypothetical cohort of patients or within the trial sample. Some patients would probably gain fewer QALYs and some would gain more, but the absolute QALY measure is what would be expected over the cohort or sample. This variability in absolute QALYs will have no impact when using the values to inform decisions affecting populations of patients but may limit the interpretation of the relevance of absolute QALY values at the individual patient and clinician level.

The absolute QALY gain is suggested as another input for decision-makers developing CGs for populations of patients. There is potential, however, for decision-making based on the absolute QALY to diverge from other decision-making criteria, such as recommendations based on cost-effectiveness. For example, an intervention that had a large absolute QALY gain but was expensive might not be a cost-effective option, with the incremental benefit not justifying the incremental cost for a given cost-effectiveness threshold. Likewise, the text-book approach when making decisions between mutually exclusive options based on cost-effectiveness is to compare interventions with the next best alternative. Comparisons with a do-nothing approach (e.g. mean cost-effectiveness) may produce misleading estimates of the true displaced activities when compared with the next best alternative. The suggested solution to this challenge is to generate estimates of absolute QALYs only once the relative cost-effectiveness and most cost-effective option for an intervention in a defined patient population have been determined.

Finally, from a guideline development perspective, a key limitation of the absolute QALY gain approach is that it relies on having access to a suitable economic model. As shown in Chapter 2, fewer than one-fifth of CRQs have an associated de novo economic model, and about half have no economic evidence associated with them at all. From this perspective, comparing absolute benefits in terms of trial clinical outcomes is more straightforward, since these may be more easily available.

Time to benefit

Time to benefit and time horizon to benefit are analogous terms used in the literature (hereafter we will use TTB). The literature on TTB has focused on the need to identify the time point at which the benefits of an intervention (exclusive of its harms) occurred. TTB can be demarcated, and interpreted, as either a biological or an epidemiological concept. 168 The biological interpretation means that it is feasible to calculate a precise TTB from an intervention for an individual. Practically, this interpretation involves generating a measure of the time it takes for an intervention, generally a pharmacological therapy, to reach its intended therapeutic target and reduce a measurable outcome by a certain pre-set threshold. An example of this biological interpretation would be the time for plasma levels of LDL cholesterol to reach the pre-defined target after starting a statin therapy. The biological interpretation allows an individualised TTB to be calculated based on observed responses. However, individualised times would often be suitable only for surrogate clinical outcomes, such as the reduction in LDL cholesterol, rather than the patient-relevant final end point, such as a cardiovascular event. This limits the practical relevance of using the biological interpretation of TTB in informing clinical decision-making. To expand TTB so that it takes account of outcomes that clinicians and patients value, such as a reduction in the risk of mortality, TTB needs to be interpreted as an epidemiological construct. TTB as an epidemiological construct requires assessment of reduction in risk at the population level, and evidence generated from appropriately designed clinical trials. 168

To date, however, there has been relatively little work examining TTB, reflecting the fact that clinical trials are typically designed to run for, and examine effectiveness after, a defined period of follow-up. 168 This observation is not surprising given the lack of formal statistical methods developed to estimate TTB in an unbiased manner, although some statistical methods have been proposed. Lee et al. suggest an inference about TTB can be made by visual inspection of the published Kaplan–Meier curves and identifying the point of separation. 169 In the absence of any formal way of assessing when divergence happens, this seems likely to overestimate the presence of significant risk reduction when one does not exist, while also underestimating the actual TTB. 170

Another potential statistical approach to estimate TTB is to use published evidence post hoc, with the associated caveats for inference associated with post-hoc analysis. Ray and Cannon171 provided an example of this post-hoc approach when they re-examined a published clinical trial to assess whether or not the benefits of high-intensity statin treatment (40 mg of atorvastatin daily) occurred more quickly than low-intensity treatment (40 mg of pravastatin daily) for those patients with a recent diagnosis of acute coronary syndrome. The original clinical trial, published by Cannon et al. 172 in 2004, identified a significant statistical difference between treatments at mean 2-year follow-up in favour of the high-intensity approach (RRR 0.16; p = 0.005) when using a composite outcome of mortality or a serious cardiovascular event. The authors of the original clinical trial also made reference to an observed divergence of the Kaplan–Meier curves, suggesting that the clinical beneficial effects may occur sooner than the full follow-up time at which the formal statistical test was applied in the pre-specified main trial analysis. Ray and Cannon171 identified that a statistically significant benefit occurred at 4 months’ follow-up, implying a much shorter TTB than trial duration. Lee et al. 169 also retrospectively fitted statistical models to published meta-analysed data to identify the TTB for colorectal and breast cancer screening programmes. This analysis made use of a large sample of pooled meta-analysis data and found that it took 10.7 years (95% CI 4.4 to 21.6) before one death from breast cancer screening was prevented per 1000 patients screened. Similar results for colorectal cancer were also found, with a TTB of 10.3 years for 1000 patients screened (95% CI 6.0 to 16.4). 169 However, all such essentially observational post-hoc analysis risks finding spurious results in a similar way to post-hoc subgroup analysis. 173

In practice, therefore, it is more usual for TTB to be defined simply as the median or mean follow-up in clinical trials showing that treatment is effective. This is likely to be conservative but is consistent with the original, pre-specified trial design and power. Theoretically, therefore, interventions in different disease areas could be compared where TTB for a standardised outcome is assumed to be trial duration. For a common outcome such as a reduction in all-cause mortality, then, all things being equal, the intervention with the shortest trial duration, and therefore the shortest TTB, would be preferred. However, a range of important caveats must be taken into account even when using the concept of trial length as TTB. For example, as Holmes et al. 170 describe, the TTB estimated in this way will be inherently linked to the trial design. Accordingly, all things being equal, larger trials will indicate shorter TTB than smaller trials because trial duration is likely to be shorter, as outcomes will become statistically significantly different earlier. 170

Consequently, developments in methods are required to allow quantification of statistical estimates of TTB ex ante to trial design, which are likely to parallel developments in the analysis of trials with early stopping rules. Stopping rules are built into trial protocol and involve setting interim time periods to test for differences between experiment and control, often to protect patients from clearly harmful drugs. 174 However, the use of stopping rules is associated with controversy, most notably when stopping trials early for a perceived benefit of the experiment versus the control. As trials are typically powered to find clinically meaningful differences at trial follow-up, it has been found that truncated trials, which stop early because of an apparent treatment effect, can provide misleading and biased estimates of treatment effect compared with those studies that are not truncated. 175 Similar problems will be relevant to using trials to estimate TTB. Repeated statistical tests increase the likelihood of a false positive, while large random fluctuations early in the trial, where the sample size is small, could wrongly suggest statistical TTB and overestimate treatment effect.

In summary, the highest-quality evidence on statistical TTB is likely to come from trials that have been designed from the outset to identify interim time points at which benefit occurs within the trial, but methods to validate such an approach are still required. 168 A further, less robust approach, which would generate a conservative view of TTB, is to assume that it occurred at trial completion or at median trial follow-up. 171 Alternatively, TTB could be estimated from ex-hoc analysis of already published evidence or using qualitative visual inspection of where published survival curves appeared to separate. All of these methods of analysis are still in their infancy and in need of further development.

Given the current state of play of the literature, the incorporation of a statistical TTB as the means by which a temporal dimension could be incorporated within CGs was judged not to be feasible. Therefore, the use of a temporal dimension in already existing model-based CEAs supplemented by the pay-off time framework was explored as an alternative way of informing the development of CGs.

Pay-off time

The following text has been adapted from Thompson A, Guthrie B, Payne K. Using the ‘pay-off time’ in decision-analytic models: a case study for statins in primary prevention. Med Decis Making 2017. Published online 25 April 2017. 182 This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 3.0 License (http://www.creativecommons.org/licenses/by-nc/3.0/) which permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access page (https://us.sagepub.com/en-us/nam/open-access-at-sage).

The pay-off time is defined as the minimum elapsed time in which the expected cumulative net benefits of an intervention exceeds its expected cumulative harms. 176 The pay-off time concept, first proposed by Braithwaite et al. 176 in 2009, more readily captures a temporal dimension that can be readily incorporated into a model-based CEA to inform CGs. The rationale for the pay-off time approach is similar to statistical TTB in that short-term harms must be balanced against longer-term benefits, and stakeholders can use this evidence to prioritise care. When using the pay-off time approach, less attention is placed on the explicit timing of benefit within trial and more focus is on the relative timing of benefit compared with harms from the intervention. Braithwaite et al. 176 argued that CG recommendations should not be implemented in individuals if the overall ratio of expected benefits to harms is likely to be negative within their life expectancy. 176 Equivalently, this can be expressed as that an intervention should be used if the life expectancy of a patient is greater than the pay-off time from the intervention. 36,177,178

When applying the pay-off time concept, benefits are classified as outcomes that can improve mortality, morbidity or both in a patient. Harms are outcomes that can worsen mortality, morbidity or both. Initially the pay-off time concept was applied using disparate outcomes such as reductions in mortality (for benefit) as well as adverse events (for harms). 177 In a later application, benefits and harms were standardised using a common metric, the QALY, to measure both benefits and harms in a single measure. 36 In this context, Braithwaite used the QALY in the pay-off time approach to assess whether or not to (1) screen chronically ill 50-year-old women for colon cancer and (2) use intensive glucose control for chronically ill diabetic patients. In 2013, Yuo et al. ,178 including Braithwaite, applied the pay-off time framework to assess whether or not the potential upfront harms of revascularisation for patients with asymptomatic carotid artery stenosis (where the operation itself causes stroke in 1–3% of patients) are worth the deferred benefits (a 5% absolute stroke risk reduction over 5 years). 178 These published applications of the pay-off time concept all examined interventions with upfront, one-off harms, and the pay-off time was readily estimated using algebraic mathematical calculations. It was clear from these applications that the QALY metric together with the pay-off time concept is a potentially useful measure to capture harms and benefits in a way that allows comparison across interventions and an understanding of the impact of a temporal dimension.

The published examples of using the pay-off time approach show a move away from simple algebraic calculations to using a decision-analytic approach in which outcomes and probabilities are combined to provide a systematic assimilation of multiple evidence sources and calculation of the probability-weighted outcomes (QALYs). 36,176–179 The decision-analytic approach proposed is similar to the use of model-based analysis to generate estimates of incremental costs and benefits to inform CEA. McCabe et al. have demonstrated how a pay-off based approach can be integrated within a decision-analytic model to generate information for reimbursement agencies. 180,181 Such an approach can produce an estimate on the time it takes for a new innovative technology to ‘pay off’ given an original cost. In a similar way it is, therefore, straightforward to combine the use of the pay-off time concept, QALYs and model-based CEA to incorporate a temporal dimension into existing model-based CEAs that follow patients, or cohorts of patients, over their lifetime and estimate when the benefits of an intervention should outweigh the harms or when the intervention pays off.

Case study selection

The feasibility of using the pay-off time in a model-based CEA within guidelines for the three pre-selected exemplar conditions (type 2 diabetes, depression and chronic heart failure) was explored. These exemplar CGs did not use model-based CEA based on Markov-type models, which was needed to implement the pay-off time approach. Therefore, a manual search of published CGs relevant to the management of long-term conditions was conducted, and identified the 2014 lipid modification guidance (CG181) as suitable. 197 The decision-analysts named in published full NICE guidance were then contacted using e-mail. We were then granted access to a full executable version of the de novo model developed by the National Clinical Guidelines Centre under the standard NICE licence.

The model

Table 21 summarises the key attributes for the de novo model used in this analysis.

The model included 15 health states, and is fully documented in the published NICE guideline CG181. 198 The model was populated with the same data as used for the production of CG181. The model was designed to support the use of the QRisk2 tool for predicting risk in people without diabetes being considered to receive statins for primary prevention. QRisk2 (10 years) is a cardiovascular risk tool developed by Hippisley-Cox et al. 199 based on the QResearch UK primary care cohort. It estimates an individual’s risk of experiencing any of fatal or non-fatal angina, MI, TIA or stroke over the following 10 years, and can be found at www.qrisk.org/index.php. The model allows subgroup analysis based upon sex, age (40, 50, 60 and 70 years) and baseline QRisk2 (10 years).

In 2005, NICE conducted a technology appraisal of the use of statins for the primary prevention of CVD (published in 2006). 200 In 2007, the NIHR funded a HTA,201 with a subsequent report published by the same authors as the original 2005 work. Neither the NICE technology appraisal nor the NIHR HTA-funded study included quantification of the impact of DTDs in the base-case analysis, the sensitivity analysis or a scenario analysis. 200,201 The justification for this omission was given as:

. . . as adverse events and side-effects are rare, patients receiving statin treatment in the model do not receive a penalty utility due to their medication. As statins are prescribed for life, there may be a disutility associated with this, but it is assumed that this is small in comparison to the benefits received and as such is not modelled.

Ward et al. 201

The potential for disutility losses associated with adverse events and side effects following treatment was considered but thought to be inconsequential in comparison with the potential benefits. In 2014, a subsequent economic model was developed for the NICE CG on lipid modifications (CG181) appraising the use of statins for both primary and secondary prevention. This model had a similar structure to the two existing models and did not include DTD either in the base-case or in scenario analysis. Instead scenario analysis was used to explore the impact of assumptions about patients’ desire to continue with the treatment on 6-month adherence to capture some negative impact associated with taking the medicine. The justification given for the lack of inclusion of DTDs in the sensitivity analysis was that the analysis should be consistent with the previous NICE technology appraisal (2005)200 and published NIHR-funded HTA report. 201

For the purpose of this study, additional harms were then built into the model to reflect DTD. All patients in the model cohort were assumed to suffer from an annual disutility associated with treatment. The values for the DTD were informed from the rapid review of the literature described previously.

Patient vignettes

Patient vignettes were used to provide clinically relevant scenarios to quantify the impact of pay-off time. Three patient vignettes were created with input from clinicians on the project team (n = 3) and the PRG (n = 4). The patient vignettes were built around scenarios defined by three 10-year QRisk2-estimated CVD risks of 10%, 15% and 20% (Table 22), which span the range between the currently recommended treatment threshold of 10% 10-year CVD risk and the previously recommended threshold of 20%.

Analysis

Quality-adjusted life-year profiles were generated for each of the three patient vignettes, representing three different levels of baseline QRisk2, and also stratified according to different DTDs for treatment with high-intensity statins compared with do-nothing. The following were calculated: pay-off time in QALYs; pay-off time in net benefit assuming a threshold of £20,000 per QALY gained; absolute QALYs; time for the peak investment in QALYs gained (years); and size of the peak investment (net benefit, £). DTDs were applied by ‘adding’ in the defined (negative utility) value to the total utilities for each cycle and assuming that the whole cohort had the DTD. This approach means that a constant utility decrement is applied to each cycle in the model for the lifetime of each patient in the cohort who is taking the treatment. This is consistent with the current methods of eliciting DTDs, which often use time–trade-off exercises (with questions spanning different hypothetical time periods, typically 10 years) to estimate the utility decrement to use for each 1-year time period. 185,194–196

Exploratory analysis

In a supplementary exploratory analysis, the pay-off time was calculated for a sample of hypothetical patients with multimorbidity. Values for all-cause mortality used in model-based CEA are commonly taken from life tables from the Office for National Statistics (ONS). 202 From these life tables, age- and sex-adjusted mortality rates can be calculated. However, these rates represent population average mortality for men and women at different ages. A reasonable hypothesis is that those with multimorbidity are likely to have a higher risk of all-cause mortality. Menotti et al. 203 found evidence to support an increasing relative risk of all-cause death for patients with one condition, two conditions and three or more conditions, for populations of patients from Finland, the Netherlands and Italy, respectively. In this exploratory analysis, it was assumed that patients’ risk of all-cause mortality would increase in proportion to the number of conditions. A range of pay-off times were calculated by varying the value for relative risk, assumed to increase with increasing numbers of conditions associated with multimorbidity. Five examples of values for the relative risk (1, 0.5, 2.0, 3.0 and 4.0) were used. Two values of DTD (0.005 and 0.010) were used as examples of harm from the treatment.

Patient vignette 1

The following text has been adapted from Thompson A, Guthrie B, Payne K. Using the ‘pay-off time’ in decision-analytic models: a case study for statins in primary prevention. Med Decis Making 2017. Published online 25 April 2017. 182 This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 3.0 License (http://www.creativecommons.org/licenses/by-nc/3.0/) which permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access page (https://us.sagepub.com/en-us/nam/open-access-at-sage).

This patient is a 60-year-old man whose cardiovascular risk is largely driven by age and sex. Table 23 shows that when there is no DTD the high-intensity statin treatment is a cost-effective option, with a positive net benefit of £2865 for a cost-effectiveness threshold of £20,000. Likewise, the absolute QALY gain was estimated to be 0.20 QALYs per patient, equal to approximately 73 days at full health. Figure 12 displays much of the same information as QALY profiles for this vignette. Of note is that even low DTD is associated with pay-off times measured in years, and the expected absolute QALY gain is sensitive to the presence of DTD. When the size of the DTD is set at ≥ 0.015 then statin treatment never pays off and the intervention is no longer cost-effective at a decision-maker’s threshold of £20,000 per QALY.

FIGURE 12.

Quality-adjusted life-year profiles for patient vignettes 1–3 with four levels of DTD (harm) from treatment (0–0.015 QALYs annually). (a) Patient vignette 1: 60-year-old man with 10% 10-year cardiovascular risk; (b) Patient vignette 2: 60-year-old man with 15% 10-year cardiovascular risk; and (c) Patient vignette 3: 70-year-old man with 20% 10-year cardiovascular risk. This figure has been adapted from figure 3 in Thompson A, Guthrie B, Payne K. Using the ‘pay-off time’ in decision-analytic models: a case study for statins in primary prevention. Med Decis Making 2017. Published online 25 April 2017. 182 This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 3.0 License (http://www.creativecommons.org/licenses/by-nc/3.0/) which permits non-commercial use, reproduction and distribution of the work without further permission provided the original work is attributed as specified on the SAGE and Open Access page (https://us.sagepub.com/en-us/nam/open-access-at-sage).

Patient vignette 2

This patient is a 60-year-old man whose 10-year cardiovascular risk of 15% is largely driven by age, sex, smoking and lipid profile. Table 23 shows that when there is no DTD the high-intensity statin treatment is a cost-effective option, with a positive net benefit of £4936 for a cost-effectiveness threshold of £20,000. Likewise, the absolute QALY gain was estimated to be 0.29 QALYs per patient, equal to approximately 106 days at full health. Figure 12 shows the QALY profile for this vignette. Of note is that even low DTD is associated with pay-off times measured in years (although, as expected, pay-off times are lower and absolute QALY gains are greater when baseline risk of CVD increases), and the expected absolute QALY gain is sensitive to the presence of DTD. When the size of the DTD is set at ≥ 0.020 then statin treatment never pays off and the intervention is no longer cost-effective at a decision-maker’s threshold of £20,000 per QALY.

Patient vignette 3

This patient is a 70-year-old man whose 10-year cardiovascular risk of 10% is driven by age, sex and lipid profile. Table 23 shows that when there is no DTD the high-intensity statin treatment is a cost-effective option, with a positive net benefit of £3847 for a cost-effectiveness threshold of £20,000. Likewise, the absolute QALY gain was estimated to be 0.22 QALYs per patient, equal to approximately 79 days at full health. As there is no harm included in the model the pay-off time is within the first cycle (year) and not relevant. Figure 12 shows the QALY profile for this vignette. Of note is that even low DTD is associated with pay-off times measured in years (although again, as expected, pay-off times are lower and absolute QALY gains are greater when baseline risk of CVD increases), and the expected absolute QALY gain is sensitive to the presence of DTD. When the size of the DTD is set at ≥ 0.020 then statin treatment never pays off and the intervention is no longer cost-effective at a decision-maker’s threshold of £20,000 per QALY.

Competing risks of death in patient populations with multimorbidity

Table 24 shows the results for a hypothetical patient population with an assumed baseline QRisk2 score of 10%, with different levels of multimorbidity represented by increasing levels of relative risk of death from all-cause mortality. Two analyses are presented for two assumed levels of DTD. Figure 13 shows the equivalent QALY profiles for each assumed level of DTD, 0.005 and 0.010.

FIGURE 13.

Quality-adjusted life-year profile for patients with different levels of total mortality due to varying multimorbidity (expressed as relative risk of total mortality) assuming a DTD of 0.005 and 0.01. (a) Absolute QALY gain for a 60-year-old man with 10% 10-year CVD risk and DTD 0.005; and (b) Absolute QALY gain for a 60-year-old man with 10% 10-year CVD risk and DTD 0.01.

Making an adjustment to the assumed relative risk for all-cause mortality has a large impact on the absolute QALY gain. In scenario 1, when the DTD is set at 0.005, for individuals with a fourfold increased risk of all-cause mortality, the absolute QALY gain is estimated to be fewer than half of individuals with unadjusted all-cause mortality (0.12 vs. 0.05). In scenario 2, when the DTD is set at 0.010, the absolute QALY gain for the base-case analysis, with no adjustment to the risk of all-cause mortality, is now much lower, at 0.05. In scenario 2, the absolute QALY gain is estimated to be zero in individuals with a fourfold increased risk of all-cause mortality. In scenario 2, individuals never acrue net benefit from treatment unless there is a reduced all-cause mortality of half that in the base-case analysis. In both scenarios, the adjustment on relative risk has no effect on pay-off time, which is the time to reach peak investment (in years).

Selection of the case study

The PRG was involved in the selection of the case study to model the impact of including more than one condition in a model-based CEA suitable for informing a CG. The PRG was guided by three pre-defined criteria to select a relevant case study. First, there must be an accessible existing model-based CEA that has been used to inform a CG produced by NICE. Second, to make the modelling approach feasible, two conditions that are commonly comorbid should be selected. Third, the conditions to be considered should occur together with sufficient frequency to provide a clinically relevant scenario of multimorbidity. On this basis, and after extensive discussion at the first meeting of the PRG, the case study selected was pharmacological interventions in patients with depression and CHD. In CG91, patients who had not benefited from low-intensity psychosocial interventions were recommended an antidepressant (usually a SSRI) or high-intensity psychological treatment. 17 The discussion within the PRG acknowledged that much of the specific guidance provided in the CG for patients with depression and chronic physical conditions (CG91) on the selection of antidepressants was contingent on the side-effect profile of the antidepressants and the potential interacting effects the antidepressants had on the patient’s chronic physical health condition. CG91 also recommended not using SSRIs where patients were also likely to be using medications for CHD, such as warfarin and other antiplatelets or anticoagulants, because of potential antiplatelet effects. Moreover, given the growing literature investigating the impacts of depression on CHD outcomes and vice versa, there was thought to be a sufficient pool of evidence on which to generate exploratory economic analysis and cost-effectiveness results investigating some of the interactions between these conditions.

Scope of the model-based cost-effectiveness analysis

Box 2 summarises the research question and scope for the selected base-case analysis for the model-based CEA.

Modelling approach

This study used a DES model with a lifetime horizon to capture the costs and benefits of antidepressant treatment for patients with depression at risk of CHD (some of whom will go on to develop CHD). Caro209 and Caro et al. 210 provide useful discussions of the advantages of DES models.

Economic models used in the context of model-based CEA can be categorised in many ways, but one useful way is the aggregation level of the object of interest (the patient). Generally there are two classifications. The first is the cohort approach (used in the NICE CG90 model assessing the cost-effectiveness of 12 new-generation antidepressants), whereby the patient population as a whole is modelled at an aggregate level. 45 The patient population is described as having certain characteristics that affect the transitions within the models between health states. By making the population homogeneous (example characteristics: male, aged 65 years, smoker), transition probabilities will be that of the mean profile conditional on the various characteristics. The vast majority of models take a cohort approach. Decision trees and Markov models are usually cohort based. A problem with these cohort-type models is that they lack the flexibility to readily incorporate parameters such as risk when the inputs are not usually normally correlated (they are skewed), or to incorporate patient characteristics that may interact or correlate.

A key problem pertinent to cohort models is the way that time is captured. In decision trees the time element associated with events and treatments is not adequately captured. In Markov models a time element can be captured but this is done by setting arbitrary cycle lengths within the model structure. Another key problem is the ability to incorporate memory within the model. When a patient cohort moves from one state to another within a Markov model, the process is blind to the previous disease history of the cohort. Therefore, a patient who has had three recurrent depressive episodes and is now in remission is bundled within the same health state as a patient who has had only one previous episode. The only way to avoid this omission, and capture the impact of memory within a Markov model, is to use tunnel states, which massively increase the complexity of the model yet still do not adequately capture the time element.

Rather than adopt a cohort approach, it is possible to model the individuals in the patient population. Using this approach means that each individual’s characteristics then determines his or her progress. Patient attributes can be tracked as individuals progress, dynamically changing the risk of moving between health states. Moreover, time is explicitly captured in the model, allowing patients to age within the model and again dynamically altering the risk structure between the various competing risks. There are some published DES models relevant to depression, but none has included another condition. 211–215 The characteristics of depression alone, and particularly the characteristics of depression alongside another condition, make DES the ideal modelling approach to address the stated research question.

Model conceptualisation

Model conceptualisation is a key stage in developing a model structure. This study followed recommendations published by the International Society for Pharmacoeconomics and Outcomes Research/Society for Medical Decision Making and also referred to in a NICE DSU document on model conceptualisation. 183,216 The process of conceptualisation followed a problem-orientated approach. The problem-orientated approach is intended to elicit current clinical knowledge of the relevant characteristics of the disease as well as the important events associated with the disease. Once the important aspects have been captured, the model conceptualisation moves on to how best to capture these aspects in a mathematical model structure. This process generally includes a review of existing modelling structures relevant to the research question.

Systematic review studies were identified that investigated the use of economic models for depression. 212,215,217 The most recent review, by Ali Afzali et al. ,218 investigates 14 depression models from the literature and finds evidence of substantial heterogeneity in model structures and types, techniques employed, time horizons and data sources used. The authors identified two key potential problems arising in some of the models reviewed: first, a short time horizon for analysis, particularly where decision trees were employed, which limited the ability of the analysis to capture long-term effects of the interventions (e.g. on suicide); second, the inability of some models to fully capture the natural course of depression in the health states (e.g. relapse and remission). Consequently Ali Afzali et al. 218 proposed to use a DES model with a lifetime horizon, which is in keeping with a growing body of model developers using discrete event models for cost-effectiveness modelling in depression. 212–215

Key attributes that were identified as being attractive for using a DES model for depression and CHD were the ability to:

• model long-term disease effects beyond the initial acute stage

• adjust the risk of relapse dependent on the number previous episodes (or CHD status) so that individuals within the model have memory

• incorporate competing risk more simply by modelling time to event

• capture adverse drug reactions more completely.

Following a review of the available DES models, the model by Vataire et al. 214 was considered to be the most relevant DES model for depression and was used as the basis for informing the structure and events within the proposed new model.

Cost-effectiveness models obtained through NICE were also used to help conceptualise both the depression and CHD aspects of the economic model. In particular, the NICE cost-effectiveness model investigating the use of statins for the prevention of CHD was considered in detail.

The key attributes of the depression and CHD model were then conceptualised to be:

• the natural course of the two diseases and how this could be translated into associated health states

• the impact that the health states have on HRQoL and costs

• the impact that the comparator has on the movement between the health states

• the potential interactions between the two diseases.

An initial meeting was conducted with the PRG to decide upon the research area for the CEA. Follow-up meetings were used in order to provide peer-review to the model structure, to ensure relevant attributes were included and to identify relevant sources of data to populate the model. The conceptualisation of the model underwent numerous iterations to reflect the feedback from the clinical experts as well the limitations of the evidence base. Prior to each meeting, the model was explained using non-technical, non-mathematical language as well as through the use of graphical methods. In total, four meetings were held, and the final one concluded with a presentation of some preliminary results from the model-based CEA.

Model overview

The cost-effectiveness research question was answered by structuring a DES model developed using Simul8 software (version 2014; Simul8 Corporation, Boston, MA, USA). The model tracks patients with depression who go on to develop CHD and so includes events linked to both disease types. Current non-DES economic models for depression typically use a time frame ranging from 6 to 15 months. The time horizon used in this analysis was selected to capture all relevant benefits and costs for depression and CHD (and varied from 1 year through 5 and 10 years to lifetime). The perspective adopted was that of the NHS and personal social services, in keeping with the NICE reference case. 59 Life expectancy statistics were sourced from ONS interim life tables. 202 Patients were followed up from initial treatment until death, which has multiple potential causes (all-cause, suicide, CHD mortality), or until the time horizon is reached. Patients with depression were assumed to go through periods of response and remission and to be at risk of relapse, and were also assumed to be at risk of developing CHD with its associated costs and its impact on ADE probability. Patients treated for CHD were assumed to have a higher risk of an ADE from the antidepressant.

Model structure

Figure 14 shows the general model structure. Patients (male and aged 60 years) enter the model when they experience a new episode of major depression for which they are going to be offered treatment with an antidepressant following a GP consultation.

FIGURE 14.

Schematic of the overall model structure.

Treatment effectiveness and adverse drug reactions

Patients were assumed to either respond, or fail to respond, to the first selected antidepressant (first-line intervention) within 8 weeks of starting treatment. Response probabilities for first-line intervention were taken from a published network meta-analysis conducted by Cipriani et al. ,219 which was also used in the economic analysis conducted for NICE CG90. These data were selected because (1) they were deemed to be of high quality, (2) they were consistent with the NICE guideline and (3) they provided a set of parameters for all the comparators. In the original network meta-analysis, the population was not described explicitly as including patients with both depression and CHD, but for this analysis it was assumed that the response rates to antidepressants would not be affected by the existence of the second condition (CHD) in the population. Uncertainty around the point estimates for response was represented by using normal probability distributions making use of the upper and lower CI for the treatment probabilities in the PSA (see Table 25).

A rapid review was conducted to find the most suitable data on utilities and combined with the evidence synthesised from the systematic review by Ali Afzali et al. 218 Patients who had responded to their treatment were assumed to experience an increase in their health status that was reflected by a higher utility value. Table 26 summarises the utility values used in the model.

Figure 15 illustrates that the model assumed patients could experience an ADE following either non-response or response to a treatment. ADEs that did occur were assumed to be experienced within the first 4 weeks of treatment, prompting a repeat GP consultation at this time. The probability of any ADE was estimated using evidence on dropout rates for the different types of antidepressants from Cipriani et al. 219 as a proxy measure. A priori an assumption made was that patients who have had a CHD event were more likely to be on concomitant treatment and, therefore, the chance of any ADE was likely to be higher. For patients who had undergone a CHD event, the probability of any ADE (see Table 25) was sourced from an expert elicitation exercise, which used the Sheffield Elicitation Framework technique223 to elicit values from members of the PRG. Appendix 4 summarises the design and analysis of the expert elicitation exercise.

FIGURE 15.

Schematic of the ‘response’ section of the model. ADR, Adverse Drug Reaction.

For patients who respond, and who have an ADE, the model assumed that patients would require an additional GP consultation in which the GP had the choice of (1) switching, (2) changing the dose or (3) keeping things as they are. Appendix 5 summarises the assumed probabilities for these events.

The model also accounted for the probability of patients discontinuing treatment following an ADE (see Appendix 5). Patients who discontinued treatment were assumed to have a period of down-time (6 months) in which they did not interact with the health-care system. For those who did not respond to treatment, an assumption was made that clinicians could not keep things as they were but must change treatment (to start a second-line treatment). It was assumed that the second-line antidepressant has the same clinical effectiveness as first-line treatment.

If a patient experienced an ADE, a temporary disutility was used to reflect the decrease in health status associated with that adverse event (Table 27). The disutility was assumed to last for a 4-week period. The ADE disutilities were applied additively.

Subsequent response and recovery

Patients who responded following either a first or a second line of treatment were assumed to have an increase in their HRQoL. Patients were also assumed to be at risk of a relapse of their depression. Time to relapse was modelled as being dependent on the number of previous episodes of depression the patients had experienced. Weibull curves were fitted to evidence taken from a 10-year multicentre naturalistic study to simulate this effect. 225 Figure 16 shows the diminishing expected time it was assumed to take for relapse to occur with each increasing number of episodes of depression. Patients who did not relapse within 6 months were said to be in the state of ‘recovery’. The utility value for the recovery was set as the same value as response.

FIGURE 16.

Time to relapse. Source: Solomon et al. (2000). 225

Long-term events

Patients were assumed to be at risk of a long-term event. This risk was pre-specified at the beginning of the model and took account of (1) all-cause mortality (excluding suicide and CHD death), (2) suicide and (3) a CHD event. The likelihoods of all-cause mortality and suicide were taken from ONS interim life tables and causes of death statistics. 202 Suicide risk for depressed patients was assumed to be twice that of the general population.

The baseline risk and the type of first CHD event were taken from Ward et al. 201 (Table 28). The potential CHD health states were:

1. stable angina

2. unstable angina

3. MI

4. TIA

5. stroke

6. heart failure.

Following a first CHD event, the time to the next secondary CHD event was sampled from a transition matrix of time-to-event data. The type of secondary event CHD probabilities and time-to-event data were adjusted using 5-year age bands as reported by Ward et al. 201 and the shortest time to event was selected (see Appendix 6 for the transition matrix).

The model accounted for the impact on the patient’s health status measured in utility values from experiencing a CHD event. Where patients experienced both a CHD event and a depression state they were allocated a joint health state utility value. To calculate the joint health state utility value, the utility value for the depressed health state (depression, response) was combined with the CHD health state utility score using the additive method as detailed by Ara and Wailoo. 226 Currently there is no best-practice recommended method for combining joint health state utility values204 so the additive method was chosen for simplicity.

Resource use and costs

The model assumed a NHS and personal social services perspective. Costs used were from the price year 2013/14. The key items of resource use and costs in the model are described in Table 29 for depression and Table 30 for CHD. The costs associated with time spent within CHD health states were sourced from the assumptions made within NICE CG181. 148 Costs associated with depression were taken from the resources used by patients in attending GP appointments (first and follow-ups) as well as the ongoing antidepressant cost.

The model assumed that there was no cost burden associated with ADEs apart from the additional GP consultation, with the exception of patients who have a GI bleed, who were assumed to cost an additional £2850 because of the need for in-hospital and post-discharge care associated with bleeds. 229

The cost for each CHD event was calculated by multiplying the time spent in the health state with the per-day unit cost of the health state. These costs were adjusted according to whether the CHD health state was in its first year or subsequent follow-up years (see Table 30).

Model logic

Patients were assumed to move through the events in the model, and experience the response and recovery periods as specified. The associated QALYs and costs were tracked for each event throughout the model. Patients were also assumed to be at risk of long-term CHD events, all-cause mortality and suicide. If the cumulative time a patient spent within the model exceeded the shortest of the long-term events, then a patient was assumed to move to these states. Patients were assumed to leave the model if their time in the model exceeded the shortest of the long-term events (all-cause mortality, suicide, CHD death) or if the time horizon was exceeded.

Calculation of quality-adjusted life-years, costs and net benefit

A total of 5000 patients were simulated moving through the model. This number of patients was used to allow stability in the model outputs. Each patient’s QALYs and costs were calculated. Costs and QALYs were discounted at 3.5% equally. Incremental net benefit was calculated using the formula for a cost-effectiveness threshold (λ) of £20,000:

Model verification, structural stability and sensitivity analyses

The recommendations contained within the NICE DSU document on DES model was used to inform model verification processes. 230 In particular, the model was verified by:

1. stepping through the experience of individual patients within the model

2. recording interim outputs

3. comparing the results with ex-ante expectations for face validity

4. comparing deterministic and probabilistic results

5. internal peer-review by the analyst responsible for building the model

6. discussions with clinical experts within the PRG and research team on the model structure.

The model was run separately for each comparator. For the deterministic analysis, the mean value for each parameter was calculated to capture mean QALYs, costs and net benefit. For the PSA, pre-defined parameters within the model had an associated distribution from which a value was sampled in order to capture uncertainty. For the PSA, 1000 samples (bootstrap replicates) were completed, as convergence in the results was assumed to occur at this point.

Exploratory scenario analyses: impact of disease–disease interaction

Two exploratory scenario analyses were conducted to account for an interaction between CHD and depression. Conducting these scenario analyses demonstrates the potential flexibility of the discrete event model to allow a range of scenario analyses to be performed.

Several prospective studies have shown evidence of the predictive role of depression to determine the onset of CHD. This evidence base suggests that depression may be an independent risk factor for CHD. 231,232 There is also evidence of depression having an unfavourable impact on patients with new-onset CHD in terms of their risk of mortality and future CHD events. In theory, therefore, the effective treatment of depression, using for example a SSRI, could potentially improve CHD outcomes including mortality risk through at least two potential mechanisms:233

1. behavioural – for example, an improvement in depression could improve smoking habits, medication adherence, alcohol consumption or physical activity, which could improve CHD outcomes

2. physiological – SSRIs are thought to attenuate platelet function, which therefore leads to cardiac benefits.

The first scenario analysis assumed no risk of CHD in the population (scenario 1) and the second scenario analysis assumed that treatment with a SSRI antidepressant lowered the risk of developing CHD by 5% (see scenario 2). Table 33 presents the results for two most cost-effective treatment options identified in the base-case analysis: citalopram and sertraline.

Calculation of absolute quality-adjusted life-years: an exploratory analysis

A third exploratory analysis was conducted to identify the potential gains in absolute QALYs from the two treatment options identified as being the most effective options in the base-case analysis. This analysis involved redefining the relevant comparator to be ‘do-nothing’. To calculate the absolute QALYs for the comparators it was necessary to make an assumption on some of the key parameters for the do-nothing treatment option. For simplicity the model structure was kept the same but the values for the probability of response to treatment and probability of an ADE were changed. The new values were assumed to be 0.26 for the probability of response within 8 weeks and 0.15 for the probability of an ADE. Absolute QALYs were calculated by subtracting the QALYs that would have been received without any treatment from those gained from treatment with either citalopram or sertraline. Table 34 shows the results from the deterministic analysis and also incorporates two scenario analyses (scenario analysis 1 and scenario analysis 2).

Figure 19 shows the absolute QALYs gained for sertraline compared with a do-nothing approach for different assumptions regarding the model time horizon and for the base case and two scenario analyses (scenario analysis 1 and scenario analysis 2). For the base-case analysis, it can be seen that most of the absolute QALYs gain occurred within the first 5 years of treatment. The gain in absolute QALY then falls as the time horizon increases, for two likely reasons: (1) the smaller population size as patients begin to leave the model because of the competing risk of the CHD and (2) the diminishing effect that discounting has on the quality-of-life gains.

FIGURE 19.

Absolute QALYs gain for sertraline for the base case and two scenario analyses with different model time horizons.

In scenario 1, it was assumed that there was no competing risk of CHD included in the model. The results of this analysis give the largest absolute QALY gains, indicating the negative impact that the competing risk of CHD has on the potential for patients to accrue QALY gains. This persists from a 1-year time horizon up until the lifetime horizon.

In scenario 2, the absolute QALY gains were calculated assuming that there is a risk of CHD as in the base-case analysis but the antidepressant reduces the risk of developing CHD. There was very little difference between the results in the base case and for this scenario until the model time horizon reaches lifetime, in which case the reduction in risk starts to lower the impact of having CHD on the absolute QALY gained.

The use of epidemiological data to characterise the guideline population

In contrast to the very systematic identification and synthesis of evidence relating to treatment benefit or diagnostic accuracy in guideline development, characterisation of the guideline population is more haphazard. GDGs do already sometimes qualify recommendations in terms of comorbidity and highlight some interactions, but as we understand it this is largely driven by the knowledge and expertise of the individual GDG members who happen to have been recruited. There is now reasonably straightforward access to large and representative UK epidemiological data sets. These can be used to characterise the population for which guideline recommendations are being made. It is therefore feasible to examine more systematically the extent of the extrapolations being made and the interactions that are likely to happen when making treatment recommendations to inform GDG decisions about if and how these recommendations should be qualified.

Guideline development groups would then have to consider whether they wish to make a single recommendation for all people with a condition, or stratified or otherwise qualified recommendations for different subgroups. Such judgements already happen but we believe that systematic use of epidemiological data to inform them is required. Based on discussion with the PRG, factors that GDGs might consider when making such judgements about drug–drug and drug–disease interactions include (Figure 20):

• how common coprescription is likely to be in the guideline population (based on whether drugs are recommended for all patients with the condition or a subset, and on how commonly interacting drugs are used in the guideline population), or how common interacting conditions are in the guideline population

• the severity of any interaction in terms of the harm that the interaction causes

• how common the interaction is in people coprescribed two drugs or prescribed a drug in the presence of an interacting condition.

FIGURE 20.

Using epidemiological data to inform single-disease guideline development.

Based on discussion with the PRG, for issues of applicability and extrapolation from trial populations to important groups of the guideline population who would have been excluded from trials, factors that GDGs might consider when making such judgements include:

• the nature of the treatment, in terms of whether or not its mechanism of effect is likely to apply across all patients, and its potential for harm

• the duration over which the treatment will be used, which is relevant when extrapolating to populations with limited life expectancy from other conditions, age or general frailty

• the absolute size of the observed benefit in the trials, which is relevant because large benefits are less likely to be sensitive to small variations in benefit or harm in people not eligible for trials

• the nature of the differences between trial and non-trial populations, including age, comorbidity, coprescribing and likely life expectancy, in terms of whether or not these are large enough to matter in the context of the previous factors.

Such considerations are particularly likely to apply when the outcomes being improved by treatment are not observable by clinicians or patients. For example, clinicians and patients can observe change in pain during treatment with an analgesic. In contrast, most preventative treatments require clinicians and patients to take it on trust that meaningful outcomes are better because a prevented heart attack or other prevented future event is not observable in an individual.

The creation of measures of absolute benefit to allow comparison of net benefit across treatments for different conditions

There are no major technical barriers to using a consistent method to produce estimates of the absolute benefit of treatments for different conditions across a range of clinical outcomes. However, all such estimates rely on making a number of significant assumptions, including that relative risk of benefit is constant across populations, that competing risks of death are not significant, that baseline risk has been accurately measured in the guideline population or its important subgroups and that the relative risk of harm is constant across populations. All of these assumptions will be untrue at least sometimes, but it is important to note that any clinician who prescribes a treatment to a patient who would not have been eligible for the original trial is making some or all of these assumptions implicitly. Similarly, there are no technical barriers to using absolute QALYs to compare the benefit of treatments, although again only provided that guiding principles are followed to maximise comparability. In practice, an absolute QALY approach will be restricted in the short term by the relative lack of suitable economic models for many clinical interventions. In both cases, we believe that absolute benefits should be estimated only for interventions with demonstrated clinical benefit at a minimum, and ideally cost-effectiveness.

The NICE multimorbidity guideline due to be published in September 2016 will develop a method for extracting absolute benefit information from existing NICE guidelines,42 which is likely to form the core of a resource to compare the absolute benefit of treatments for different conditions. In practice, we believe that this will be of most value when the benefit of treatment is not observable by clinicians and patients. For treatments of symptoms, irrespective of the average benefit in the population, both clinician and patient can at least partly judge if treatment is successful. In contrast, for treatments that reduce the risk of future events, treatment decisions are largely informed by trial evidence, and it is in this context that we believe that comparisons of the absolute benefit of different treatment would be most useful, to inform decision-making shared with people with limited life expectancy and/or significant treatment burden.

However, current ways of delivering guidelines using downloadable versions, electronic pathways online and paper-based decision support tools will not make the large amounts of information about absolute benefit easily usable, and any absolute benefit comparison resource will require regular refreshing as guidelines are updated. We therefore believe that effective delivery will require the creation of a suitable electronic resource where clinicians and patients can choose which conditions and which treatments to compare, can calculate baseline risk or choose plausible values for it, and can choose how to view absolute benefit (in words, in numbers of various kinds or graphically). Creating and evaluating such a resource was beyond the scope of this project, but will probably require investment from either a research funder or a national decision-making body such as NICE.

The use of alternative output of model-based cost-effectiveness analysis to help guideline development groups examine the implications of time to benefit and competing risks

Our empirical study applied the concepts of pay-off time, DTD and graphical QALY profiles from an existing model-based CEA used for the production of CGs for the primary prevention of CVD with statins. These three concepts have the potential to be an informative approach to include a temporal dimension in model-based CEA. The calculation of pay-off time together with using visual presentation of the information (QALY profiles) provides a potentially useful tool to complement existing cost-effectiveness results and potentially aid guideline decision-making when choosing between interventions with different pay-off times, or considering the implications of estimated pay-off times in people with limited life expectancy.

For surgical or screening interventions with clear upfront harms, pay-off time is clearly relevant, and we believe it should be actively considered by GDGs by examination of QALY profiles. For long-term drug treatments, pay-off time is more driven by the extent to which DTD applies. However, in this context, even small harms may mean that pay-off times exceed the life expectancy of important subgroups of the population given the presence of life-limiting comorbidity.

The method can be easily and rapidly applied using current model-based CEA. The pay-off time can be stratified according to various baseline characteristics that are pre-built into economic models, and it can be calculated as a pay-off time in purely QALY terms or it can also be inclusive of financial costs associated with an intervention by using net benefit. Another potential extension, which would make the pay-off time informative in the context of patients with comorbidity, is to explore the impact of adjusting the relative risk of all-cause mortality, which is generally a parameter included in Markov model-based CEA. In this scenario, the adjustment in the relative risk of all-cause mortality is used as a proxy for a patient population with competing risk of death from comorbidities.

Generating estimates of the pay-off time, using the QALY or QALY inclusive of costs (net benefit) form, could therefore be used by GDGs as a complement to the outputs from existing model-based CEA already used to inform NICE CGs. Although there have been some empirical studies of it, more research is needed to better quantify DTD using well-designed stated preference studies in appropriate samples of the patient population and members of the public. In the interim, the use of pre-defined DTD could enable exploration in sensitivity analysis of the potential impact if patients do associate some level of harm with taking a long-term treatment. A series of one-way sensitivity analyses could therefore be used to present the impact of plausible levels of DTD, and also the potential impact of comorbidity on mortality risk. Practically, we believe that pay-off time should be explored only in the context of interventions that have already been shown to be a cost-effective use of resources in the main model-based CEA.

Peer-reviewed papers

Guthrie B, Payne K, Alderson P, McMurdo MET, Mercer SW. Adapting clinical guidelines to take account of multimorbidity. BMJ 2012;345:e6341.

Dumbreck S, Flynn A, Nairn M, Wilson M, Treweek S, Mercer SW, et al. Drug–disease and drug–drug interactions: systematic examination of recommendations in 12 UK national clinical guidelines. BMJ 2015;350:h949.

Thompson A, Guthrie B, Payne K. Do pills have no ills? Capturing the impact of direct treatment disutility. Pharmacoeconomics 2016;34:333–36.

Thompson A, Guthrie B, Payne K. Using the ‘pay-off time’ in decision-analytic models: a case study for statins in primary prevention. Med Decis Making 2017. Published online 25 April 2017.

Presentations

Alderson P. Addressing Multimorbidity in Clinical Guidelines: Experience From NICE and Future Plans. Ariadne International Symposium, Frankfurt, Germany, October 2012.

Guthrie B. Polypharmacy and Changing Emphasis Towards the End of Life. British Geriatrics Society autumn meeting, Dunfermline, UK, October 2013.

Guthrie B. New Approaches for Clinical Trials and Guidelines. REPOSI International Seminar, Aging, Multimorbidity and Polypharmacy: Strategies for the Third Millennium, Milan, Spain, September 2013.

Dumbreck S, Flynn A, Guthrie B. Better Guidelines for Better Care: Accounting for Multimorbidity in Clinical Guidelines. Poster, SIGN 20th Anniversary Event, Glasgow, UK, December 2013.

Guthrie B. How Useful are Evidence-Based Guidelines to Primary Care? DECIDE international Conference, Edinburgh, UK, June 2014.

Thompson A, Payne K, Nairn M, Alderson P, Sutton M, Guthrie B. Accounting for Multimorbidity in Economic Models: Implications for Guideline Development. Guidelines International Network Conference, Melbourne, Australia, August 2014.

Thompson A, Payne K. Economics and Multimorbidity. NICE Evidence Synthesis Network, Economics Event, Manchester, UK, June 2014.

Thompson A, Payne K. Economics and Multimorbidity: The Context. NICE Evidence Synthesis Network, Economics Event, Manchester, UK, June 2014.

Guthrie B, Thompson A, Payne K. Multimorbidity in Guidelines. NICE Technical Workshop, Manchester, UK, May 2015.

Guthrie B. Multimorbidity in Guidelines. Briefing document and oral presentation, SIGN Strategy Group, Edinburgh, UK, June 2015.

Guthrie B. Multimorbidity in Guidelines. Briefing document and oral presentation, NICE Board Strategy day, London, UK, June 2015.

Guthrie B. The Challenges of Making Guidelines in a World of Multimorbidity. SIGN/Health Improvement Scotland, Edinburgh, UK, June 2015.

Guthrie B. Multimorbidity: New Paradigm of the Emperor’s New Clothes? Society for Academic Primary Care Conference, Oxford, UK, July 2015.

Dumbreck S, Flynn A. Comparing Treatment Effectiveness Across Guidelines to Support Decision Making for People with Multimorbidity. Guidelines International Network Conference, Amsterdam, the Netherlands, October 2015.

Thompson A, Payne K. Including a Temporal Dimension in Model-Based CEA: An Application to Clinical Guidelines for the Use of Statins. Guidelines International Network Conference, Amsterdam, the Netherlands, October 2015.

Guthrie B. Multimorbidity in Guidelines. Guidelines International Network Conference, Amsterdam, the Netherlands, October 2015.