The REFER (REFer for EchocaRdiogram) study: a prospective validation and health economic analysis of a clinical decision rule, NT-proBNP or their combination in the diagnosis of heart failure in primary care

Article history

The research reported in this issue of the journal was funded by the EME programme as project number 09/160/13. The contractual start date was in July 2011. The final report began editorial review in August 2015 and was accepted for publication in July 2016. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The EME editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

FD Richard Hobbs has received grants from Roche Diagnostics outside the submitted work. The work was supported by funding from the National Institute for Health Research Efficacy and Mechanism Evaluation programme. Roche Diagnostics provided the N-terminal pro-B-type natriuretic peptide testing equipment but did not have any influence on study design, conduct or reporting.

Copyright statement

© Queen’s Printer and Controller of HMSO 2017. This work was produced by Taylor et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

Heart failure diagnosis

Heart failure is a chronic disease associated with significant mortality and poor quality of life for patients. 3–5 Making an accurate and timely diagnosis is crucial and requires referral for objective testing but deciding who to refer can be challenging. 6 The symptoms of heart failure are often non-specific and include gradual-onset breathlessness, fatigue and ankle swelling. 7 However, these symptoms are not unique to heart failure and can be associated with other conditions or patients may have several coexisting diseases. 8–10 Patients with symptoms that may be associated with heart failure often present to primary care. Identifying the patients likely to have heart failure, and therefore requiring referral for diagnostic testing, can be difficult. 11,12

Clinical decision rules

Clinical decision rules (CDRs) can help clinicians to assess the probability that a patient has a particular condition. 13 They are used widely in medicine to inform decisions about investigation and management. For example, the Ottawa ankle rule can be useful when deciding if a patient with an ankle injury is likely to have a fracture and, therefore, whether radiography would be beneficial or unnecessary. 14 Similarly, the CHA2DS2-VASc score can be used in patients with atrial fibrillation to determine the likely benefit of anticoagulation to reduce the risk of stroke. 15 There is not currently a CDR for patients with heart failure presenting to primary care.

The ‘MICE’ rule was developed from a systematic review of the evidence for symptoms of heart failure. 16 The review identified 11 prospective heart failure studies set in primary care. The decision rule was derived from an individual patient data set from one of these studies (Zaphiriou et al. 17) and externally validated on four others that included relevant variables. 18–21

The MICE rule comprised four clinical elements – Male, history of myocardial Infarction, Crepitations at the lung bases and oEdema – and was combined with natriuretic peptide levels to identify those likely to have heart failure and who should be referred for further diagnostic testing. The elements of the CDR are shown in Box 1.

Natriuretic peptide testing

Natriuretic peptides can be used to identify those patients with symptoms who have an increased likelihood of heart failure. 22 B-type natriuretic peptide (BNP) and N-terminal pro-BNP (NT-proBNP) are released from the chambers of the heart in response to pressure or fluid overload, acting on the kidneys to induce a diuresis and on the vasculature to cause both arterial and venous dilatation, thereby reducing preload and afterload. Heart failure is associated with an increased natriuretic peptide level. A rising natriuretic peptide level can be an early sign of heart failure; however, other factors such as renal impairment and angiotensin-converting enzyme (ACE) inhibitors can influence the results, which therefore need to be interpreted in light of these other factors. The European Society of Cardiology (ESC)6 suggests a NT-proBNP threshold of 125 pg/ml to rule out heart failure, whereas the National Institute for Health and Care Excellence (NICE)7 in England suggests a NT-proBNP threshold of 400 pg/ml.

Aim of the study

This study aimed to assess the performance of the CDR, NT-proBNP and their combination in identifying patients with heart failure presenting to primary care.

Participants

The REFER study used a prospective, observational, diagnostic validation design to assess the performance of the MICE rule and NT-proBNP levels in identifying patients with heart failure. The study population was primary care patients aged ≥ 55 years presenting with recent new-onset shortness of breath, lethargy or peripheral ankle oedema of > 48 hours’ duration for which there was no other obvious cause. Patients were excluded if they were unable to give consent or had a previous confirmed diagnosis (i.e. with objective evidence) of heart failure, an obvious alternative diagnosis, severe symptoms requiring immediate management or recent (within 60 days) acute coronary syndrome.

Participants were recruited from 28 general practices around the West Midlands region of the UK (central England) based on their presenting symptoms. Participating practices were asked to invite all presenting patients who met the inclusion criteria to join the study consecutively. If a patient expressed an interest in taking part, the general practitioner (GP) completed an online portal with contact details and printed out a patient information sheet. Patients could choose whether to contact the study team using the designated telephone number or provide their own number to be contacted by a member of the study team. An appointment for participants to attend the research clinic was arranged during the initial telephone call.

Test methods

Participants attended the research assessment clinic within 7 days of recruitment. The clinics were held at two practices – one in the north and one in the south of the region. Assessments were carried out by trained research nurses and a British Society of Echocardiography (BSE)-accredited echocardiographer. The purpose and process of the study was explained and written consent was taken from patients. A research nurse then completed a clinical record form containing a clinical history and examination findings. Blood was sampled for NT-proBNP testing and renal function. Two attempts at blood-taking were allowed. The NT-proBNP level was determined using a point-of-care device (Roche Diagnostics, Burgess Hill, UK). The echocardiographer then carried out an electrocardiogram (ECG) and an echocardiogram. Finally, participants were asked to complete two quality-of-life questionnaires: the Short Form questionnaire-12 items (SF-12) (version 1) and the European Quality of Life-5 Dimensions (EQ-5D) (three-level version). The SF-12 is a validated questionnaire that has been used to measure health-related quality of life. 24 The EQ-5D focuses on five domains: mobility, self-care, activity, pain and anxiety. 25

The test results were faxed back to the GP on the day of the research assessment clinic. Patients were encouraged to make a follow-up appointment with their GP to discuss the results and allow any further management to be arranged.

The reference standard was an expert consensus panel of three cardiology specialists, who reviewed each case blinded to the assessments by other panel members. The ESC 2012 guideline was used to define heart failure. 6 The panel was presented with clinical information and investigation results in three separate stages to assess incorporation bias. At step 1 the clinical assessment (excluding the MICE variables) and ECG and echocardiogram findings were presented. At step 2 the CDR components (male, history of myocardial infarction, crepitations and oedema) were added and, finally, at step 3 the NT-proBNP result was included. The cardiology specialists were asked to record whether the patient did or did not have heart failure at each of the three steps. At least two cardiologists needed to agree the diagnosis independently at each step before the panel could move on to the next step. In the case of disagreement a third cardiologist adjudicated the case, blind to the other panel members’ assessments, before the panel could move to the next step.

Statistical methods

A sample of 500 symptomatic patients attending their GP for breathlessness, lethargy or ankle swelling was proposed. This sample size was sufficient to estimate the sensitivity of the CDR to within 4% and specificity to within 6% at the 95% confidence level. Calculations were based on a sensitivity of 94% and specificity of 48% obtained from the previous individual patient data meta-analysis22 and prevalence of heart failure in a symptomatic population of 30%.

Participants with and without a diagnosis of heart failure at step 3 were compared using independent t-tests or Wilcoxon rank-sum tests for continuous measures and chi-squared tests for categorical variables. The main outcome measures were test performance of the CDR and natriuretic peptide test – alone and in combination – in estimating a diagnosis of heart failure. The findings of the expert consensus panel determined whether heart failure – the ‘observed disease’ – was present or absent. The CDR and NT-proBNP results were also used to determine whether heart failure was likely to be present – the ‘test disease’ – and referral for echocardiography would have been indicated. Observed compared with test disease status was then cross-tabulated to determine the sensitivity and specificity, positive predictive value (PPV) and negative predictive value (NPV) for the CDR, NT-proBNP and CDR and NT-proBNP in combination and also by the NT-proBNP cut-off points of 125 pg/ml and 400 pg/ml suggested by the ESC⁴ and NICE⁵ guidelines, respectively; 95% confidence intervals (CIs) were calculated using the binomial exact method. Receiver operating characteristic (ROC) curves were generated to determine the overall discriminatory ability of each test in predicting a diagnosis of heart failure. Comparisons were made between the performance characteristics of the current cohort and those observed in the original derivation data set. 22

Participants

The recruitment phase of the REFER study started on 1 May 2011 and was completed on 31 August 2013. Participants were recruited from a random sample of 28 general practices in central England, stratified by practice list size and deprivation quartile. 26 Assessment was undertaken at the research clinic within 7 days of participants presenting to their GP.

Figure 1 shows a flow diagram of patient recruitment. In total, 397 patients were eligible for inclusion, of whom 45 were excluded: 37 did not provide consent, three did not attend the appointment, four lost contact with the study team and one was hospitalised before attending the study appointment. Of the 37 patients who did not provide consent, it was normally the case that they expressed an interest to their GP at the time of the consultation but either did not subsequently contact the research team or did not want to take part when the study team called to arrange an appointment at the assessment clinic. Of the 352 remaining participants, 48 did not have a blood test because of failed venepuncture or, in a small number of cases, a nurse not being available. These participants did not proceed to step 3 for the reference standard diagnosis of heart failure because of an unknown level of NT-proBNP and were excluded from the analysis presented in this paper.

FIGURE 1.

Flow diagram showing the number of participants in the REFER study. Step 1 = clinical information + ECG + echocardiogram; step 2 = CDR variables; step 3 = NT-proBNP result.

The remaining 304 participants formed our validation cohort; participants were similar to those excluded with respect to demography and medical history except for previous record of heart failure, with those without a NT-proBNP test having a higher prevalence (2.3% vs. 8.3%). These heart failure labels from the routine clinical records were, however, not necessarily confirmed with objective evidence or a formal diagnosis.

The most common presenting symptom was breathlessness; however, over half of the participants had all three symptoms of breathlessness, ankle oedema and lethargy, as shown in Figure 2.

FIGURE 2.

Venn diagram showing the presenting symptoms of participants.

The clinical and demographic characteristics of the study population are shown in Table 1. The mean age was 73.9 years [standard deviation (SD) 8.8 years] and 124 (40.8%) participants were male. The cohort had a mix of ethnicities, including 18.4% Asian or Asian British participants. Cardiovascular risk factors such as hypertension and diabetes mellitus were prevalent: 221 (72.7%) participants reported being hypertensive and 86 (28.3%) had diabetes mellitus. Comorbidities were common: 183 (60.2%) participants had arthritis and 73 (24.0%) had depression. Seven participants had a record of previous heart failure but this was not confirmed with objective evidence or a formal diagnosis. Cardiovascular medications were commonly prescribed, possibly relating to the high rate of hypertension in the cohort.

Comparison with the clinical decision rule derivation data set

The REFER cohort, although similar in age and sex to the derivation data set,17 had fewer referrals because of shortness of breath and more referrals because of ankle oedema and lethargy. Hypertension and diabetes mellitus were observed in greater frequency in the REFER population but a lower proportion of patients had chronic obstructive pulmonary disease (COPD). The prescribing of diuretics was less frequent in the REFER cohort but a higher proportion was prescribed ACE inhibitors.

Rates of confirmed heart failure

The expert panel reviewed the data for each participant in three steps and determined whether or not a heart failure diagnosis was present at each stage. Forty-eight participants did not have a NT-proBNP level measured and so could not progress to step 3. Of the 304 cases with a NT-proBNP level recorded, 66 (21.7%, 95% CI 17.2% to 26.8%) were diagnosed with heart failure at step 1, 89 (29.3%, 95% CI 24.2% to 34.7%) at step 2 and 104 (34.2%, 95% CI 28.9% to 39.8%) at step 3. The objective abnormalities found on ECG and echocardiogram are shown in Table 2.

The characteristics of participants with and without heart failure are shown in Table 3. Participants with heart failure were older. Half of those with heart failure were male, whereas about one-third of those without heart failure were male. The presenting symptom profile was similar between the groups. Proportionately more patients with heart failure had a history of myocardial infarction (16.4% vs. 8.5%) but there was no significant difference in other comorbidities such as hypertension, COPD and arthritis. Depression was more common in the non-heart failure group. Cardiovascular medications were more likely to be prescribed in those with heart failure than in those without heart failure. The median NT-proBNP level was significantly higher in the heart failure group. At the lower 125 pg/ml cut-off point, over half of patients without heart failure had a NT-proBNP level above the threshold for referral to echocardiography.

Diagnostic accuracy estimates

The diagnostic accuracy of the CDR, NT-proBNP level and CDR and NT-proBNP level in combination is shown in Table 4. The clinical information (MICE symptoms) of the CDR had a sensitivity of 44.2% (95% CI 34.5% to 54.3%) but, with the addition of the NT-proBNP level at the lower cut-off point, the sensitivity improved to 90.4% (95% CI 83.0% to 95.3%) and specificity was 45.5% (95% CI 38.5% to 52.7%). The NT-proBNP level alone with a cut-off point of 400 pg/ml had a sensitivity of 76.9% (95% CI 67.6% to 84.6%) and a specificity of 91.5% (95% CI 86.7% to 95.0%). At the lower cut-off point of 125 pg/ml, sensitivity was 94.2% (95% CI 87.9% to 97.9%) and specificity was 49.0% (95% CI 41.9% to 56.1%). These performance characteristics were mostly lower in magnitude than the corresponding values observed in the derivation data set. However, comparison of the CIs suggests that the differences were not statistically different at the 5% level.

Figure 3 shows the ROC curves for each index test for predicting heart failure. Significant differences (p < 0.0001) were observed between the areas under the ROC curves (AUROCs) shown in Table 4. NT-proBNP had the best discriminatory power, with an AUROC of 0.91 (95% CI 0.88 to 0.95), and the clinical element of the MICE CDR the poorest, with an AUROC of 0.54 (95% CI 0.48 to 0.60).

FIGURE 3.

Receiver operating characteristic curves for CDR and NT-proBNP for predicting heart failure.

There were no significant differences in diagnostic accuracy measures of the CDR between those with and those without missing NT-proBNP results, as shown in Table 5.

Assessing incorporation bias

The performance characteristics for the CDR and NT-proBNP at steps 1–3 are shown in Table 6. The diagnostic accuracy of all tests increased at each step, with the largest changes observed when NT-proBNP was used without the clinical element of the CDR. A NT-proBNP cut-off point of 400 pg/ml showed a statistically significant increase in the detection of cases without heart failure from step 2 to step 3 (p < 0.05).

Summary of findings

At a low cut-off point of 125 pg/ml, NT-proBNP testing alone showed similar diagnostic performance to the CDR (clinical features and natriuretic peptide level) in determining which patients presenting with possible heart failure should be referred for diagnostic testing. The current NT-proBNP cut-off point of 400 pg/ml recommended by NICE7 in the UK is too high. Sensitivity for a diagnosis of heart failure at this level was 77% (95% CI 68% to 85%), meaning that one in five patients with heart failure presenting to primary care may not be appropriately referred for further investigation and diagnosis.

Strengths and limitations

This study included patients presenting prospectively to their GP. A large proportion of health care in the UK is provided through general practice and testing the CDR in a real-life clinical setting where most patients are managed allows accurate validation of the rule. 27

Participants underwent thorough phenotyping including clinical and objective assessment and then review by a panel of three experienced cardiologists, using a staged system to allow for assessment of inclusion bias, to agree a formal diagnosis so that the ‘observed disease’ was accurate. 28

The study was slow to recruit and failed to meet the initial target of 500 patients. The main reason for this slow recruitment, in common with many studies requiring opportunistic recruitment during routine consultations, was the additional administrative burden on busy GPs. This was at a time of unparalleled increased workloads in UK general practice. Furthermore, when the study was designed, natriuretic peptide assays were not routinely available to UK GPs and, therefore, the provision of a rapid natriuretic peptide and echocardiography service might have provided some ‘compensation’ for the additional administration needed for the study. However, shortly after the study commenced, natriuretic peptide assays became an open-access diagnostic for practices, alongside one-stop heart failure diagnostic clinics. 7

The number of participants with heart failure with reduced ejection fraction (HFREF) was unexpectedly low in the cohort. This may reflect the nature of heart failure presentation, with those with HFREF perhaps more likely to present acutely directly to secondary care or perhaps already being under the care of a cardiologist for a known cardiovascular comorbidity such as coronary artery disease. 29

The increase in performance across the stepped diagnosis suggests that the sensitivity of the index tests may have been overestimated because of incorporation bias. However, evaluation of the results at step 2 (at which point NT-proBNP testing is excluded from clinical diagnosis) confirms that the diagnostic accuracy of the NT-proBNP test alone at the lower cut-off is similar to that of the CDR (in combination with NT-proBNP testing).

Comparison with existing literature

Heart failure can be a difficult diagnosis to make and the idea of a CDR to help primary care clinicians make the decision of who to refer for objective testing is justifiable. 30 The role of CDRs as an aid to clinical decision-making however remains controversial. There are many examples of CDRs being generated and validated with the hope of improving clinical accuracy but performance characteristics are often modest at best. 13,31–33 Furthermore, remembering the components of a CDR and applying them during the consultation can be challenging for busy generalist clinicians seeing patients with undifferentiated illness. 34

The reason why the CDR performed no better than NT-proBNP testing alone may be because the diagnosis of heart failure in the cohort was largely heart failure with preserved ejection fraction (HFPEF). The derivation and initial validation of the CDR relied predominantly on epidemiological studies that included heart failure as a result of left ventricular systolic dysfunction, or HFREF, and so may not directly apply to this population. 16 In addition, the way that symptoms were recorded may have differed – the study used to derive the CDR was carried out by cardiologists in a secondary care clinic, whereas the REFER study data were collected by research nurses; however, both studies relied on referral from primary care.

The prevalence of atrial fibrillation and valvular disease was also very high in the REFER cohort. However, this is likely to be an increasing reality in primary care, with the number of patients with HFPEF and/or other cardiovascular comorbidities increasing. 35

The threshold for NT-proBNP below which heart failure can be reasonably excluded is also an area of ongoing research. 36 Guidelines differ in the threshold currently recommended. For example, the ESC⁴ suggests that a NT-proBNP level of < 125 pg/ml should be used to rule out heart failure but, in the UK, NICE7 suggests a much higher NT-proBNP threshold of 400 pg/ml. For any test there must always be a trade-off between sensitivity and specificity. 22 A high sensitivity will ensure that fewer cases are missed, but at the expense of more patients undergoing echocardiography, which is an expense to the health-care system and may be an unnecessary inconvenience for the patient. However, accepting a test with a sensitivity that is too low could result in a diagnosis of heart failure being missed.

Implications for practice and research recommendations

Echocardiography remains the most commonly used modality for providing objective evidence of heart failure,37 but all health-care systems must work within the constraints of finite resources and in many countries there is a limited supply of adequately trained echocardiography technicians. Access to a CDR and/or point-of-care blood test that can reliably rule heart failure in or out quickly would be attractive to clinicians. This study shows that, in patients suspected of having heart failure, a negative NT-proBNP blood test alone, at a threshold of 125 pg/ml, means that heart failure is unlikely and, thus, this could be used as a ‘rule out’ test. Provision of point-of-care testing in a primary care setting could aid clinicians in their decision on whether or not to refer for echocardiography, which could improve care for patients and reduce the burden on echocardiography services.

In the REFER study, a large proportion of the participants with heart failure had HFPEF. Estimates of the proportion of patients with HFREF compared with HFPEF vary but these findings may reflect the contemporary population presenting to general practice with symptoms suggestive of heart failure. This has important implications for how patients in general practice might be managed, given the current lack of effective treatments. Further research to find strategies to improve quality of life, reduce hospitalisations or extend survival in HFPEF patients is needed.

Natriuretic peptide levels are influenced by multiple factors. Arrhythmias such as atrial fibrillation are associated with high circulating levels, whereas medications such as ACE inhibitors reduce natriuretic peptide production. Interpreting the NT-proBNP level in clinical practice needs to be carried out in the context of a patient’s comorbid conditions and treatment regime. Further research to provide guidance on natriuretic peptide correction factors could be helpful to clinicians.

Conclusions

Natriuretic peptide testing alone performed as well as a validated CDR, based on the best available epidemiological evidence, in determining which patients went on to have a diagnosis of heart failure. The current guidelines, which suggest measuring NT-proBNP in patients with symptoms suggestive of heart failure, are justified, but the cut-off point must be low enough to ensure that GPs are not falsely reassured that referral is not required.

The REFER study data set

The REFER study was a prospective, observational, diagnostic validation study of the MICE CDR, with natriuretic peptide testing, for diagnosing heart failure in primary care. The full methods for the REFER study have been published previously. 23 The CDR was developed from an individual patient data meta-analysis of epidemiological studies of heart failure screening in primary care, commissioned as a health technology assessment by NICE. 4

Briefly, primary care patients aged ≥ 55 years presenting to their GP with symptoms suggestive of heart failure were recruited across 28 central England practices in the UK. All consenting patients underwent a full clinical assessment, which included a NT-proBNP test, an echocardiogram and a quality-of-life questionnaire, at a research clinic within 1 week of recruitment. Follow-up quality-of-life and resource-use questionnaires were mailed to the patients at 6 and 12 months after attending the clinic.

The diagnosis of heart failure or no heart failure was determined by an expert panel of cardiologists using the ESC 2012 definition. 6 Clinical information, including the variables of the MICE rule and NT-proBNP level, was presented in stages to quantify any incorporation bias.

Economic evaluation

The aim of this study was to assess the cost-effectiveness of using the MICE CDR in heart failure diagnosis in general practice from a NHS and Personal Social Services perspective. To do so, a decision tree was developed comparing different diagnostic strategies against the CDR. The economic analysis utilised the REFER data set to determine which symptomatic patients would receive the correct diagnostic decision. The cost and quality of life consequences of a correct and incorrect referral were taken from the data set and the literature. The economic evaluation took a lifetime horizon and all costs and outcomes were discounted at 3.5% in accordance with NICE guidance on the methods of technology appraisal. 39

Economic evaluation diagnostic pathways

Decision tree structure

A decision tree, presented in TreeAge Pro 2014 (TreeAge Software, Inc., Williamstown, MA, USA) and developed in Microsoft Excel^®, version 2010 (Microsoft Corporation, Redmond, WA, USA), was structured to represent the various diagnostic strategies (Figure 4). A decision tree was the most appropriate method to use here as a ‘one-off’ decision is involved (i.e. whether or not to send patients on for a confirmatory diagnosis for heart failure). The branch probabilities were estimated from the REFER data set. To ensure correct representation of the statistical uncertainty in the model, patients were categorised into groups on the principle that two patients would be in the same group if and only if they followed the same pathway in all strategies considered. For example, consider four patients of the same sex and with the same clinical signs, with no previous history of myocardial infarction but with NT-proBNP levels of 110 pg/ml (A), 220 pg/ml (B), 330 pg/ml (C) and 440 pg/ml (D). Under the NT-proBNP 125 strategy, patients B, C and D would be referred for echocardiography but patient A would not, whereas under the NICE strategy only patient D would be referred. However, no strategy in the model has a cut-off point between 220 pg/ml and 330 pg/ml and so patients B and C can be placed in the same group. Table 7 shows the breakdown of patients into groups based on this principle.

FIGURE 4.

Decision tree of the different diagnostic strategies for patients presenting with heart failure symptoms in primary care. Echo, echocardiogram; HF, heart failure; MI, myocardial infarction.

Effects of early heart failure diagnosis

The methodology behind the benefits of each strategy was drawn largely from a previous Heart Failure Health Technology Assessment report. 40 It was assumed that a confirmed heart failure diagnosis leads to patients being initiated on heart failure drug treatment if they have HFREF. Current trials do not suggest a survival advantage of ACE inhibitors or beta-blockers for patients with HFPEF. 15 Thus, the benefits from early detection of heart failure were weighted by the number of heart failure patients in the population (11.5%) who have a reduced ejection fraction.

The type of heart failure drug and dose used were obtained from a UK retrospective cohort study of patients treated for heart failure in primary care; 36.6% of patients were treated with beta-blockers, 58.9% with ACE inhibitors and 13.4% with angiotensin receptor blockers (ARBs). 40 Patients can be on more than one heart failure drug therapy; to determine the proportion of patients on ACE inhibitors but not on beta-blockers (not reported in the study) we took a simple average of the upper bound and lower bound range of the possible values. This gave a value of 36.3% for patients on ACE inhibitors but not on a beta-blocker.

The initiation of heart failure treatment after correct early detection was assumed to have a survival benefit for the heart failure patient compared with a missed heart failure diagnosis.

Effects of delayed heart failure diagnosis

A missed heart failure diagnosis was assumed to delay the diagnosis by 6 months. The delayed diagnosis patients were assumed to incur a further GP visit and an echocardiogram scan to confirm the diagnosis. We assumed that the delayed diagnosis patients had the same prognosis as untreated patients for the first 6 months. After 6 months, the delayed diagnosis patients were put on treatment and thereafter had the same survival probability as someone who was already on treatment.

Treatment effects on mortality

As in a previous model,41 we took the survival data from patients with heart failure in the Framingham Heart Study to estimate prognosis in the absence of drug therapy. An individual patient data meta-analysis indicated that beta-blockers compared with placebo gave a hazard ratio of 0.73 (95% CI 0.67 to 0.80) for all-cause mortality; 95% of these beta-blocker patients were also on ACE inhibitors/ARBs. 42 The results of a systematic review showed that the mortality odds ratio for patients taking ACE inhibitors compared with a placebo was 0.80 (95% CI 0.74 to 0.87). 43 A Cochrane review found no significant effect of ARBs on mortality and we made a conservative assumption based on this analysis that patients on ARBs have the same survival rate as untreated patients. 44

The survival data and the drug efficacies were used to plot sex-specific survival curves for the untreated patients and patients on different drug therapies (assuming no temporal changes in drug efficacy). The survival curves were extended beyond the 10-year survival data by linear extrapolation to achieve a lifetime horizon. Tables 8 and 9 depict the estimated probabilities associated with treatment and no treatment of men and women, respectively. The patients who were put on treatment earlier after correct detection of heart failure (treat early) and the patients with a delayed diagnosis (treat late) were weighted by the proportion of patients on the different drug therapies.

Figures 5 and 6 show the discounted survival curves of patients treated early and treated late for men and women, respectively. The survival benefit from early treatment is the area between the treated early and treated late curves. These curves were discounted at 3.5% to obtain discounted life-years gained from early treatment. The sex-specific survival benefit was then combined by the weighted average number of men and women in the data set with heart failure.

FIGURE 5.

Discounted survival curves for male patients on and off treatment.

FIGURE 6.

Discounted survival curves for female patients on and off treatment.

Patients with heart failure in the REFER data set who answered the 6-month follow-up questionnaire gave a mean EQ-5D score of 0.615 (SD 0.31) (Table 10). The discounted quality-adjusted life-year (QALY) gain can be estimated from the product of the EQ-5D and the discounted life-years gained. Current trials do not suggest a survival advantage of ACE inhibitors or beta-blockers for patients with HFPEF. 45 Thus, the survival gain from early detection of heart failure is weighted by the number of heart failure patients in the population (11.5%) who have a reduced ejection fraction (HFREF).

Early heart failure treatment effect on hospital admissions

The early detection of heart failure also has the benefit of a reduction in hospitalisations. Patients on beta-blockers compared with placebo have a hazard ratio of 0.71 (95% CI 0.65 to 0.77) for first heart failure-related hospital admission. Similarly, treatment with ACE inhibitors reduces the likelihood of a hospital admission, with an odds ratio of 0.67 (95% CI 0.61 to 0.74). 42

For the purpose of the model we were interested in the increased rate of admission to hospital for an untreated population because of a 6-month delay in heart failure diagnosis. In a treated population following a new diagnosis of heart failure, 59% of people in a south London heart failure incidence study had a subsequent hospital admission over a span of 19 months. 46 This would result in a 19-month ‘hospital-free survival’ probability of 41% (odds 0.70). We assumed that the ‘hospital-free survival’ curve followed the same pattern (constant odds ratio) as the survival curve for untreated males. From this, the estimated survival at 19 months was 50.6% (odds 1.02 in favour of survival), with survival at 6 months of 67.7% (odds 2.09). This gave estimated odds of no admission at 6 months if treated of 1.42, which gave a probability of no admission of 58.7%, hence a probability of admission if treated of 41.3%.

Using the odds ratio of hospital admission with ACE inhibitor treatment, the estimated rate of admission within 6 months if untreated is 0.41/0.67 = 0.62. Using the hazard ratio for treatment with beta-blockers (0.71), assuming a constant hazard over the 6 months, the estimated rate of admission within 6 months if untreated is 71.2%. A simple average was taken to calculate the estimated rate of admission with 6 months if untreated (66.4%). Thus, the increased number of hospitalisation cases if untreated is 25.1% (41.3−66.4%). This was then weighted by the number of HFREF patients in the population as only these patients receive a prognostic benefit from treatment.

Costs associated with the diagnostic strategies

A list of costs and their sources are shown in Table 11. The cost of an avoidable heart failure hospital admission was calculated from weighted average 2013/14 reference costs of heart failure admissions. 48 The cost of an echocardiography referral, consisting of a simple echocardiogram and a consultant-led outpatient first attendance, was taken from the Payment by Results mandatory tariff 2013/14. 47

The cost of a NT-proBNP test was taken from a NICE costing report,7 which based it on a provider contract price. As there was no mention of a cost year, for inflation purposes the NT-proBNP test cost was assumed to have been obtained when the report was published (2010). This cost was then inflated to 2013/14 prices using the Hospital & Community Health Service (HCHS) pay and prices index. 50 The early treatment heart failure drug cost was estimated from the drug resource usage combined with drug prices from the British National Formulary (BNF). 49 The heart failure drug therapy prescription mix was taken from a 5-year retrospective cohort study of general practices involved in the Doctors Independent Network (DIN-LINK) database. 41 As the typical drug dose for a heart failure patient is variable, we pragmatically assumed that the same percentage of patients reached the target dose, as defined by the 2005 ESC guidelines51 and the remainder achieved half the target dose. When the drug listed was not recommended by the 2012 ESC guidelines,6 the target dose indication for heart failure in the BNF was used. The drug therapy costs were weighted by the percentage of patients on the different regimes (ACE inhibitors, beta-blockers and ARBs). For the base-case analysis, the cost of the generic drugs for the first 6 months was used (£10); the cost of the equivalent drug therapy with branded drugs was substituted in the sensitivity analysis (£30).

The results are presented as the total costs and effects of each strategy. Effectiveness was measured in QALYs. The total costs and outcomes of each strategy are ordered by increasing effectiveness. Incremental cost-effectiveness ratios (ICERs) were calculated from the difference in costs and effects between two options. Options that are more costly and less effective (dominated) are excluded from consideration. Likewise, options that suffer from extended dominance are removed from consideration. Extended dominance occurs when an option would be dominated compared with a mixed option of two other strategies.

The incremental analysis was designed to generate the cost per additional QALY gained of using one diagnostic strategy over another. Cost-effectiveness was assessed in relation to the lower NICE cost-effectiveness threshold of £20,000 per QALY gained.

To assess model robustness, deterministic and probabilistic sensitivity analysis was performed.

Deterministic sensitivity analysis

For the deterministic sensitivity analysis the following scenarios were explored to test the robustness of the base-case results:

• doubling and halving the cost of a BNP test

• altering the drug efficacies to their lower and upper CIs

• substituting in branded drug prices for drug therapy prices

• increasing the proportion of HFREF patients from 12% to 24%, 50% and 100%.

Probabilistic sensitivity analysis

For the probabilistic sensitivity analysis, distributions were attached to the clinical parameters, drug efficacies and heart failure utility (Table 12). The model was run for 10,000 iterations and the results presented as a cost-effectiveness acceptability frontier (CEAF). The CEAF shows the uncertainty associated with the optimal diagnostic option across a range of different cost-effectiveness thresholds.

Positive count data in groups formed the parameters for the Dirichlet distribution (see Table 12). On each replication a vector of probabilities was sampled from the appropriate distribution. When there were no count data (no patients) in a patient group, we assumed that the appropriate combination was impossible and no distribution was attached to clinical parameters with zero occurrences in the data set. Treating the probability as fixed for these empty patient groups will slightly underestimate the uncertainty in the model rather than the alternative of adding an occurrence and positively biasing the amount of occurrences for these groups. 52

Deterministic sensitivity analysis results

The results of the sensitivity analysis are shown in Table 14. The MICE cut-off point options are excluded from the table as they remained dominated in each scenario. As in the base-case results, the NICE strategy remained the most cost-effective option for each sensitivity analysis scenario except when the proportion of HFREF changed to ≥ 50%. When the proportion of HFREF patients is 50%, the NT-proBNP 125 strategy becomes cost-effective, and when this proportion reaches 100% it becomes cost-effective to refer all patients for immediate echocardiography. Threshold analysis indicated that the NT-proBNP 125 strategy becomes cost-effective compared with the NICE strategy when the proportion of HFREF rises above 36%.

Probabilistic sensitivity analysis results

Figure 8 illustrates the overall uncertainty related to the optimal decision across a range of plausible willingness-to-pay values, where willingness to pay is measured in cost per additional QALY. Taking NICE’s willingness to pay for an additional QALY as £20,000, the likelihood of the NICE strategy being the optimal option (i.e. highest net monetary benefits) is 99.9%. As the willingness-to-pay threshold increases beyond £68,000, the NT-proBNP 125 option becomes more likely to be the optimal option.

FIGURE 8.

Cost-effectiveness acceptability frontier showing the optimal diagnostic strategy across a range of willingness-to-pay thresholds.

Summary of findings

The results indicate that, for the REFER population, the current NICE guidelines on diagnosing heart failure provide the most cost-effective strategy compared with the MICE CDRs and other diagnostic strategies. The MICE CDRs with upper and lower cut-off points were excluded from the base-case and the sensitivity analyses because of dominance by the other strategies. The cost-effectiveness results were robust to deterministic and probabilistic sensitivity analyses.

The modest gains in effectiveness for each strategy, even with a lifetime horizon, can be largely explained by the patient characteristics in the REFER data set. The benefits of early detection of heart failure and the subsequent prognostic window of improvement (treated early vs. treated late) used in the model are limited as only 12% of the heart failure participants in the REFER study were found to have a reduced left ventricular ejection fraction. In this analysis, survival benefit and reduced hospitalisation risk from early detection are restricted to this subgroup, that is, we model no benefits from earlier detection in the 88% of the REFER population detected with non-HFREF. This is an appropriate conservative assessment but it is unlikely that these patients will derive zero benefit from detection.

In a sensitivity analysis scenario, increasing the proportion of HFREF patients raised the total QALYs produced by each diagnostic strategy because of the higher rewards of correct early detection. Most population surveys suggest that approximately half of patients with heart failure have HFREF,53 in contrast to the 12% detected in this study, and in this scenario the dominant cost-effective strategy is the reduced NT-proBNP cut-off point of 125 pg/ml for referral for echocardiography rather than the NICE level of 400 pg/ml. The echo all strategy becomes the most cost-effective option when we assume that all of the heart failure patients will receive a prognostic benefit from treatment. However, it must be acknowledged that there may be practical barriers to the implementation of such a strategy. Delays to echocardiography may potentially offset the advantage of early detection and referring all patients for an echocardiogram will put pressure on local diagnostic services. The cost used for a NT-proBNP test was an inflated cost from 2010.

Strengths and limitations

Since NT-proBNP testing came into more general use in the health-care system, the cost per test may have dropped and so the cost-effectiveness of natriuretic peptide testing may be greater than the estimates presented. This was explored in the sensitivity analysis by halving the cost of a NT-proBNP test, which resulted in the NT-proBNP 125 strategy becoming more cost-effective than the echo all and do nothing strategies.

The heart failure therapy drug mix for patients used in the economic evaluation relates to a 2009 publication. 41 As it is very likely that there are now higher rates of drug use, the proportion of patients on ACE inhibitors and beta-blockers may be more than assumed and so the benefits of diagnosis may be greater.

The main strength of this analysis is that the diagnostic accuracy of the various strategies tested was calculated based on a consistent primary data set. This was reinforced by the process used in the clinical study to develop an appropriate gold standard against which to compare imperfect diagnostic strategies. The main limitations are that the REFER data set may not be typical of patients in other geographical areas and that assumptions have had to be made in projecting the lifetime costs and outcomes of a correct diagnosis.

Comparison with existing literature

This is the first cost–utility analysis of heart failure diagnosis in this patient population, making comparison with other studies difficult. A cost–consequence analysis, in which only costs are considered, suggested that a diagnostic strategy using a BNP referral level of 100 pg/ml had a higher test accuracy and was only marginally more expensive than a consultant-led ECG strategy. 54 Other studies55,56 have looked at the costs of using NT-proBNP testing as a means to rule out heart failure and to reduce the levels of echocardiography referrals but these do not come close to answering the question of the optimal cost-effective strategy and also do not place a cost on a missed heart failure diagnosis.

Implications for practice and research recommendations

The findings of this study suggest that further research around the cost-effectiveness of different natriuretic peptide thresholds in specific heart failure populations (HFREF vs. HFPEF) is needed to determine the optimal strategy for diagnosis in primary care. The organisation of health-care systems globally is associated with differing costs of and access to investigations. Cost-effectiveness analyses must be carried out in the context of individual health-care economies to determine the most cost-effective local strategy.

Conclusions

Overall, this analysis provides evidence based on primary data that the current strategy recommended by NICE for the diagnosis of heart failure is appropriate, that is, patients presenting with symptoms suggestive of heart failure should be referred straight for echocardiography if they have a history of myocardial infarction or if their NT-proBNP level is ≥ 400 pg/ml. However, based on sensitivity analysis, as the proportion of patients with HFREF increases from the 12% seen in the REFER study, it becomes more cost-effective to change the NT-proBNP threshold from 400 pg/ml to 125 pg/ml.

The REFER study investigators

Investigators also included Professor Martin R Cowie (Professor of Cardiology, Faculty of Medicine, National Heart and Lung Institute, Imperial College London, UK), Dr Russell Davis (Consultant Cardiologist, Department of Cardiology, Sandwell and West Birmingham Hospitals, UK), Professor Jon Deeks (Professor of Biostatistics, Institute of Applied Health Research, University of Birmingham, UK), Professor Jonathan Mant (Professor of Primary Care, Department of Public Health and Primary Care, University of Cambridge, UK), Dr Deborah McCahon (Lecturer in Primary Care, Institute of Applied Health Research, University of Birmingham, UK), Professor Theresa McDonagh (Professor of Heart Failure, Department of Cardiology, King’s College Hospital, UK), Dr George Sutton (Consultant Cardiologist, Faculty of Medicine, National Heart and Lung Institute, Imperial College London, UK) and Dr Lynda Tait (Senior Research Fellow, School of Health Sciences, University of Nottingham, UK).

Contributions of authors

Clare J Taylor (NIHR Academic Clinical Lecturer and GP) was a grant applicant, provided clinical input throughout the study and first drafted the main results paper for the REFER study.

Mark Monahan (Health Economist) was the health economist on the study and drafted the health economic evaluation paper for the REFER study.

Andrea K Roalfe (Statistician) conceived of and designed the study, was a grant applicant and study statistician and contributed to writing the draft papers.

Pelham Barton (Reader in Mathematical Modelling) was a grant applicant and senior health economist on the study and contributed to writing the health economic evaluation paper for the REFER study.

Rachel Iles (Research Fellow) was the study co-ordinator throughout the study and commented on the draft papers.

FD Richard Hobbs (Professor of Primary Care and GP) had the original idea, conceived of and designed the study, was a grant applicant and principal investigator, provided clinical input throughout the study, compiled the report, co-drafted the papers and is corresponding author and guarantor.

Contributions of others

Martin Cowie (Professor of Cardiology) conceived of and designed the study, was a grant applicant, was on the expert panel and commented on the draft papers.

Russell Davis (Cardiologist) was on the expert panel and commented on the draft papers.

Jon Deeks (Professor of Biostatistics) was a grant applicant, provided statistical advice and commented on the draft papers.

Jonathan Mant (Professor of Primary Care) conceived of and designed the study, was a grant applicant and commented on the draft papers.

Deborah McCahon (Lecturer in Primary Care) was study co-ordinator for part of the study and commented on the draft papers.

Theresa McDonagh (Professor of Cardiology) was on the expert panel.

George Sutton (Cardiologist) was on the expert panel.

Lynda Tait (Research Fellow) conceived of and designed the study, was a grant applicant, was senior study co-ordinator for most of the study and commented on the draft papers.

Publications

One paper has been published and two further papers have been submitted to peer-reviewed journals. Standards for Reporting of Diagnostic Accuracy (STARD) and Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklists were completed for the REFER study and health economics analysis papers, respectively, before submission.

Tait L, Roalfe AK, Mant J, Cowie M, Deeks JJ, Iles R, et al. The REFER (REFer for EchocaRdiogram) protocol: a prospective validation of a clinical decision rule, NT-proBNP, or their combination, for the diagnosis of heart failure in primary care. Rationale and design. BMC Cardiovasc Disord 2012;12:97.

Taylor CJ, Roalfe AK, Iles R, Hobbs FR, REFER investigators, Barton P, et al. Primary care REFerral for EchocaRdiogram (REFER) in heart failure: a diagnostic accuracy study. Br J Gen Pract 2017;67:e94–102.

Monahan M, Barton P, Taylor CJ, Roalfe AK, Hobbs FDR. MICE or NICE? An economic evaluation of clinical decision rules in the diagnosis of heart failure in primary care [published online ahead of print 2 March 2017]. Int J Cardiol 2017.

Data sharing statement

The original data used in the analysis are available from the corresponding author, Professor FD Richard Hobbs (Nuffield Department of Primary Care Health Sciences, Radcliffe Primary Care Building, Radcliffe Observatory Quarter, Woodstock Road, Oxford, OX2 6GG; richard.hobbs@phc.ox.ac.uk).

Disclaimers

This report presents independent research. The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, the MRC, NETSCC, the EME programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the EME programme or the Department of Health.