A systematic review and economic evaluation of intraoperative tests (RD-100i OSNA system and Metasin test) for detecting sentinel lymph node metastases in breast cancer

Proposed clinical pathway

Either OSNA or Metasin could be used as a replacement for current normal practice, in which case the SLNs are used in their entirety. However, OSNA or Metasin could also be used adjunctively, in which half of each LN (one piece or alternate slices) is used for OSNA or Metasin intraoperative testing and the remaining half is examined using standard postoperative histopathology. Where the new tests have been introduced in practice the first model is the most commonly followed as it maximises the claimed benefits of the new technology.

The proposed clinical pathways are illustrated in Figures 5 and 6. This emphasises that changes to the clinical pathway occur only in patients whose ultrasound is negative or in patients whose ultrasound is positive but whose FNAC is negative.

FIGURE 5.

Proposed clinical pathway for staging of breast cancer and subsequent surgery to the axilla: OSNA or Metasin used as a replacement for SLNB. a, In cases in which the OSNA/Metasin test does not provide a result or provides a result that is uninterpretable, the OSNA/Metasin test would first be repeated using the remaining solubilised LN. Failing this, four-node sampling could be carried out, followed by delayed ALND in the case of positive LNs being identified in the sample or immediate ALND with the patient’s prior informed consent for contingency.

FIGURE 6.

Proposed clinical pathway for staging of breast cancer and subsequent surgery to the axilla: OSNA or Metasin used adjunctively. a, In cases in which OSNA/Metasin does not provide a result or provides a result that is uninterpretable, the OSNA/Metasin test would first be repeated using the remaining solubilised LN. Failing this, histopathology is performed on the remaining 50% of the SLN. b, No histopathology is performed on the SLN if the OSNA/Metasin test is positive. The ALND sample is still examined histopathologically as normal. –, negative; +, positive.

Clinical considerations

The intermediate measures for consideration include:

• diagnostic test accuracy (DTA)

• test failure rate

• discordant test results

• time to test result

• duration of anaesthesia.

The clinical outcomes for consideration include:

• patient anxiety associated with waiting time for results and not knowing the extent of surgery before the operation

• the number of repeat operations (except for re-excision of positive margins)

• time in operating theatre

• time to start of and nature of adjuvant therapy

• morbidity and mortality from biopsies, axillary dissections, first and second operations and treatment of cancer

• adverse events from false test results including patient distress and sequelae.

Data on these outcomes are likely to be used along with clinical utility scores to estimate quality-adjusted life-years (QALYs).

Cost considerations

The cost analysis will be based on the UK NHS setting and will comprise both NHS and Personal Social Services costs.

The costs for consideration include:

• the costs of equipment, any additional tests (pre-screening), reagents and consumables

• staff and staff training costs

• equipment maintenance costs

• costs associated with surgeon time and the management of operating theatre time

• medical costs arising from ongoing care following test results, including those associated with surgery, time spent in hospital and treatment of cancer

• medical costs arising from adverse events, including those associated with biopsies, surgery, cancer treatment and false test results.

The cost of the hardware for the RD-100i OSNA system is approximately £70,000 [excluding value-added tax (VAT)] The cost of consumables is approximately £150–250 per patient (excluding VAT). This consumable cost is dependent on the number of tests performed per theatre day and the number of patient samples tested. The maintenance cost is £6180 per annum (excluding VAT) following the expiry of the 1-year warranty.

Population

Studies of individuals with invasive breast cancer who underwent a (sentinel) LN biopsy during the primary operation to excise a suspected breast cancer were included.

Interventions and comparators

Studies of OSNA or Metasin as used at the thresholds recommended by the manufacturer or designer were included. The reference standard was postoperative histopathology, performed on fresh sections of tissue. Frozen section and touch imprint cytology were excluded as comparators as they were not felt to be sufficiently feasible for widespread implementation (or intervention).

Outcomes

No study was excluded on the basis of outcomes, provided that it appeared relevant to those listed in the decision problem:

• test failure rate

• DTA

• discordant test results

• time to test result

• duration of anaesthesia/time in operating theatre

• number of repeat operations (except for re-excision of positive margins)

• time to start of, and nature of, adjuvant therapy.

The clinical outcomes for consideration include:

• patient anxiety associated with waiting for results and not knowing the extent of surgery before the operation

• adverse events from false test results including patient distress and sequelae

• morbidity and mortality from biopsies, axillary dissections, first and second operations and treatment of cancer.

Study design

For the review of test accuracy, the protocol made provision for all study designs unless evidence on the intervention and outcome of interest was already available from designs less open to bias as judged with reference to standard hierarchies of evidence. 42

Systematic reviews were used as a source for finding further studies and to compare with our systematic review. For the purpose of this review, a systematic review was defined as including:

• a focused research question

• explicit search criteria that are available to review, either in the document or on application

• explicit inclusion/exclusion criteria defining the population(s), intervention(s), comparator(s) and outcome(s) of interest

• a critical appraisal of included studies, including consideration of internal and external validity of the research

• a synthesis of the included evidence, whether narrative or quantitative.

Studies were excluded if they did not match the inclusion criteria and, in particular, were:

• preclinical and animal studies

• reviews, editorials and opinion pieces

• case reports

• studies with < 10 participants.

Internal validity

The instruments sought to assess the following considerations.

External validity

External validity was judged according to the ability of a reader to consider the applicability of findings to a patient group and service setting. Study findings can be generalisable only if they provide enough information to conclude that a cohort is representative of the affected population at large. Therefore, studies whose samples appeared to be typical of the UK breast cancer population were judged to be externally valid.

Meta-analysis

Meta-analysis of DTA was performed using the bivariate method44 implemented in Stata/SE 12.1 (StataCorp LP, College Station, TX, USA) using the command metandi. 45 Studies were included in the meta-analysis only if the numbers of true positives, true negatives, false negatives and false positives were all reported in the text or could be unambiguously inferred from other figures in the text. Meta-analysis using the full bivariate method was not performed when there were fewer than four included studies as the model cannot generally be estimated with fewer than four studies. When the full bivariate model could not be estimated (either because of too few studies or because of other convergence errors) we reduced the complexity of the model (as, for example, in Pennant et al. 46) by setting the correlation parameter to zero (effectively reducing the model to two independent univariate random-effects analyses) and performing the analysis directly using the Stata command xtmelogit.

The bivariate method, when calculated using maximum likelihood estimation and without covariates, is equivalent to the hierarchical summary receiver operating characteristic (HSROC) model47–49 and this can be used to provide a summary receiver operating characteristic (SROC) curve and prediction region as well as a summary estimate and confidence region of sensitivity and specificity.

The SROC curve is designed to show how sensitivity and specificity are traded off against each other in different studies, through variation of the positivity threshold. If, and only if, there is reason to believe that the positivity threshold might vary between studies, we provide a SROC curve and prediction region.

Test accuracy

In most of the studies, the accuracy of the interventions has been evaluated against the reference (gold) standard of postoperative histopathology. The results are generally reported as follows:

• Sensitivity = true positives/(true positives + false negatives). This is the probability of detecting the presence of metastases in someone with metastases.

• Specificity = true negatives/(false positives + true negatives). This is the probability of not detecting metastases in someone without metastases. In this instance a high specificity is required to avoid unnecessary ALND.

• PPV = true positives/(true positives + false positives). This is the probability of someone with a positive result actually having metastases.

• NPV = true negatives/(true negatives + false negatives). This is the probability of someone with a negative test result actually not having metastases.

• Accuracy or concordance with reference standard = (true positives + true negatives)/(true positives + true negatives + false positives + false negatives). This is the percentage of test results correctly identified by the test, that is, the rate of agreement with the reference standard.

• Discordance = cases of disagreement between the reference standard and the index test

Discordant case analysis

Many studies used a single-gate design whereby, in a single sample of individuals with unknown metastatic status, portions of the same node were allocated to either a molecular assay or histopathology. However, because of the spatial distribution of metastasis within a LN, and the use of different parts of the LN for different diagnostic tests, TAB may occur (i.e. discordant results between tests may result from genuine differences between the tissue samples). Unfortunately, it is not possible to use the same tissue for both tests; the tissue used for the molecular assays cannot be used for histology, and the formalin-fixed and paraffin-embedded tissue required for permanent sections is not suitable for quantitative mRNA measurements.

Some studies have attempted to address this issue by performing extensive histopathological and molecular analysis on discordant cases to ascertain whether the test results are true and occur because of differences in the allocated tissue. The results are then adjusted (discordant cases resulting from TAB are removed).

Number of studies identified

The electronic searches retrieved a total of 665 titles and abstracts. Fifty-nine additional papers were found by searching the bibliographies of included studies and by forward chasing. A total of 589 papers were excluded, based on screening titles and abstracts. The full texts of the remaining 135 papers were requested for more in-depth screening, of which 16 published papers50–65 (two reporting the same study) and two unpublished papers40,66 were included in the review. The process of study selection is shown in Figure 7.

FIGURE 7.

Summary of study selection.

Number of studies excluded

Papers were excluded for at least one of the following reasons: duplicate publication, narrative review and publication (systematic reviews and individual studies) not considering relevant intervention, population, comparison or outcomes. The bibliographic details of studies retrieved as full papers and subsequently excluded, along with the reasons for their exclusion, are detailed in Appendix 3.

Number and description of included studies

Sixteen test accuracy papers were included for OSNA,50–65 with two papers reporting the same study. Two unpublished papers40,66 for Metasin were identified and assessed using the Standards for the Reporting of Diagnostic Accuracy Studies (STARD) in Table 7.

The search also identified 49 abstracts,67–115 some of which repeated the data in the full papers, some of which provided supplementary information and some of which were not associated with a full paper. The data in the abstracts are presented in Appendix 4. However, because of the lack of accompanying details, no quality assessment has been performed.

Summary information for the 18 test accuracy studies included in the review is provided in Table 8.

Study characteristics

One-step nucleic acid amplification and Metasin have standard procedures and thresholds; therefore, unless otherwise stated, the included studies have complied with the manufacturers’ instructions. Both methods are semiquantitative and differentiate between micro- and macrometastases, (academic-in-confidence information has been removed).

In contrast to the molecular methods, there is some heterogeneity with regard to the reference standard, particularly in the number of levels examined, for example one-level analysis will involve analysis of one section of node, whereas five-level analysis will examine five sections. As such, for one-level histopathology it is likely that macrometastases will be identified but micrometastases may be missed. A micrometastasis is considered to be > 0.2 mm and < 2 mm and a macrometastasis is considered to be > 2 mm. As there is no way to analyse the whole node, this method cannot be 100% sensitive.

Some studies report cases of ITCs with histopathological analysis. These will fall below the threshold for OSNA and Metasin and are generally considered LN negative as their clinical significance is unknown. 60

The majority of included studies comply with a single-gate design, in which a single sample of individuals with unknown metastatic status is assessed by both the diagnostic test under scrutiny and the reference standard. No studies utilising a two-gate design, in which the test under scrutiny is performed on a sample that includes individuals with known metastatic status (using the reference standard), were identified. However, three cohort studies have been included in which different patient populations were utilised for each test. The inclusion of cohort studies enabled the identification of data based on whole-node analysis, something that would not be possible for OSNA or Metasin using classic DTA study designs, such as the single-gate or the two-gate design.

A general issue for all of the studies is that a portion of node tissue is allocated to the index test and a portion is allocated to the reference standard. As such, the tests are analysing different samples of tissue, which cannot be reused between tests. As metastases may be distributed unevenly, TAB may occur, that is, metastases may exist in the tissue sample provided for one test but not in the tissue sample provided for the other test.

It should also be noted that studies examining ALNs as well as, or instead of, SLNs were included.

Population characteristics

In general, patient characteristics were poorly reported, as were inclusion and exclusion criteria. Not all studies provided the age range of participants and, often, only information on tumour staging was provided. Patient characteristics are presented in Table 9.

Assessment of study quality

A summary of the quality assessment of studies included in the review is provided in Table 10; this is followed by a narrative summary. The main issue within the included studies is TAB, which occurs when tissue cannot be reused between the index test and the reference test. In short, because of an uneven distribution of metastases, the test results may be discordant but accurate for the tissue received. Many studies attempted to address this, either by more extensive histology or by additional molecular analyses, such as qRT-PCR and Western blotting for CK19 protein. The results should then be recalculated after adjustment for TAB, generally with an increase in OSNA sensitivity and specificity.

Other concerns included lack of detail on patient recruitment, minimal information on patient characteristics and unclear sampling methods, that is, no evidence was given on sample replicates and reproducibility for molecular analysis. Furthermore, unless otherwise mentioned, test failures, such as operator or instrument error, were not reported. It was also often not mentioned whether or not the outcome assessors were blinded. For example, it is not inconceivable that the pathologist performing the intraoperative molecular analysis also performed the more subjective postoperative histopathology. Papers were also often unclear on the robustness of the histopathology, for example if results were checked by a second party.

A potential conflict of interest also features heavily, (academic-in-confidence information has been removed) and the majority of the OSNA studies were financially supported by Sysmex.

Metasin

(Academic-in-confidence information has been removed.)

The second paper on Metasin, by Sundaresan,40 which like the first is unpublished and therefore has not undergone formal peer review, reports a sensitivity of 92%, a specificity of 97% and an accuracy of 96%. (Academic-in-confidence information has been removed.) Sundaresan40 used three-level histology as the reference standard. In total, 20 cases (1.6%) were reported as failed assays, which the authors consider may have been because of insufficient RNA in the material submitted for molecular analysis. The authors report 56 patients to be discordant and, although they do not present evidence, they consider TAB to be largely responsible (Table 12).

One-step nucleic acid amplification

Discordance

Studies show a range of discordance, from 1.8% to 8.6% (Table 25); further details are provided in Table 26. The greatest discordance was generally for OSNA false positives. This is a concern as it would potentially lead to unnecessary complete axillary dissection in patients. This may be due to epithelial contamination although, for OSNA, the CK19 copy number cut-off corresponds to the presence of 5000 tumour cells. Therefore, it is unlikely that positive results would occur because of epithelial displacement or illegitimate transcription, which involves only 500–1000 non-tumour cells. 65 By contrast, false-negative tests may result in a lack of treatment in the early stages of cancer, leading to a worse prognosis. The patient may also be required to undergo a second operation for ALND.

Although the true number of discordant cases and the number that result from TAB cannot be fully differentiated, the studies that have adjusted their results display increased specificity and sensitivity for OSNA. Interestingly, the studies that analyse a full node with extensive histology showed no significant difference in positive rates between OSNA and histology,52,55 whereas the study using one-level histology demonstrated a significantly higher positive rate for OSNA. 59

Summary of test accuracy

The results are summarised and stratified according to SLNs, ALNs, patients, before TAB and after TAB in Tables 27–32. An overall summary is provided in Table 33.

With regard to Metasin, the results must be used with caution as they have been taken from unpublished, non-peer-reviewed papers. These papers are also very much in draft form and therefore lacking in detail. That said, with regard to quality, many of the issues (such as lack of information about replicate measurements and therefore no estimate of the reproducibility and the robustness of the test) apply to both papers reporting OSNA and (academic-in-confidence information has been removed).

One-step nucleic acid amplification detects only CK19 mRNA expression. It has been reported that 98.2% of breast cancer tumours express CK19 protein, leaving 1.8% of patients for whom this technique may be invalid. 60 However, Tamaki et al. 62 suggest that it is not necessarily the case that low expression of CK19 protein is reflected in low expression of CK19 mRNA. In contrast, Metasin, which also identifies mammaglobin, (academic-in-confidence information has been removed) compared with that of OSNA, which ranges from 77.8% to 80%. 40

As the reference standard, immunohistochemistry is capable of detecting ITCs, unlike OSNA (and presumably Metasin). However, as the American Society of Clinical Oncology (ASCO)117 and the NICE clinical guideline for early breast cancer13 indicate that ITCs have an unknown clinical significance, and there are insufficient data to recommend appropriate treatment, ITCs were considered histologically negative throughout this review.

Although the aim of OSNA and Metasin is to perform intraoperative molecular analysis, some studies employed a more experimental approach and used frozen samples of RNA, either initially or in discordant case analysis. Both storage and the freeze–thaw effect have been known to adversely affect RNA, effectively leading to a reduction in concentration.

Histopathology will always be limited by sampling bias as only a certain number of slices are taken. This can be increased or reduced according to requirements but will have cost implications and there will always be unanalysed tissue. Furthermore, there is the possibility of observer subjectivity. In contrast, for OSNA or Metasin the whole node can be used and the semiquantitative test is objective.

There is also the issue of TAB. Unfortunately, because of the nature of the OSNA/Metasin tests and of histopathology, it is not possible to use the same tissue sample for both tests and thus TAB is likely to occur. There is no DTA study design that can resolve this issue and, although cohort studies can provide us with some data on whole-node analysis, the tissue being analysed by each test is still, of course, different. Studies have attempted to mitigate the issue of TAB by investigating discordant cases. However, there is a lack of consistency between studies with regard to the methods used to investigate discordant cases, and there will always be uncertainty attached to decisions about whether TAB has occurred.

With regard to the overall results, the range for specificity is lower for results presented by patient rather than by node, before and after adjustment for TAB, as shown in Tables 27–33. The results produced by meta-analysis agree with this (Table 34). Sensitivity before adjustment for TAB is similar for both patients and nodes, whereas, after TAB adjustment, sensitivity is slightly greater for nodes. This may be because some studies use a larger number of nodes from a small number of patients.

With the exception of the study by Feldman et al. ,54 the ranges for sensitivity and specificity for ALNs and SLNs appear to be similar. The results of the meta-analysis for OSNA are shown in Table 34.

For the meta-analysis of test accuracy based on analysis of SLNs without adjustment for TAB, we were not aware of a compelling reason to believe that the positivity threshold might vary between studies and so for consistency with meta-analyses for the other subgroups (see below) we do not include a HSROC curve or prediction region. Figure 8 shows the results of this meta-analysis.

FIGURE 8.

(a) Meta-analysis and (b) forest plot of test accuracy of OSNA for patients (based on analysis of SLNs) without adjustment for TAB. a, results of the ‘routine use’ patient sample reported by Khaddage et al. ;57 b, results of the ‘validation’ patient sample reported by Khaddage et al. 57 (see Table 18). FN, false negative; FP, false positive; TN, true negative; TP, true positive.

As there were only three studies available to estimate the test accuracy of OSNA for patients after adjustment for TAB, we set the correlation parameter to zero as described in Assessment of test accuracy. The results are displayed in Figure 9.

FIGURE 9.

(a) Meta-analysis and (b) forest plot of the test accuracy of OSNA for patients (based on analysis of SLNs) with adjustment for TAB. a, Results of the ‘validation’ patient sample reported by Khaddage et al. 57 (see Table 18); b, adjusted results of the patient sample reported by Le Frère-Belda et al. 58 (see Table 17, bottom), which includes 18 discordant cases that could not be evaluated for TAB. FN, false negative; FP, false positive; TN, true negative; TP, true positive.

For the meta-analyses of the test accuracy of OSNA for SLNs and ALNs we do not present SROC curves or prediction regions because the studies report the same copies/µl thresholds of 250 and 5000 for the OSNA results of + (micrometastases) and ++ (macrometastases) respectively.

We attempted to perform meta-analyses of the test accuracy of OSNA for SLNs with and without adjustment for TAB using the full bivariate method. In both of these meta-analyses we encountered atypical results in the output such that the between-study correlation parameter (σAB/σA2σB2 in Reitsma et al. 44) is estimated as –1 with no estimate of standard error (SE) or CI. This phenomenon has been recognised before and Riley et al. 118 conclude that it is likely to occur when there are few studies or the within-study variation is large (i.e. the studies themselves are small) and is due to sensible bounds being placed on the model to avoid the maximum likelihood estimation including a correlation outside the range [–1,1]. Riley et al. 118 also conclude that there is no systematic bias in the summary estimates of sensitivity and specificity introduced by this phenomenon and that CIs of the summary points are conservative. However, as described in Assessment of test accuracy, we set the correlation parameter to zero and repeated the analysis, the results of which are shown in Figures 10 and 11. Computed summary points and CIs differed only at the third decimal point.

FIGURE 10.

(a) Meta-analysis and (b) forest plot of the test accuracy of OSNA in SLNs without adjustment for TAB. FN, false negative; FP, false positive; TN, true negative; TP, true positive.

FIGURE 11.

(a) Meta-analysis and (b) forest plot of the test accuracy of OSNA in SLNs with adjustment for TAB. a, Adjusted results of the SLN sample reported by Feldman et al. 54 (see Table 16), which includes 26 cases that could not be evaluated for TAB; b, adjusted results of the ‘validation’ SLN sample reported by Khaddage et al. 57 (see Table 18), which includes two discordant cases that could not be evaluated for TAB; c, adjusted results of the SLN sample reported by Le Frère-Belda et al. 58 (see Table 17, top), which includes 22 discordant cases that could not be evaluated for TAB; d, adjusted results of the SLN sample reported by Tsujimoto et al. 64 (see Table 23, middle), which includes one discordant case that could not be evaluated for TAB. FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Figures 12 and 13 show the results of the meta-analyses of the test accuracy of OSNA in ALNs before and after adjustment for TAB respectively. The bivariate method converged with no abnormal results. Note that there were six studies to inform the meta-analysis before adjustment for TAB but only four studies to inform the meta-analysis after adjustment and we believe that this accounts for the significantly larger confidence region shown in Figure 13 as removing two studies from the subgroup without adjustment for TAB gives a confidence region of a comparable size.

FIGURE 12.

(a) Meta-analysis and (b) forest plot of the test accuracy of OSNA in ALNs without adjustment for TAB. a, Results of the trial 1 ALN sample reported by Tamaki et al. 62 (see Table 21); b, results of the trial 2 ALN sample reported by Tamaki et al. 62 (see Table 21). FN, false negative; FP, false positive; TN, true negative; TP, true positive.

FIGURE 13.

(a) Meta-analysis and (b) forest plot of the test accuracy of OSNA in ALNs with adjustment for TAB. a, Adjusted results of the ALN sample reported by Schem et al. 60 (see Table 19), which includes 13 discordant cases that could not be evaluated for TAB; b, results of the trial 2 ALN sample reported by Tamaki et al. 62 (see Table 21); c, adjusted results of the ALN and SLN sample reported by Tsujimoto et al. 64 (see Table 23, top); d, adjusted results of the ALN sample reported by Visser et al. 65 (see Table 24), which includes seven discordant cases that could not be evaluated for TAB. FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Study design

Both studies were observational and are therefore open to bias and confounding. They both stated their research question and the approach to economic evaluation but no justification was given by either study for the economic evaluation study design used in relation to the research question. The viewpoint of both analyses was implicitly justified by the public health systems in which the studies were conducted and the local practice at the respective centres, and both studies acknowledged the limited generalisability of their findings because of the availability of different intraoperative testing technologies in other centres.

Data

Details of the methods of patient recruitment were given. Although both studies were based on single-gate designs and are thus subject to TAB, neither adjusted for it in the analysis of accuracy. Both studies reported the methods of collecting health-care resource use data and applying unit costs to them, but only the Spanish study56 explicitly stated the primary outcome measures of its evaluation in terms of the health benefits of intraoperative diagnosis. Both studies reported unit costs and quantities separately but they provided explanations only for the estimation of unit costs not quantities. The Spanish study56 did not state the date for the unit costs used and neither study provided details on whether any price and currency conversion adjustments were made. Neither study valued health benefits nor examined changes in productivity or its associated costs.

Analysis and interpretation of the results

Neither study analysed outcomes beyond the end of the diagnostic phase and therefore the studies did not require the use of a discount rate. As no sensitivity analyses were provided, the degree to which the reported cost differences may have been the result of chance alone cannot be established.

In summary, only one study was found for OSNA56 and none for Metasin. One study was found on GeneSearch,123 which served as the basis for the development of Metasin (see Chapter 1, Metasin) but may not produce the same outcomes. Although the OSNA study56 is likely to reflect the study centre’s practice in Spain, the GeneSearch study123 is no longer relevant as the technology has been withdrawn from the market. The results in both studies are likely to be biased because of the inherent limitations of observational studies, which commonly lack control for the effect of confounders on outcomes. The validity of these studies’ findings is made more uncertain by their lack of reporting of the methods used to measure diagnostic accuracy outcomes. Further, the degree of uncertainty in the estimates, and the extent to which the observed differences may be explained by chance, cannot be established. Nor can it be known to what extent the results may be generalisable across countries or indeed jurisdictions within the UK. This suggests that the existing evidence on economic outcomes is unlikely to serve to inform medical decision-making in the context studied here.

Diagnostic pathway

This section of the model is used to both inform the rest of the model and provide the intermediate outcomes described in the definition of the decision problem (see Chapter 2, Decision question).

The Evidence Review Group diagnostic pathway

Patients enter the model having had a SLNB performed during their initial tumour removal. The model then splits into three different strategies to encompass each of the possible combinations of diagnostic tests: intraoperative only (OSNA or Metasin, not in combination), histopathology only or a combination of an intraoperative assay and follow-up histopathology. These three strategies are shown in Figure 14. In the following descriptions, positive test results refer to results that indicate metastases in the SLN. The three modelled pathways are therefore:

• Current practice. The SLNB is analysed using histopathology of the full node. If positive for metastases a second surgery is performed in which ALND occurs.

• Add-in strategy. Half of the SLN from the SLNB is analysed during removal of the tumour using one of the intraoperative tests (OSNA or Metasin). Those with a positive result receive ALND during that surgery. For those with a negative result, the other half of the SLN is kept to be analysed by histopathology. Patients whose metastases were not detected at the intraoperative stage but whose histopathology is positive receive ALND as a second operation.

• Replacement strategy. The full SLN is assessed using the intraoperative test, with no histopathology. Those with a positive result will receive ALND during their tumour removal.

FIGURE 14.

Diagnostic pathway to test for axillary metastases.

At this stage we calculate the intermediate costs and patient outcomes. This analysis was developed in Microsoft Excel 2010 (version 14.0.7128.5000, Microsoft Corporation, Redmond, WA, USA).

Post diagnosis (management pathway)

After the diagnostic pathway our cohort splits into various management subgroups:

• Patients with SLN metastases who test positive for metastases and who receive ALND (in either their first surgery or as a separate second surgery for ALND) and who also test positive for additional axillary metastases when the material from their ALND is tested. The proportion of patients assumed to have additional axillary metastases after a false-positive SLN metastases diagnosis is assumed to be small enough that it can be modelled as 0%. The proportion of patients assumed to have additional axillary metastases after a true-positive SLN metastases diagnosis is taken from the Z0011 trial37 and is set to 27.3%.

• Patients who test positive for metastases and who receive ALND (in either their first surgery or as a separate second surgery for ALND) but who do not test positive for additional axillary metastases when the material from their ALND is tested. These patients include those who were correctly identified as having SLN metastases and those who were incorrectly identified as having SLN metastases.

• Patients without SLN metastases who test negative for metastases and do not receive ALND.

• Patients with SLN metastases who test negative for metastases and do not receive ALND.

A visual representation of these subgroups is provided in Figure 15.

FIGURE 15.

Post-diagnosis subgroups. This diagram shows the possible pathways of patients immediately post diagnosis and their true SLN metastases status.

Once these subgroups are established, the model moves into its second section: the management pathway. This section of the model calculates the long-term outcomes, most notably the costs and QALYs. We model each of the groups from this point using as a basis the discrete event simulation (DES) model previously utilised by the School of Health and Related Research Technology Assessment Group (ScHARR-TAG) team,9,122 with updated parameters.

The School of Health and Related Research model structure

The ScHARR uses a DES model built in Simul8 2009 (Simul8 Corporation, Boston, MA, USA) for its economic evaluation of diagnostic imaging leading to SLNB or ALND without SLNB (without intraoperative testing) for the diagnosis of metastasis in early breast cancer. A DES models individual patients with individual attributes, such as the patient’s diagnostic and disease history. Unlike Markov state transition models, in which the transition probabilities are evaluated during fixed time intervals, the transit of patients through states in a DES may occur continuously according to a statistical distribution function. If it is possible for a patient to make a transition from the current state to multiple states then the model samples the time to each possible destination and compares them. The destination state with the shortest sampled time is the one to which the simulated patient moves to. 9 DES models are more flexible than Markov models but, as they are usually more complex, they take longer to build and run and, more importantly, are more demanding in terms of evidence to populate their parameters. 127 As the existing ScHARR model had already been subject to peer review,122 and because of the high element of uncertainty involved in extrapolating outcomes beyond the diagnostic phase in this evaluation, the benefits of building a de novo model did not justify the costs.

A basic layout of the ScHARR model is provided in Figure 16. Once the process of testing for SLN metastases, surgery for tumour removal and possible ALND has been completed, patients enter a state of adjuvant therapy. This involves chemotherapy and hormonal therapy (when appropriate) for those diagnosed with metastases and hormonal therapy alone (when appropriate) in patients diagnosed without SLN metastases. After adjuvant therapy patients may move into a disease-free state (post-adjuvant therapy state) and potentially stay there for the rest of their lifetime. During or after adjuvant therapy, some patients may have a locoregional relapse (see Glossary). Patients in the post-adjuvant therapy state may experience locoregional or metastatic relapse. Patients in the locoregional relapse state may go into remission or advance to metastatic relapse. Patients in remission may remain in that state until death or enter a metastatic relapse state (for the purposes of the model there cannot be a further locoregional relapse). The model does not consider metastatic disease curable and therefore patients in this state may move to a death state only, either from breast cancer or other causes. Patients in all states can die from other causes.

FIGURE 16.

The ScHARR model for post diagnosis of axillary metastases.

The authors of the ScHARR model claimed that there were problems in deciding whether ALN metastases identified during follow-up are the result of recurrence or a previous misdiagnosis (a previous false-negative test). As this problem was beyond the scope of their project, they did not explicitly model it. In practice, it is difficult to obtain conclusive evidence that a given recurrence results from secondary metastatic spread (from some subclinical distant site), and recurrence in the axilla would be presumed to be disease that already existed at presentation but which was not eradicated by standard therapy, that is, in a LN that was not removed by ALND. The same argument applies to extra-LN recurrences, which represent cancer cells not removed by surgery or eradicated by systemic therapy. This issue is similarly beyond the scope of our assessment and therefore we adopt the model’s approach as reasonable for our purpose.

The following are the ScHARR model assumptions:

• Lymphoedema is the only long-term adverse event considered and is classified as either mild/moderate or severe, based on the research data available. It affects both costs and quality of life for the rest of the patient’s life. We also model only lymphoedema as no relevant papers on other long-term adverse events were found. To incorporate the effect of lymphoedema on patients we populated the respective parameters using evidence from studies of patient-reported outcomes for lymphoedema.

• Adjuvant therapy occurs for a maximum 5-year period, consistent with the recommended follow-up period stated in the NICE guidelines for early breast cancer,13 but the model allows for patients to move to another state before the end of this 5-year period. In this state, patients who test positive for additional axillary metastases, confirmed on ALND, receive chemotherapy for 6 months followed by hormonal therapy for 4.5 years. Patients who test negative for SLN metastases or who test positive for SLN metastases only and not for further axillary metastases on ALND receive hormonal therapy for 5 years.

• After a locoregional relapse further locoregional relapse cannot occur; only metastatic relapse can occur. This is a simplification of clinical practice but was retained so that the model could be kept simple and because this was not the main focus of our assessment.

• Death rates for non-breast cancer causes are based on UK mortality statistics and are applied across all health states. These are not adjusted to exclude breast cancer mortality and so they may overestimate the risk of dying from non-breast cancer causes. In our model we use mortality statistics for England as this was the population stated in the objective in our protocol. As in the ScHARR model we use the statistics for women only as a very small proportion of breast cancer cases (< 1%) occur in men. 14

Because we relied on the treatment phase of the ScHARR model with limited modifications, we quality assessed the model and found that it met the standard quality criteria required for health economic evaluation models. 125 The ScHARR model depicts a very similar problem to that analysed in this study, but does not consider intraoperative testing approaches. Although this is presumably because intraoperative tests were outside its remit, the authors do not state this. However, this has a limited impact as it affects only the diagnostic stage of the model, which we do not use. The model does not value productivity changes, which are beyond our remit too. Overall, the ScHARR model was well reported and transparent and we therefore felt comfortable adapting it to our purposes. During the course of our review process we have worked with two of the ScHARR model authors in adapting and updating their model for our purposes; one (YM) has become a co-author of this report and the support of the other (KC) has been acknowledged.

Diagnostic pathway

Treatment phase

Discount rate

Costs and utilities are discounted at a rate of 3.5% in accordance with NICE guidelines. 147

Intermediate results

The following measures were used to synthesise costs and health benefits in the diagnostic phase:

• incremental cost per patient

• incremental cost per case correctly diagnosed

• incremental cost per additional positive case detected

• incremental cost per additional negative case detected.

Although we had set out to present, in addition to these measures, estimates of the cost per second operation avoided, this measure turned out to be of no additional informational value and these results are therefore not presented.

Long-term analysis

The following measures were used to synthesise costs and health benefits over the lifetime of patients:

• incremental costs

• incremental QALYs

• incremental cost per QALY gained.

Because the status quo, histopathology, was assumed to be the gold standard in the clinical effectiveness studies, it tended to be the option with the highest benefits (only one study reported evidence that diagnostic accuracy with OSNA was sufficiently close to that of histopathology to generate more QALYs with OSNA than with histopathology). To avoid the complication in interpretation that arises with the incremental cost-effectiveness ratio (ICER) of the new technology, intraoperative testing, which has lower costs than the status quo, the results are presented for the more relative to the less effective among the three options, of half-node intraoperative diagnosis, full-node intraoperative diagnosis and histopathology, according to total QALYs.

Diagnostic phase

The studies used to inform the economic analyses are presented in Table 40 (see Chapter 3, Assessment of study quality, for a discussion on the strengths and weakness of the selected studies). For the purpose of the economic analyses the results are presented by test option and whether estimates of test accuracy were controlled for TAB.

Ranking the diagnostic strategies by diagnostic accuracy and comparing the incremental costs and yields between increasingly accurate diagnostic strategies allowed the calculation of incremental costs per additional patient correctly diagnosed ratios and the identification of dominated options, that is, those that had higher costs and lower yields than the alternative. Table 49 summarises the cost–accuracy analysis for accuracy estimates unadjusted for TAB. Under the NHS reference costs130 costing system, half-node OSNA dominated histopathology for each additional node-positive case detected, but histopathology dominated half-node OSNA for each additional node-negative case detected and had an incremental cost per additional negative case detected of £8994 relative to full-node OSNA, which is lower than the conventional cost-effectiveness threshold. When YHEC cost values124 were used, histopathology dominated half-node OSNA for all measures of accuracy and consistently had an ICER of < £13,000 compared with full-node OSNA.

The discrepancy between the YHEC cost results and the NHS reference cost results occurred for a couple of reasons. First, YHEC costs for surgery were significantly lower than the NHS reference costs for surgery, giving greater weight to other costs, such as adverse event costs, but there was also a difference in the ratios between the different types of surgery (breast surgery with SLNB alone, breast surgery with SLNB and ALND, breast surgery for ALND alone). In particular, the cost ratio between histopathological and intraoperative test-positive patients using YHEC costs was significantly lower than that obtained using the NHS reference costs. This meant that the cost impact of a second surgery for patients diagnosed by histopathology was greatly reduced and costs such as the additional cost of histopathology in OSNA test-negative patients and the cost of short-term adverse events in the half-node strategy had a greater influence. This made the half-node strategy more expensive than histopathology alone. This gave rise to the seemingly inconsistent results between the NHS reference cost analysis, in which half-node OSNA was not dominated by histopathology, and the YHEC cost analysis, in which half-node OSNA was dominated by histopathology.

This analysis was repeated for the three individual OSNA studies that adjusted for TAB and the incremental results are reported in Table 50. Similar results were found for the studies by Le Frère-Belda et al. 58 and Snook et al. ,61 but not for the best-case scenario provided by Khaddage et al. ,57 in which sensitivity was 100% and specificity was 97.2%. Here, full-node OSNA had the same accuracy as half-node OSNA and so histopathology was compared directly with full-node OSNA. Full-node OSNA dominated histopathology for a node-positive diagnosis and any additional node-negative patients detected by histopathology cost £27,300 per case identified under the NHS reference costs. This reduced to £17,100 under YHEC costing.

Part of the cost-effectiveness assessment of the short-term outcomes (i.e. of the diagnostic phase) examined the disutility of second operations and waiting for histopathology results. The diagnostic costs and utility per patient are reported in Table 51. For strategies that do not involve histopathology, this utility was 1 as there was neither the need to wait for test results nor a second surgery directly resulting from the intraoperative test. Using the NHS reference cost scenario, full-node OSNA dominated half-node OSNA and histopathology as its increased QALY gains, measured until the end of the diagnostic phase, were less costly. Under the YHEC costing strategy, half-node OSNA provided a small QALY gain over histopathology, with an ICER of £4832 per QALY gained, but full-node OSNA continued to dominate half-node analysis and histopathology. It is estimated that 4.1% of the 76.5% of patients who have to wait for confirmatory postoperative histopathology results under the half-node testing strategy end up with a positive diagnosis as opposed to the 20% expected positivity rate under the postoperative histopathology-only strategy.

According to these short-term costs and benefits, full-node OSNA is the most cost-effective test for short-term utility. Using the meta-analysis accuracy values for OSNA (sensitivity 84.5%, specificity 91.8%), histopathology appears to be cost-effective in terms of short-term accuracy, although this begins to look questionable when the effect of TAB is accounted for and OSNA has a higher sensitivity and specificity. In all scenarios OSNA is less costly than histopathology.

Long-term analysis

Thus far, the analysis has considered only the short-term costs and consequences of the different diagnostic options, effectively assuming that no utility benefits accrue from increased diagnostic accuracy. In the long-term scenario we examined costs and QALYs, which are presented in Table 52 for the studies with no adjustment for TAB, and account for all costs and the benefits of accurate diagnosis through improved patient management net of the disutility of anxiety from waiting for the postoperative diagnosis and undergoing a second operation. The strategies were then ordered by increasing number of QALYs, with full-node OSNA having the fewest QALYs (9.222) and histopathology having the most (9.321). As this demonstrates, the QALY difference was approximately 0.10 (i.e. the equivalent of 5 weeks of life in full health). This QALY difference occurs as a direct consequence of the accuracy of the tests, as a higher accuracy leads to a greater number of correct diagnoses. As the purpose of diagnosis is to inform management strategies, an increase in the number of correct diagnoses leads to an increase in the correct management of patients and therefore patients gain the most QALYs possible. The QALY gain from the higher accuracy of histopathology is small, but it may be seen as cost-effective; under the NHS reference costing and ignoring half-node OSNA, as this is a strategy that is extendedly dominated and unlikely to continue once OSNA has been validated, histopathology had an ICER of £4324 per QALY gained compared with full-node OSNA. Under YHEC costing, histopathology dominated half-node OSNA, that is, OSNA had higher costs and produced fewer QALYs. In this scenario the ICER comparing histopathology with full-node OSNA was £2150 per QALY gained.

For the TAB-adjusted studies (presented in Table 53), half-node OSNA remained extendedly dominated using the Le Frère-Belda58 values. In this case, histopathology compared with full-node OSNA had ICERs of £9493 per QALY gained using NHS reference costs130 and £5215 using YHEC costs. 124 Using the values of Snook et al. ,61 half-node OSNA was extendedly dominated using YHEC costs (and histopathology had an ICER compared with full-node OSNA of < £4850 per QALY gained) but was no longer extendedly dominated using NHS reference costs, and had an ICER compared with full-node OSNA of £8063 per QALY gained. Comparing histopathology with half-node OSNA, the ICER was £14,967 per QALY gained. More importantly, Khaddage et al. 57 suggested a much better result for OSNA, with full-node OSNA dominating half-node OSNA and histopathology (full-node OSNA had a higher QALY gain and lower costs). This was because of the influence of the short-term disutility for anxiety and a second operation applied (when applicable) to patients undergoing histopathology testing in the histopathology and OSNA half-node arms. Without this disutility the results using the Khaddage et al. 57 values for sensitivity and specificity altered slightly so that, although in the long term half-node OSNA was still dominated by full-node OSNA, this was now because they had the same QALY gain but full-node OSNA was less expensive. The ICER between histopathology and full-node OSNA was £68,432 per QALY gained using NHS reference costs and £41,619 per QALY gained using YHEC costs, suggesting that histopathology was not that cost-effective compared with full-node OSNA.

Accuracy

As the TAB-adjusted results have already indicated, test accuracy has a direct impact on the cost-effectiveness of the tests. A threshold analysis was therefore conducted to investigate sensitivity and specificity separately. In the case of threshold analysis for sensitivity, specificity was held constant and sensitivity was increased in steps of 5% over a range of 70–100%. The opposite was then performed for specificity. This sensitivity analysis was conducted on the full-node OSNA results, for which the change in accuracy would be most notable (the overall sensitivity for half-node OSNA was fixed at 100%). Short-term utility results are not reported as the utility for OSNA was not affected by the accuracy of the test.

The short-term results of the threshold analysis for sensitivity are presented in Table 54. Table 54 demonstrates that, when specificity is held at 91.8%, the proportion of node-positive cases detected increased by 1% each time the sensitivity increased by 5%. The short-term costs of OSNA also increased by approximately £18 for each step increase in sensitivity (therefore decreasing the difference in costs between OSNA and histopathology) and the combination of both increased the cost–accuracy ICERs between histopathology and OSNA. The ICER for cost per case correctly identified increased from £5104 per additional case correctly identified by histopathology when OSNA had 70% sensitivity to £8162 per additional case correctly identified by histopathology when OSNA had 100% sensitivity. The ICERs for cost per additional node-positive case detected were consistently larger, ranging from £10,685 per additional node-positive case detected when OSNA had 70% sensitivity to OSNA dominating histopathology when OSNA had 100% sensitivity. However, in this case, the ICER increased much faster as the sensitivity of OSNA neared 100%, as Figure 17 shows.

FIGURE 17.

Comparison of cost–accuracy results for threshold analysis of OSNA sensitivity.

The long-term results, presented in Table 55, show a similar finding to the accuracy results: as the sensitivity of OSNA increased, so did the ICER for the cost per QALY gained by histopathology. This ranged from £2119 per QALY gained when OSNA had a sensitivity 70% to £14,193 per QALY gained when OSNA had a sensitivity of 95%. At 100% sensitivity OSNA dominated histopathology, having more QALYs at a lower cost. Unlike the accuracy results, the cost difference between histopathology and OSNA also increased each time the sensitivity increased. Again, as the sensitivity of OSNA neared 100% the ICER began to increase much faster, as demonstrated in Figure 18.