What carcinoembryonic antigen level should trigger further investigation during colorectal cancer follow-up? A systematic review and secondary analysis of a randomised controlled trial

Review question

The review question was, ‘What is the accuracy of single-measurement blood CEA as a triage test to prompt further investigation for colorectal cancer recurrence after curative resection?’.

Search

The search terms used are reported on the Cochrane Library website. 16 We avoided the use of search filters and did not restrict our review to English-language publications. Two reviewers (BN and IP) extracted data independently and, with a third reviewer (BS), independently assessed the methodological quality. We retrieved and analysed all full-text articles that we felt could be potentially relevant based on the title and abstract. We based additional searches on the citations of full-text articles to reduce the risk of missing relevant studies. Foreign-language articles were translated or assessed or both by colleagues of the authors who were proficient in the language in question.

Studies included

The following studies were included: cross-sectional diagnostic test accuracy studies, cohort studies or randomised trials, conducted in primary care or hospital settings, involving adults with no detectable residual disease after curative surgery (with or without adjuvant therapy) and reporting results extractable in a 2 × 2 format (i.e. test +/– by case +/–).

Index test and reference standard

The index test was blood CEA level. The reference standard was clinical diagnosis of recurrence of colorectal cancer following primary treatment confirmed by imaging, histology or clinical follow-up (for details of the reference standard, see the full report on the Cochrane website16).

Quality assessment

Two review authors extracted data independently and three authors independently performed a quality assessment [using a modified Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) schedule18] of the included studies, with subsequent discussion to reach consensus on overall judgements of risk of bias and applicability.

Statistical analyses

Binary diagnostic accuracy data were extracted from all excluded studies as 2 × 2 tables. We used the bivariate model to perform meta-analysis of sensitivity and specificity19 using the xtmelogit command in Stata version 12(StataCorp LP, College Station, TX, USA). 20 The quality assessment was used to identify the probable sources of heterogeneity (including threshold) and subgroup analyses and meta-regression were implemented to explore these factors.

As an exploratory analysis, we estimated the absolute numbers of false alarms (false positives) and missed cases (false negatives) per 1000 patients tested for each 3-monthly testing interval by applying the pooled sensitivity and specificity derived from this review to (1) the observed median reported prevalence of recurrence divided by 15 (because a recommended schedule of 3-monthly testing for 2 years followed by 6-monthly testing means that, typically, 14–15 tests are undertaken during 5 years of follow-up) and (2) the incidence of recurrence data per follow-up period reported by Sargent et al. 21 (because in reality the proportion developing recurrence between tests is not constant but falls over time).

Main findings

We identified 52 studies amenable to meta-analysis, including in aggregate 9717 participants [median sample size 139, interquartile range (IQR) 72–247]. The median proportion of recurrences in each study was 30% (IQR 24–36%). The diagnostic accuracy of CEA was reported at 15 different thresholds, ranging from 2 to 40 µg/l.

Diagnostic accuracy at the most commonly recommended threshold in national guidelines of 5 µg/l (23 studies, 44%) is shown in Figure 1. The filled black circle indicates that the pooled sensitivity is 71% (95% CI 64% to 76%) and the specificity is 88% (95% CI 84% to 92%). Each box in the figure represents the 2 × 2 data extracted from each study. The width of the box is proportional to the number of patients who did not experience recurrence in each study and the height is proportional to the number of patients who did develop recurrent colorectal cancer. The smaller dashed ellipse represents the 95% credible region around the summary estimate; the larger dashed ellipse represents the 95% prediction region for individual study estimates.

FIGURE 1.

Summary receiver operating characteristic (ROC) plot of CEA accuracy at detecting recurrence at a threshold of 5 µg/l (reference standard clinical diagnosis of recurrence confirmed by imaging, histology or clinical follow-up; see the Cochrane Library website16 for full details of authors’ definitions). This figure is a copy of Figure 8 in the review; copyright is held by the Wiley Online Library and it is reproduced with their permission (http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD011134.pub2/full; last accessed 6 November 2016).

Implications of applying other thresholds

In the seven studies that reported accuracy at a threshold of 2.5 µg/l, the pooled sensitivity was 82% (95% CI 78% to 86%) and the specificity was 80% (95% CI 59% to 92%). Seven studies also reported accuracy at a threshold of 10 µg/l, with a pooled sensitivity of 68% (95% CI 53% to 79%) and a specificity of 97% (95% CI 90% to 99%). Table 1 shows the implications of applying these thresholds in clinical practice, assuming a 2% incidence of recurrent disease in each testing interval. The estimate of 2% is based on an observed median prevalence of recurrence of 30% in the studies included in this review and on national guidance to conduct 14 or 15 CEA tests during follow-up. 22

Implications of applying current UK national guidelines

Table 2 shows the clinical implications of applying the recommended absolute CEA threshold of 5 µg/l to trigger further investigation at each recommended test during 5 years of follow-up, applying the pooled estimates of sensitivity and specificity to the most recently published estimates of the number of cases of recurrence occurring in each test interval21 rather than the constant 2% incidence rate derived from the studies included in the review. As the sensitivity estimate applied is constant and independent of the number of cases of recurrence, the reported small differences in the percentage of cases missed reflects rounding; they are included to highlight the limited sensitivity of the test. The rate of false alarms (people investigated who do not have a recurrence) is dependent on the number of recurrences at each time point but the difference between the time points (range 78–92%) is less striking than the high level of false alarms at every time point if the recommended threshold of 5 µg/l is applied.

Quality and robustness

The subgroup analyses to assess the impact of variance in quality and analytical method applied by included studies focused on the 5 µg/l threshold. The two possible approaches to constructing the 2 × 2 tables (i.e. evaluate the CEA measurement taken closest to the time point at which recurrence was detected or look across all measurements to assess whether or not any had crossed the threshold during the entire follow-up period) gave similar estimates of test performance: the estimated sensitivity was 69.0% and 64.5% and estimated specificity was 90.0% and 89.5%, respectively.

Although we were unable to carry out a subgroup analysis based on specific laboratory techniques because of incomplete reporting, we were able to compare studies carried out before and after the introduction of the International Reference Preparation (IRP) 73/601 calibration. 23 Before the introduction of the IRP, the pooled sensitivity was 73.6% and pooled specificity was 88.5%; after IRP introduction, pooled sensitivity was 67.9% and pooled specificity was 88.6% (95% CI 80.0% to 93.7%). The effect of the covariate in the meta-regression was non-significant (p = 0.96).

Restricting the analyses to the studies deemed to be at low risk of bias in the patient selection, index test, reference standard and timing/flow domains of the QUADAS-2 assessment made little difference to the estimates of pooled sensitivity and specificity and, when the risk of bias in each domain was added as an ordinal covariate in the meta-regression analysis of diagnostic accuracy, the effect on the model was non-significant (patient selection p = 0.77; index test p = 0.90; reference standard p = 0.29; flow/timing p = 0.66).

Differences from the review cited in the original application

The systematic review cited in the original application13 identified only 20 studies and reported the accuracy of CEA for the diagnosis of colorectal cancer recurrence using the Moses–Littenberg method. 24 We identified more than twice as many studies and implemented the bivariate meta-analyses recommended in the Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. 16,19,25 This is a statistically more rigorous method that directly accounts for the correlation between sensitivity and specificity. However, our pooled estimate of specificity at a CEA threshold of 5 µg/l was the same as that of Tan et al. 13 (88%). Although our pooled estimate for sensitivity was noticeably higher than that of Tan et al. 13 (71% vs. 63%), the variance around both estimates means that the difference is not statistically significant.

The method used by Tan et al. 13 to identify 2.2 µg/l as the ‘optimum’ CEA threshold was based on linear extrapolation beyond the data (the lowest threshold included in their study was 3 µg/l). We now question the recommendation of 2.2 µg/l (which was based on achieving high sensitivity), not just on the basis of the low specificity (and high false-alarm rate), but also because there appears to be a ‘ceiling’ effect in terms of sensitivity; even at a threshold of 2.5 µg/l, around one in five cases of recurrence would be missed. The failure to exceed a sensitivity of about 80% even with a low threshold and poor specificity reflects the well-documented fact that some recurrent cancers are not associated with a rise in blood CEA level. Our interpretation of the data is therefore that (1) the use of an absolute single test threshold should be questioned; (2) if an absolute single test threshold is applied, it should be higher than the level of 5 µg/l recommended in most national guidelines, using another monitoring modality in parallel with CEA level (such as a single CT scan at 12–18 months) to deal with the ‘sensitivity gap’ – trying to increase sensitivity by lowering the CEA threshold achieves little at high cost.

Limitations of the review

The major limitation of the review is the quality of the studies included. There was also considerable between-study variation in the reporting of factors known to impact on CEA level (e.g. stage of primary disease included, approach to ensuring no residual disease, reporting of smoking, reporting of chemotherapy treatment, and the location of recurrence). Over half of the included studies were at high risk of selection bias, mainly because of inappropriate patient exclusions. The methods used to measure CEA were also poorly reported: three studies (6%) did not report the CEA threshold used to determine a positive result, 15 studies (29%) did not report which laboratory technique had been used and 43 studies (83%) failed to report any indicator of method accuracy or an estimate of CEA test reproducibility.

The reference standard was also inadequately described in many studies. In nine studies the reference standard was assessed only if a rise in CEA was detected, raising the possibility of verification bias. Furthermore, most studies implemented a composite reference standard but failed to report consistently which investigation (within the composite) actually diagnosed recurrence; in half of these studies, positive results for certain reference tests triggered the use of other reference tests. Although three studies were assessed as having no risk of bias or applicability concerns, they reported accuracy at different CEA thresholds, implemented different CEA laboratory techniques and used differing composite reference standards to detect recurrence. It was therefore infeasible to provide pooled diagnostic accuracy estimates restricted to these studies.

The time between the CEA measurement and the reference test used in the 2 × 2 table was not reported in any of the studies. There is therefore a high chance of misclassification because of disease progression during the time between the CEA test and the reference test. Understanding this relationship is important because (1) a high-grade recurrence will progress more quickly than a low-grade recurrence and (2) this information is required to estimate lead time. Furthermore, no study reported 2 × 2 data for each 3- to 6-month period of follow-up, which would be desirable given that the incidence of recurrence falls over the 5 years of follow-up.

Implications for the main analysis

Despite the difference between our findings and those of Tan et al. ,13 in most aspects the review underlined the importance of the questions and analysis set out in our protocol. It highlighted a number of methodological aspects of the FACS study data, including four key strengths: CEA measurement was centralised in a laboratory with high-level quality assurance, the time between index test and diagnosis is known, the reference standard was quality assured [by local oncology multidisciplinary teams (MDTs)] and we obtained data on recurrence and test performance at each testing point during the scheduled 5 years of follow-up. It also highlighted an unanticipated weakness, namely the potential for verification bias. In the FACS trial, the threshold for triggering further investigation was 7 µg/l. When we submitted the application we assumed that our interest would be limited to thresholds below this level. The review meant that this was not the case.

Before we undertook the review, we were less clear about the plateau effect on sensitivity (i.e. decreasing the absolute threshold does not increase sensitivity above what appears to be a maximum achievable value of about 80%). This finding made us expand the planned analysis to investigate whether factors other than preoperative CEA level could characterise in advance those at high risk of experiencing recurrence without a rise in CEA. The high rate of false alarms identified by the review also reminded us of the importance of characterising test performance in terms of predictive value as well as sensitivity and specificity. The failure of previous studies to report and adjust for factors other than cancer recurrence that are known to influence CEA levels also encouraged us to investigate the potential of such adjustment to improve test performance.

We had already specified in the original protocol that we would investigate (1) whether making the decision to investigate further on the basis of the trend in CEA levels over time, rather than the absolute level of a single test, would provide better diagnostic accuracy and (2) whether the recommended testing intervals in national guidelines (commonly every 3 months for 2 years and then every 6 months for 3 years) are optimal. The review reminded us that these two issues are likely to be interdependent. It also reminded us of the limitations of retrospective analysis in predicting clinical performance over time. We therefore augmented the static analytical method described in the original protocol by attempting to model real-time performance prospectively.

Implications for future research (other than our main analysis)

The generic outcome from this review is the overall poor quality of reporting of diagnostic accuracy studies in this field. This poor reporting is compounded by the considerable between-study heterogeneity and limitations of study quality. In response to the methodological limitations highlighted in this review, authors of future research investigating the diagnostic accuracy of CEA should take care to report the CEA threshold and technique used, with an indication of method accuracy and CEA reproducibility; the time point at which the CEA level is measured; the reference test used; and the time between the CEA test and the reference test.

The lack of significant improvement in diagnostic accuracy following a sensitivity analysis limited to studies at low risk of bias (based on a QUADAS-2 assessment) also suggests that modifications to the QUADAS-2 schedule may be warranted in assessing the quality of diagnostic tests used for follow-up monitoring. Any future studies commissioned would be more useful if augmented by a cost–benefit analysis of different strategies for the timing of monitoring tests and the optimal combination of CEA blood testing and CT imaging.

Evaluating the diagnostic accuracy of carcinoembryonic antigen level as a single diagnostic test

To evaluate the accuracy of CEA for diagnosing recurrence based on the data from the FACS trial, receiver operating characteristic (ROC) curves are presented alongside 95% bootstrap CIs using the pROC package in R version 3.1.3 (The R Foundation for Statistical Computing, Vienna, Austria). 26 CIs were calculated using 2000 stratified bootstrap replicates. The areas under the receiver operating characteristic curves (AUCs) are also reported. Sensitivity and specificity, likelihood ratios and predictive values (along with their respective 95% CIs) are summarised for the most commonly recommended and implemented threshold of 5 µg/l.

In contrast to the traditional diagnostic accuracy paradigm in which only one measurement is taken, in this scenario we evaluated multiple CEA measurements taken within a single individual. Two alternative methods for evaluating the accuracy of CEA testing were identified in our Cochrane review: some studies reported the accuracy of the CEA measurement closest to the time at which recurrence was detected by the reference standard, whereas others defined the CEA level as positive if any CEA measurement crossed the threshold at any time within the follow-up period. As the latter method better represents how CEA levels are interpreted in practice, that is, CEA is monitored and all CEA measurements are interpreted prospectively, we applied this method in the analyses here.

For comparison with the results of the Cochrane review, we also report as a secondary analysis results based on the measurement taken closest to the time at which recurrence was detected. For patients who did not experience recurrence, the CEA measurement taken at the end of follow-up was evaluated.

An operational analysis of the probable impact of CEA testing if used prospectively in clinical practice was also conducted, hypothetically applying the four most commonly reported thresholds in the systematic review (2.5, 5, 7.5 and 10 µg/l) to trigger further investigation on the basis of the result of each individual test carried out during the follow-up period. Two outcomes are reported: (1) the proportion of recurrences that would have been missed and not referred for further investigation because the CEA level was below the threshold and (2) the proportion of patients with a CEA level above the threshold who were subject to a false alarm (i.e. an unnecessary referral for further investigation of a patient who did not experience recurrence throughout the whole follow-up period). The data for the 7.5-µg/l and 10-µg/l thresholds are less robust than those for the 5-µg/l and 2.5-µg/l thresholds, as these thresholds were above the action threshold applied in the FACS trial and therefore were influenced more by work-up bias.

We also calculated for each individual the lead time (i.e. the number of days earlier that the CEA test would have triggered further investigation compared with the reference standard) by applying different thresholds retrospectively. As a CEA value of 7 µg/l triggered further investigation within the trial, we could look only at thresholds up to this level. The median lead time across all individuals, alongside the proportion of recurrences detected earlier than by the reference standard, is presented.

Evaluating the diagnostic accuracy of baseline-adjusted carcinoembryonic antigen testing

The aim of this analysis was to ascertain whether or not individualising the interpretation of CEA results by adjusting for each individual’s postoperative CEA measurement improves the diagnostic accuracy of CEA testing. Two methods of adjustment were compared: (1) taking the difference between the CEA level at each time point and each individual’s postoperative CEA level and (2) using the ratio between the CEA level at each time point and each individual’s postoperative CEA level.

Comparative ROC curves are presented, again assuming that the CEA test was positive if any CEA measurement crossed the threshold at any time within the follow-up period. The AUC for each interpretation method is also reported. The median lead time and proportion of false alarms (as previously defined) were compared across interpretation methods by selecting fixed levels of sensitivity (ranging from 60% to 80%).

Evaluating the effect of preoperative carcinoembryonic antigen levels on the subsequent performance of carcinoembryonic antigen testing to detect recurrence during follow-up

Because of the limited number of preoperative CEA measurements available, a comparative ROC analysis broken down by preoperative CEA level was not possible. Instead, we report a very simple 2 × 2 analysis taking the difference between individuals’ CEA levels at the time that recurrence was diagnosed (i.e. when it should be most elevated) and their preoperative CEA levels and using a threshold of 2.5 µg/l.

Exploring factors that may predict missed cases and false alarms

The following factors were included in this analysis: (1) patient characteristics (age and smoking status); (2) primary tumour characteristics [colon or rectum, N stage (number of nearby lymph nodes involved) and whether treated with chemotherapy or radiotherapy]; (3) the time elapsing between surgery and the first follow-up CEA measurement; and (4) the site of recurrence.

The relationship between these variables and a ‘missed case’ (defined here as a patient suffering a recurrence without a rise in CEA level above 5 µg/l) and a ‘false alarm’ (defined here as a patient who does not have a recurrence during the follow-up period but who has at least one CEA measurement above 5 µg/l) was explored. Univariate relative risks are reported alongside adjusted ORs derived by logistic regression in which all of the listed explanatory variables have been forced into the model. These analyses were carried out in Stata version 12.

Diagnostic accuracy of trend in carcinoembryonic antigen measurements over time

To investigate the diagnostic accuracy of assessing the trend in serial CEA measurements within an individual rather than simply interpreting the most recent CEA measurement taken, linear regression models were fitted to the CEA values for each individual over time. For those patients who recurred during the follow-up period, all CEA measurements up to the point at which recurrence was detected were included in the model. For patients who did not recur during the follow-up period, all measurements available were modelled. Patients who recurred in the first 6 months were excluded from these analyses, as there were insufficient measurements to depict a trend.

The distribution of slope coefficients for individuals who did and did not experience recurrence was compared and ROC analysis was implemented to evaluate the diagnostic accuracy of CEA trend. The preliminary analysis reported in 201427 suggested that we should not restrict the analysis to downwards trend.

In practice, trends in CEA measurements would be evaluated prospectively and modelling all of the available CEA measurements for those who do not go on to recur does not represent what would happen in clinical practice. To overcome this and to ensure that the results better represent what would occur in clinical practice, we have broken down the ROC analysis by year of recurrence so that only the measurements that would be available up to that year are included in the linear regression models.

Optimal testing interval

The optimal testing interval was explored in two ways: (1) by estimating the lead time (the time elapsed between the rise in CEA level that triggers further investigation and the confirmation of a diagnosis of recurrence) that would potentially be gained by initiating further investigation at different CEA levels; and (2) by describing when recurrences occurred during the 5-year follow-up period. The number of recurrences potentially detectable during each 3- or 6-monthly testing interval (i.e. the interval between any two CEA tests) was then estimated by subtracting different amounts of lead time (0, 3, 6 and 9 months) from the actual time elapsed between the start of follow-up and the actual date of confirmation of recurrence.

Baseline adjustment

Figure 4 compares the diagnostic accuracy of adjusting the CEA value for the baseline (postoperative) level with the diagnostic accuracy of simply interpreting the CEA result. The ROC curves for each method of interpretation substantially overlap, as do the 95% CIs around the estimates of the AUCs, suggesting that adjusting the CEA value by an individual’s baseline measurement offers no notable improvement in diagnostic accuracy.

FIGURE 4.

Receiver operating characteristic plots comparing the diagnostic performance of the single test baseline-adjusted CEA value (expressed as a ratio and difference) with that of the unadjusted CEA value (reference standard: recurrent cancer as assessed by the local hospital MDT).

Interpreting the baseline-adjusted CEA level is complicated by the fact that CEA levels appear to continue to fall after the ‘baseline’ postoperative test. Of the 6611 CEA measurements in the database, 3344 (51%) were lower than their baseline measurement and therefore had a negative ‘difference’ value or ratio values of < 1. Nearly one-quarter (133/582, 22.9%) of patients always had a negative-adjusted value (i.e. all of their CEA measurements were less than the baseline), of whom 116 remained recurrence free but 17 developed recurrence.

Effect of baseline adjustment on lead time and false alarms

Table 5 reports the estimated median lead time across a fixed range of sensitivity levels from 50% to 80% for the two baseline-adjusted CEA values (difference and ratio) compared with the unadjusted CEA value. It also shows the percentage of recurrence-free patients who are unnecessarily referred for further investigation (false alarms). When considering plausible levels of sensitivities, using a baseline transformation is akin to lowering the positivity threshold.

Adjusting for presurgery carcinoembryonic antigen level

It has been suggested that individuals who do not present with an elevated CEA level pre surgery are less likely to produce elevated CEA levels during follow-up, despite the development of recurrence. 28 Only 25 patients with recurrence in this analysis had a preoperative CEA measurement. Table 6 shows that there is no indication that a low CEA level at the time of diagnosis of the primary tumour predicts CEA level at the time of recurrence, indicating that the secretory behaviour of the recurrent tumour is not determined by that of the primary tumour.

Predicting missed cases

The other characteristics of the patient and primary tumour assessed (e.g. patient age and smoking status, site and stage of the primary tumour, receipt of adjuvant therapy, delay in commencing monitoring) were equally unhelpful in predicting the likelihood of being a missed case (i.e. a patient suffering a recurrence without a rise in blood CEA level above the 5 µg/l threshold). Although there was a potentially important increase in missed cases in patients with delayed initiation of follow-up, it was not statistically significant (adjusted OR 2.66, 95% CI 0.87 to 8.15; p = 0.09).

Predicting false alarms

Table 7 explores the possibility of predicting which patients are likely to experience false alarms with CEA monitoring. Again, neither the site nor stage of the primary tumour, nor the site of recurrence, nor the timing of the start of follow-up are significantly predictive of false alarms, although the CIs on these estimates are wide. However, current smoking is significantly predictive of multiple false alarms (adjusted OR 6.55, 95% CI 1.52 to 28.2; p = 0.01). Current smokers have a 12% (95% CI 2.55% to 31.2%) chance of experiencing more than one false alarm during 5 years of follow-up.

Distribution of beta-coefficients

Figure 5 shows the distributions of the beta-coefficients (i.e. the gradient of the linear trend in the CEA measurements) for the group of patients who did develop recurrence and the group who remained recurrence free. The difference in variance around the mean of the two distributions is stark, indicating much greater stability in CEA levels in patients who do not experience recurrence. For the recurrence-free patients, the beta coefficients are centred on zero and show relatively little variability [mean 0.000006, standard deviation (SD) 0.00003]. For patients who suffered recurrence, the distribution is much wider, with a 100 times greater SD (0.0035 vs. 0.00003). Although the distribution for the patients who suffered recurrence centres above zero (mean 0.001), indicating that a positive trend is more indicative of recurrence, the extent of the variance suggests that a negative trend may also have some diagnostic value. The distributions of the intercepts were also plotted (not shown) and heavily overlapped, indicating that there is minimal discriminatory information contained in the intercepts of the linear regression models.

FIGURE 5.

Distribution of regression coefficients of CEA levels in patients with and without recurrence.

Diagnostic accuracy of a positive trend in carcinoembryonic antigen level

Figure 6 shows the ROC curve for basing the decision to investigate further on an increase in blood CEA level over time (using the beta-coefficients from the individual patient linear regression models) rather than on the result of a single test. The ROC curve shows a clear elbow towards the upper left-hand corner, indicating that the slope in CEA level holds some discriminatory information for predicting recurrence. The AUC suggests that the rate of change provides better overall discriminatory power than the single-value CEA transformations explored so far (AUC 0.85, 95% CI 0.78 to 0.91).

FIGURE 6.

Receiver operating characteristic plot for the diagnostic performance of a positive trend in CEA level (assessed by beta-coefficients derived by linear regression of CEA levels) (reference standard: recurrent cancer as assessed by the local hospital MDT).

A key limitation of this analysis is that it is retrospective and the beta-coefficients are based on all of the CEA measurements (up to the point that recurrence is clinically detected). In clinical practice the decision to investigate will be made prospectively as results accrue. However, excluding CEA measurements taken closest to the diagnosis of recurrence (up to 1 year) made no important or statistically significant difference to the AUC.

Effect of censoring

There is a trade-off in estimating CEA trend between improving the overall accuracy of the beta-coefficient by including more measurements and diluting the effect of an increase in later measurements (i.e. if you cumulatively include all measurements in calculating the regression line, if the CEA rise starts only late in follow-up, a bigger CEA rise than at the beginning would be needed to shift the curve). The plots in Appendix 1 (see Figure 10) show that a sudden late rise in CEA after a period of relative stability (i.e. an ‘elbow’ shape) was less common than a gradual slow rise and this is reflected in Table 8, which shows that there is no obvious gain in censoring the number of CEA tests used to estimate the beta-coefficient. If anything, ignoring earlier tests and calculating the slope only on the most recent tests reduces rather than improves diagnostic accuracy.

Diagnostic accuracy of a trend in either direction

The beta-coefficients reflect the individual plots shown in Appendix 1 (see Figure 10); 13.6% of patients with recurrence (n = 12/88) had a negative beta-coefficient, indicating that, overall, their CEA levels showed a downwards trend during the follow-up period. Given that the distribution of beta-coefficients for the recurrence-free group is tightly centred around zero, allowing a negative as well as a positive trend to trigger further investigation further improves the AUC (0.91, 95% CI 0.87 to 0.96), but the 95% CIs overlap substantially; thus, both the statistical and the clinical significance of this finding remain unclear.

Clinical application

As mentioned above, a key limitation of the accuracy evaluations of trend so far is that they have been based on a retrospective analysis of all available measurements. Figure 7 shows the accuracy of interpreting trends prospectively, as measurements are collected over the follow-up period. Based on the AUCs, although there is some between-year variation in accuracy, assessing trends in CEA levels over time still appears to be more accurate than interpreting just the most recent CEA measurement.

FIGURE 7.

Receiver operating characteristic curves showing the diagnostic accuracy of the trend in CEA level by year of follow-up (reference standard: recurrent cancer as assessed by the local hospital MDT).

Optimal test interval for a single test

Table 9 shows the mean number of recurrences detected in each year of the FACS trial and an estimate of the number of recurrences detectable in each testing interval assuming (1) an equal distribution of recurrence within the year; (2) that the current UK testing schedule of 3-monthly intervals in years 1 and 2 and then 6-monthly intervals in years 3–5 was used; and (3) that the CEA level will be above the CEA threshold between 0 and 9 months prior to the FACS reference standard detecting recurrence. The results are based on the actual threshold applied in the FACs trial of 7 µg/l.

The actual lead time that should be applied in interpreting Table 9 depends on the threshold used for interpreting the test. For example, as Figure 3 shows, a threshold of 5 µg/l is associated with approximately 3 months of lead time. At this threshold, the testing interval needs to be halved in year 1 to ensure that the number of recurrences detectable, and therefore test performance, remains fairly constant over time. As the FACS trial follow-up stopped at 5 years, it is not possible to estimate in the same way the effect of applying action thresholds of < 7 µg/l on the number of recurrences detectable in year 5. However, at this threshold, the number of recurrences detected was less than half that in years 3 and 4. Although this might suggest that the testing interval should be reduced from 6- to 12-monthly intervals to achieve constant performance, this would constrain any potential diagnostic lead time gained.

Optimal test interval for carcinoembryonic antigen trend

Although trend was estimated on two measurements in 19 participants (as stated earlier), it is self-evident that the beta-coefficient is assessed with increasing precision as more measurements accrue. In patients without recurrence, this improvement in precision levelled off after about 10 measurements. It is also self-evident that you cannot assess trend on the basis of a single measurement. This in itself implies the need to increase the frequency of testing in the first year as, at present (with 3-monthly testing), trend assessment would not be possible until 6 months and it would not be measured with any precision until 9 months, by which time 31 (29.8%) recurrences had already been diagnosed (and another 13 might already have been detected if a 5-µg/l threshold had been applied).

The beta-coefficient lacks clinical face validity and could probably be implemented only within a computerised algorithm, either calculating a risk of recurrence or giving dichotomous refer/do not refer advice. Table 10 therefore presents the beta-coefficients in terms of change in CEA level per year.

It is clear from Table 10 that (1) the optimal threshold (to maintain the selected trade-off between sensitivity and specificity) is not constant but falls over time and (2) to maintain a specificity of around 90% it is necessary to accept a sensitivity of 70% rather than 80%. Aiming for a sensitivity of 80% means accepting a level of specificity associated with a substantial number of false alarms.

The between-year instability in specificity estimates reflects the relatively small numbers contributing to the analysis when it is stratified by year. CIs are not given to avoid complicating the table, but Figure 7 shows that CIs around the AUCs for each year overlap.

The importance of not triaging with carcinoembryonic antigen level alone

Our main analysis confirms the findings of the systematic review: CEA testing alone is insufficient as a triage test for colorectal cancer recurrence. Whatever threshold is applied for interpreting the CEA test result (based on a single test or trend), a significant number of patients will suffer recurrence without a detectable change in CEA levels. This underlines the importance of combining CEA testing with scheduled imaging, as recommended in most national guidelines. However, this does not imply that CT imaging has to be carried out alongside every CEA test – in the comparative analysis of the FACS trial, combining 3- to 6-monthly CEA testing with a single CT scan of the chest, abdomen and pelvis at 12–18 months achieved similar results in detecting treatable recurrence as combining CEA testing with regular 6- to 12-monthly CT scanning. 12

The advantage of making decisions on a trend in carcinoembryonic antigen levels

The diagnostic performance of the trend in CEA level, assessed by the slope (beta-coefficient) of the linear regression line, was consistently better than interpreting the results of a single test, regardless of whether the single test was adjusted for the baseline postoperative CEA level. In particular, assessing trend detects recurrence in patients with a slowly rising CEA level below the single test threshold. The main clinical limitations of making decisions to further investigate on trend rather than on a single test result are that (1) most recurrences occur early in the follow-up period when fewer test results have been accrued and (2) it has less face validity than interpreting a single test result and will require the application of a computer algorithm at the laboratory or testing facility. The observation that optimal performance was achieved by taking account of negative as well as positive trends merits further investigation. It suggests that slow post-treatment reduction in CEA level is itself a marker of recurrence.

The choice of carcinoembryonic antigen threshold

The systematic review highlighted the very high cost in terms of false alarms of trying to improve single-test sensitivity by reducing the action threshold below the 5 µg/l commonly recommended by national guidelines. The results of the main analysis confirmed that, even at this recommended threshold, more than half of patients referred for further investigation will not have recurrence; decreasing the single-test threshold to try to achieve the sort of sensitivity feasible by making decisions on trend would cause this false-alarm rate to increase to ≥ 90%. Even using trend analysis, the number of false alarms suggests that aiming for a sensitivity of 70% – augmenting CEA testing with another triage test such as a single CT scan to detect the missed 30% of recurrences – may be the clinically preferable option. In applying the trend analysis, the main results also highlight the importance of not applying the same action threshold throughout the 5-year follow-up period. For example, to achieve 70% sensitivity at a constant specificity of around 90% requires the threshold applied to be reduced from 1.75 µg/l in year 1 to 0.3 µg/l in year 5. In applying such thresholds, the current advice to use the same laboratory technique to avoid method bias error1 needs to be endorsed (to minimise both false alarms and missed recurrence) and the CEA immunoassay techniques require improved reproducibility in the 2.5–10 µg/l concentration range.

The choice of testing interval

More recurrences were detected at the beginning than at the end of the 5-year follow-up period. Although this has been reported before, the finding remains important because the FACS cohort were so thoroughly investigated for residual disease before trial entry. It implies that, ignoring the fact that static test performance (i.e. sensitivity and specificity) also changes over time because of the characteristics of presenting recurrences, operational performance (i.e. predictive value) will decline during follow-up. This is at present reflected in the common practice of reducing CEA testing from 3-monthly intervals in years 1–2 to 6-monthly intervals in years 3–5. Our results suggest that this is not optimal. On the basis of the number of detectable recurrences alone, testing would need to be carried out more frequently in year 1 (arguably twice as frequently as in year 2). Increasing the frequency of testing in year 1 is also important if the action threshold is to be based on assessing trend (as, at present, one-third of recurrences occur before a trend is measurable). Adopting this increased testing frequency would be challenging in some health-care systems and would need careful planning (as it requires rapid turnaround of results, good communication with patients and access to a clinician who is able to discuss and act on the results quickly). Although the falling incidence of recurrence would suggest that testing frequency should be reduced to one test in year 5, this would have implications for the achievable lead time.

Who should not be followed up with carcinoembryonic antigen testing

It is not possible on the basis of any of the variables we assessed to predict in advance which patients will fail to experience a rise in CEA level with recurrence and therefore should not be followed up by blood CEA monitoring. There is no suggestion that patient age, characteristics of the primary tumour or the recurrence site predict either missed cases from non-response or false alarms. However, the likelihood of multiple false alarms is significantly higher in smokers, suggesting that CEA monitoring is not an appropriate follow-up method for patients who continue to smoke.

Quality of carcinoembryonic antigen measurement

The main strengths of the FACS trial data in this context are that CEA testing was centrally managed with high compliance with scheduled testing and all analyses were carried out in one laboratory with consistent quality control. A key strength of this analysis, compared with previous diagnostic accuracy studies of CEA for detecting recurrence of which we are aware, is that we modelled the operational performance of CEA when used prospectively in clinical practice in addition to looking retrospectively at sensitivity and specificity in relation to a series of tests.

Limitations of the reference standard

The main limitation of the data is that we do not have a reference standard at all time points. We do not know the precise time when a recurrence would have been detectable by our reference standard. Our estimate of unnecessary referrals is likely to be an underestimate as it includes only patients who did not experience recurrence throughout the whole follow-up period. However, the length of follow-up and within-trial surveillance means that we are unlikely to have missed cases of recurrence. Even if we had been able to apply the reference standard at every time point, there may be a lead time between detectability of recurrence by CEA testing and detectability of recurrence by imaging. The other limitation of the reference standard that we report here (and which has been used by almost all previous studies of which we are aware) is that it assesses the diagnostic performance of CEA testing in detecting any recurrence. We did not have a sufficient number of cases of recurrence to stratify the analysis by the treatability of recurrence, as only half of the patients in the FACS trial were allocated to CEA testing.

Work-up bias

The other important limitation is work-up bias. Patients in the FACS trial with a CEA level of > 7 µg/l above their personal baseline were referred for further investigation. This relatively high threshold means that the analyses of diagnostic accuracy at the thresholds of 2.5 µg/l and 5 µg/l are not subject to work-up bias, but the estimates of operational performance at higher thresholds reported in Appendix 1 are less robust.

Statistical precision and external validation

Although this is a relatively large diagnostic study, the limited number of cases of recurrence (n = 104) limited the statistical precision of our estimates of sensitivity. The estimates based on statistical modelling may also suffer from overfitting and merit external validation before clinical implementation.

Advantages of carcinoembryonic antigen testing as a triage test

Carcinoembryonic antigen testing has three great advantages over other forms of follow-up: it is relatively inexpensive, it can be carried out in community settings and it does not expose the patient to radiation. It can therefore be perfomed more frequently than other tests and has the potential to provide important lead time in detecting recurrence (about 3 months at the currently implemented threshold). However, our data underline that it cannot be used alone as a triage test because of its low sensitivity.

Clinical implications

Sensitivity can clearly be increased by reducing the threshold but, as stated above, the impact on workload and high false-alarm rate make that a very unattractive solution. However, it is probably time to stop defining the threshold for further investigation in terms of an absolute threshold. We have already shown that much better diagnostic performance, including better sensitivity, can be achieved by considering trend over time in a series of tests. 27 This also implies that testing should be performed more frequently during the first year of follow-up because almost half of the recurrences present during this early period.

Suggested monitoring schedule

The monitoring schedule that we suggest on the basis of our results is summarised in Box 1. The suggestion that colonoscopy and CT of the chest, abdomen and pelvis may be a possible mechanism to increase sensitivity to > 70% is based on the main analysis of the FACS trial. We suggest that evidence is now sufficient to increase the testing interval in the first year and to adopt additional tests to maintain sensitivity. We also think that the evidence on false alarms from the Cochrane analysis is sufficient to advise raising the threshold for interpreting a single test from 5 to 10 µg/l and to advise smokers to quit or not be monitored for CEA level. However, we suggest strongly that the implementation of interpreting CEA levels after 3 months on the basis of trend rather than individual level is staged so that optimal cut-off points can be refined in one or two pilot sites before wider roll-out.

Other research implications

The systematic review drew attention to the poor quality of the majority of diagnostic studies on CEA monitoring. Moreover, virtually all studies assessed CEA levels as a single diagnostic test, ignoring the fact that it is used as a monitoring test performed repeatedly over time. Even the Cochrane diagnostic accuracy review methodology focuses on single test results rather than monitoring performance. This issue needs to be addressed, not just in relation to CEA testing.

As mentioned above, the most pressing further research is the need to conduct implementation studies to refine the suggested cut-off points for monitoring CEA trend. In conducting this work, it would be preferable to use a reference standard based on recurrence treatable with curative intent rather than simply any recurrence.