Methods for the evaluation of biomarkers in patients with kidney and liver diseases: multicentre research programme including ELUCIDATE RCT

Article history

The research reported in this issue of the journal was funded by PGfAR as project number RP-PG-0707-10101. The contractual start date was in January 2009. The final report began editorial review in February 2016 and was accepted for publication in June 2017. As the funder, the PGfAR programme agreed the research questions and study designs in advance with the investigators. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The PGfAR editors and production house have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the final report document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

Peter J Selby has an international patent (application no. PCT/GB2014/050768; biomarker; University of Leeds) issued. Rosamonde E Banks reports grants from various non-commercial funding sources during the conduct of the study; in addition, she has a international patent (application no. PCT/GB2014/050768; biomarker; University of Leeds) issued. Jenny Hewison is a panel member of the National Institute for Health Research (NIHR) Clinical Trials Unit Standing Advisory Committee. Grant D Stewart has a patent (1408091.5) pending to the Medical Research Council (MRC) Human Genetics Unit (HGU). James Neuberger is a consultant to Astellas Pharma, Inc., and has received speaker support funding from Astellas and Novartis UK. Claire Hulme has received grants from the NIHR during the conduct of the study and is a panel member of the Health Technology Assessment (HTA) Commissioning Board. Andrew Lewington has received personal fees from Fresenius Medical Care, Baxter Healthcare Ltd, AM-Pharma, BioPorto Diagnostics A/S and GE Healthcare, outside the submitted work; in addition, he has a patent (ACY-1 biomarker) pending. Naveen S Vasudev has received personal fees from Bristol-Myers Squibb and GlaxoSmithKline plc, outside the submitted work. William Rosenberg was an inventor of the enhanced liver fibrosis (ELF) test when he was an employee of the University of Southampton. His rights were transferred to Siemens Healthcare Diagnostics Ltd by the University of Southampton. He does not receive any payment in relation to sales of the test by the manufacturer Siemens Helathcare Diagnostics. He has received grant support and speaker fees from Siemens Healthcare Diagnostics and is a director of iQur Ltd (Southampton, UK), a company that provides ELF testing. In the context of this NIHR-funded study, all ELF testing provided by iQur Ltd was performed on a not-for-profit, cost-recovery basis. William Rosenberg is married to Julie Parkes, who is a co-investgator on the programme. Julie Parkes reports receiving support from the Speakers’ Bureau for Siemens Healthcare Diagnostics, outside the submitted work, and is married to William Rosenberg (co-applicant). Jonathan J Deeks is a panel member of the HTA Commissioning Board.

Copyright statement

© Queen’s Printer and Controller of HMSO 2018. This work was produced by Selby et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2018 Queen’s Printer and Controller of HMSO

Background

Protein biomarkers in body fluids are now being regularly identified using new techniques and are associated with the presence and activity of diseases and treatment benefits/toxicities. They are accessible, measurable in real time and inexpensive to test for. However, their potential benefit to the NHS is not being realised because of the absence of a defined pathway linking biomarker research to health services research. This programme aimed to establish a process for the stringent evaluation of promising biomarkers, encompassing methodological developments, clinical evaluation and a randomised controlled trial (RCT) to enable assessment of their impact on clinical outcomes, the process of care, resource use and service configurations.

Our specific objectives were to:

• evaluate and develop methodology for the optimal use of biomarkers for disease monitoring and to optimise benefits, for patients and the NHS

• establish a sample and clinical data bank together with a robust system for the evaluation of promising markers to facilitate their rapid assessment prior to large-scale trialling in the NHS

• conduct a RCT of an established panel of biomarkers of potential value in chronic liver disease (CLD) to diagnose cirrhosis at an early stage when interventions may reduce dangerous complications and to determine patient and NHS benefits.

Biomarkers have major potential benefits for patients and the NHS, particularly in contributing to ‘personalised’ and/or ‘stratified’ medicine and improved safety. They may supplement or replace invasive procedures or imaging tests for:

• accurate and early diagnosis

• measurement of the activity and extent of disease

• indication of prognosis

• selection and prediction of optimal treatments

• monitoring for treatment response/toxicity or disease progression.

In addition, biomarker information may inform patient counselling on lifestyle issues (e.g. alcohol avoidance or diet) and motivate patients towards healthy alternatives. Overall, better biomarkers should lead to improvements in outcomes and more efficient, cost-effective and evidence-based use of NHS resources.

With ongoing technological developments, particularly in proteomics, the rate of identification of potential new biomarkers may be expected to increase. Various stages in the ‘biomarker pipeline’ have been defined, but the translational work needed to progress through these – involving technology transfer, methodological considerations and aspirations of different stakeholders – presents challenges. The lack of a clear evaluative infrastructure means that the route from stringent evaluation and/or validation to clinical implementation and then to evaluation of impact on outcomes and health care is not yet established, representing a major threat to achieving full patient benefit. 1–9 As an indication, the number of new protein markers approved by the US Food and Drug Administration (FDA) has gradually declined, with only 10 approved in the period from 1994 to 2002, only two of which were approved in 1998–2002. 10 Encouragingly, the need for national strategies for the rapid evaluation and introduction of new biomarker tests is now better appreciated, for example by the National Institutes of Health (NIH) in the USA11 and by the Royal College of Pathologists in the UK. 12

A framework is, therefore, required for the pipeline to justify and guide the introduction of biomarkers, including specification and establishment of the infrastructure to acquire such evidence, appropriate assessment of test results, identification of whether or not RCTs are required and the means of deciding when and how biomarker development and introduction should be accelerated.

Selected diseases

Chronic liver disease and renal diseases provided ideal subjects with which to work up such a framework. Liver and renal diseases generate huge and increasing burdens on patients and the NHS. The care of patients would be transformed with improved outcomes, more appropriate use of complex and expensive therapies and avoidance of expensive and invasive investigations if biomarkers of real health-care value could be found.

Protein biomarkers

Protein biomarkers in body fluids have substantial potential to improve the quality of health care. There are active pipelines identifying them in both the commercial and the non-commercial sectors, but robust methodological approaches and well-organised rapid clinical and health service evaluation is still limited.

In a clinical setting, the value of a protein biomarker depends on test performance and its relation to health improvements. Methods of reporting, and hence judging, test performance are well developed for biomarkers and other kinds of measures when these are used in a prognostic role and in a diagnostic role. 32,33 The methodology for evaluating biomarker test performance when used in an individual patient monitoring role is, however, poorly developed.

After the initial discovery and preliminary evaluation of a protein biomarker, robust evaluation of its clinical characteristics [sensitivity, specificity, receiver operating characteristic (ROC), etc.] is often performed slowly and in limited sample numbers. We set out to establish a robust system using samples and clinical data in adequate numbers to rapidly evaluate markers that may be useful in the NHS and select those that justify formal evaluation.

Chronic liver disease can frequently progress to cirrhosis and to life-threatening complications. Early diagnosis of cirrhosis with appropriate management can reduce the incidence of these complications, and a panel of protein biomarkers that directly evaluate fibrosis has excellent clinical characteristics [ADVIA Centaur® Enhanced Liver Fibrosis (ELF) test; Siemens Healthcare Diagnostics Ltd, Camberley, UK]. We proposed and have conducted a RCT to establish whether or not its use can substantially improve patient outcomes and health-care provision. Previous work on the ELF test shows that it has a proven clinical association with cirrhosis and its complications. This provides justification for its evaluation to see whether or not it can be used to monitor patients and allow the diagnosis of cirrhosis at a time when interventions will reduce the morbidity and mortality associated with these complications. 34–37

Monitoring studies

Considerable research has been carried out into the use of biomarkers for prognosis (including prediction of response to treatment). In contrast, there is a relatively small amount of literature on the use of biomarkers for monitoring. 38,39 Methodological work has not yet been conducted on the design and interpretation of studies with repeated measurements of biomarkers, and we used value of information analysis in this context.

Monitoring may be undertaken for various purposes. In chronic diseases, it may assess whether or not interventions are keeping the disease and symptoms under control; assess the rate of progression of disease (e.g. the ELF test in our RCT); detect recurrence of disease; or evaluate the efficacy of treatments. Monitoring may also allow adverse effects to be avoided. A strategy is required for the frequency of testing and rules for clinical actions (including retesting). Although each test result can be judged on its own, there is the potential also to learn from the change since the previous test or the rate of change over time or to devise and calibrate a model of change over time. Rules may be devised using suitable cut-off points, changes between values or values of fitted model parameters or confidence intervals (CIs) thereof. In addition, it will be important to understand natural variability because of measurement error, prognostic factor-related trends (e.g. in age) and the risks of incorrect decisions.

Few biomarkers have been studied in this context. Examples in cancer include cancer antigen 125 (CA-125) for monitoring patients with ovarian cancer for recurrence and prostate-specific antigen (PSA) for monitoring men at risk of prostate cancer and for monitoring prostate cancer for recurrence. 40,41 Serum creatinine measurements are used to monitor patients with chronic kidney disease (CKD) and transplants. An important question for monitoring by repeated measurement is whether prognosis relates to the actual marker level or the change over time or more complex models. 42

Two different questions need to be considered, relating first to the interpretation of specific marker values and sequences of such values, and second to the implementation of such information, for example to determine how frequently measurements should be taken. There is a need for large, high-quality data sets with repeated measurements to inform the development of decision rules for monitoring.

Choice of the Enhanced Liver Fibrosis test for the ELUCIDATE randomised controlled trial

Within our range of liver and renal diseases, we identified the ELF test as having ideal characteristics for our RCT because:

• the ELF test has convincing clinical evaluation data showing an association with cirrhosis (e.g. ROC of > 0.8) but the clinical and health service benefits from monitoring CLD for cirrhosis are untested

• the burden of CLD and cirrhosis on patients and the NHS is huge and increasing

• early diagnosis of cirrhosis allows effective surveillance and the use of interventions to improve clinical outcomes and care

• a simple blood test could radically improve and provide cost-effective care for CLD patients

• this model will provide an excellent prototype for health service biomarker research and vital data sets for our methodologists.

In the vast majority of cases, liver fibrosis is asymptomatic and cirrhosis develops insidiously with non-specific symptoms, so that opportunities for disease modification or cure are missed. Standard biochemical tests of liver function are not specific or sensitive. Liver biopsy is hazardous and inaccurate and subject to sampling error and variation in interpretation. 43–45 Imaging has a major role in the detection and assessment of liver fibrosis. However, all imaging modalities, including ultrasound, elastography and cross-sectional imaging with X-rays or magnetic resonance, require access to expensive technology and skilled operators. 46,47

With cirrhotic diseases of the liver, we are in the relatively fortunate position of having a number of treatments (such as beta-blocker therapy or surgery for low-volume HCC) that are known to be effective at reducing complications, if the cirrhotic condition is detected early enough. The ELF test seeks to identify a ‘pool’ of patients with a slowly progressing disorder who can be treated prophylactically and in whom the incidence of severe complications can be reduced.

Evidence shows that the early detection of varices and treatment with prophylactic use of beta-blockers to reduce portal hypertension or band ligation reduces morbidity and increases survival, and respected guidelines recommend surveillance because of its benefits and health economic justification. 48–52 Similarly, the early detection of ascites and treatment has been shown to reduce the morbidity associated with bacterial peritonitis from 17% to 2%. 53 The case for surveillance and early detection of HCC is more contentious, with some RCTs showing evidence of benefit and others showing none. International guidelines now advocate surveillance. 54–56 Retrospective analyses have identified criteria, essentially the presence of small tumours, that are associated with better outcomes of HCC resection and liver transplantation, but many patients are diagnosed after the growth of their tumours has ruled them out for curative resection or transplantation. 57,58

Studies and systematic reviews have demonstrated that single direct markers are less accurate than panels of markers for the detection of liver fibrosis. 59,60 One such panel of direct markers is the ELF test, the only CE (Conformité Européene)-marked [European Union (EU) regulatory approval] test for liver fibrosis measuring constituents of liver matrix [hyaluronic acid (HA) and procollagen III amino-terminal peptide (PIIINP)] and a molecule critical to the regulation of matrix remodelling [tissue inhibitor of metalloproteinase 1 (TIMP-1)] using sensitive automated enzyme-linked immunosorbent assays (ELISAs) designed and manufactured specifically for this purpose. 61 The three individual biomarkers were selected as being optimal from among 20 candidates. The results of the individual assays are combined in an algorithm that was derived and validated in > 1000 cases of liver fibrosis to generate a score that correlates with the severity of liver fibrosis on liver biopsy. ELF values have been shown to be highly predictive of clinical outcomes, including variceal bleeding, ascites, HCC and mortality. Subsequent validation studies in hepatitis C, fatty liver disease, human immunodeficiency virus (HIV)–hepatitis C virus (HCV) co-infection and primary biliary cirrhosis have confirmed the performance of the test. 34–37 Although performance is best in the detection of advanced fibrosis and cirrhosis, the test can also detect mild and moderate degrees of fibrosis accurately, with area under the curve (AUC) ROCs of 0.83 for Ishak fibrosis stages of 0–3 compared with 4–6 and 0.86 for Ishak fibrosis stages of 0–4 compared with 5–6. The ELF test is excellent at detecting advanced fibrosis/cirrhosis in a range of CLDs and is, thus, well suited for use in screening populations at risk for cirrhosis. The ELF test has been developed by Siemens Healthcare Diagnostics Ltd (formerly Bayer Healthcare) in conjunction with the University of Southampton and iQur Ltd (Southampton, UK).

The overall shape of this programme of work is provided in Figure 1.

FIGURE 1.

The programme of work. NEQAS, National External Quality Assessment Service.

Introduction

Monitoring involves the scheduled, repeated use of a test or tests in an individual over time to make decisions about the management of a disease or condition. It is a central activity in the management of patients, making up a considerable part of the clinical workload and associated cost. 62 In contrast, the volume of published literature on the evaluation and use of tests for monitoring purposes is relatively small.

Mant66 and others62,67 have provided a framework for developing and evaluating a monitoring strategy. The framework has four main steps: (1) deciding whether or not to monitor, (2) choosing a test, (3) specifying and assessing the monitoring strategy to be used and (4) an implementation phase. Underlying this is the key concept that the ‘signal’ from the test, reflecting the status of the underlying condition, should be greater than the surrounding ‘noise’, or measurement variability, that may affect test interpretation. 66,67 If the ‘noise’ around a test measurement is too high in relation to the signal, one’s certainty in a given test result will be considerably reduced.

The repeated measurement of PSA in men who have undergone primary treatment of prostate cancer is an apparently successful example of a rule-based monitoring strategy. The behaviour of PSA following radical treatment varies, but, in general, recurrence of disease is associated with the presence of PSA (following radical prostatectomy) or some rise in [following radical radiotherapy (RTX)] PSA level. 70 When a predefined level of PSA is reached, biochemical failure is said to have occurred. The usefulness of PSA as a monitoring test is based on the assumption that biochemical failure predates clinical failure within some clinically meaningful time frame. The decision to initiate treatment for recurrence, however, will depend on multiple factors (M-Factors) rather than on a single PSA value alone. 71

We undertook a review of clinical guidelines on monitoring with PSA testing for the detection of recurrent prostate cancer to determine the extent to which they take into account key factors that should inform rule-based strategies for monitoring. Our particular focus was on the degree of consistency between guidelines, the explicit consideration of factors important for specifying a monitoring strategy and the use of supporting evidence to justify any recommendations.

Methods

Results

Guidelines (n = 7) or best practice statements (n = 2) from nine organisations were identified. 75–83 Four were North American in origin,77,79,82,83 four were from Europe75,78,80,81 and one was from Australia. 76 Eight of the guidelines75–79,81–83 scored poorly on the framework criteria, with scores between 9 and 16 out of a possible 28 points (Figure 2). The National Institute for Health and Care Excellence (NICE) guideline80 scored considerably higher, with 22 points. The highest scoring item overall was the use of systematic searches, which was reported in most guidelines, even if it was often not described in any detail. Methods for recommendation formulation were described in only three guidelines. 78–80 Only one guideline80 fully considered relevant issues for monitoring tests; this was the only guideline to consistently provide clear links between its recommendations and the underlying evidence base. It also reported its methods in more detail than most of the other guidelines in the sample.

FIGURE 2.

Rigour of guideline development. Adapted with permission from Dinnes et al. 69 Aus CN, Australian Cancer Network; AUA, American Urological Association; DUA, Dutch Urological Association; EAU, European Association of Urology; NCCN, National Comprehensive Cancer Network; NCI PDQ, National Cancer Institute – Physician Data Query; UK PCWG, UK Prostate Cancer Working Group.

Table 2 shows the lack of consistency in guideline recommendations regarding the frequency of follow-up assessments and thresholds; there does not appear to be any clear pattern in recommendations over time.

Eight75–81,83 of the nine guidelines acknowledged that PSA levels may be affected by technical or biological variability but in most cases this was presented in the introductory sections of the guidelines, with only one guideline78 tempering its recommendations with reference to the fact that a single PSA measurement may be unreliable (recommending retesting within 2 months). Three guidelines79,80,83 acknowledged the potential impact from technical variation, recommending that the same assay be used at each measurement. Four guidelines75,78,80,82 made some attempt to justify the interval between tests and three75,79,80 discussed relevant issues affecting the choice of threshold. A further three76,77,79 stated that it was not possible to provide a recommendation on the most appropriate biochemical failure definition. Only three77,79,80 of the nine guidelines commented on the difficulty of using PSA as a monitoring tool because of the uncertainties in its behaviour following radical treatment, with two79,80 clearly recognising that not all men with biochemical failure go on to experience clinical failure, such that evidence of the former alone may not be sufficient to alter treatment.

Many recommendations on frequency or threshold were made with no or few supporting citations (see Table 1). Only one guideline78 cited a primary study in support of its recommended monitoring intervals and only four78,80–82 of the nine indicated the level of evidence supporting their recommendations. The levels of evidence suggested ranged from ‘Guideline Development Group consensus only’ to ‘well-conducted clinical studies’ (see Table 1), suggesting that different guideline groups had varying views on the quality of the evidence available.

Despite the general lack of citations in individual guideline documents, a wide range of papers were cited across the guidelines. In total, 49 papers were cited71,84–131 (Tables 3–5); 31% (15/49) were reviews or consensus statements and the remainder were primary studies, almost exclusively retrospective in nature. Of the primary studies, we judged half to have studied the natural history of PSA following treatment and one-quarter to have evaluated the effect of different biochemical failure definitions on clinical outcomes. Only two primary studies examining measurement variability were cited.

Most studies were cited by only one or two of the guidelines, but a handful of studies were cited three or more times (see Table 5). Two consensus statements84,85 were among the most frequently cited studies, as was a review of biochemical failure definitions. 71 The four primary studies had among the largest sample sizes of all of the cited primary studies: three86–88 evaluated the use of different biochemical failure definitions and one89 studied the natural history of disease progression in men with raised PSA.

The findings are summarised in Table 6.

Discussion

We found considerable inconsistency in the recommendations in guidelines for the use of PSA as a monitoring test, even when they were published within a few years of each other. Factors considered to be important when specifying a monitoring strategy were given limited attention and were not well supported with reference to primary literature.

Recommendations on when to test and what action to take following a given test result were very much considered in isolation from each other. ‘When to test’ appeared to be almost exclusively determined by standard follow-up schedules rather than having any scientific basis. Although most guidelines acknowledged the potential presence of measurement variability, they did not attempt to account for its potential effect on test interpretation. A systematic review of biological variation in PSA levels has found a mean biological variability of 20%. 132 Using reference change value (RCV) methodology it concluded that, to be 95% sure that a change in total PSA level is not the result of random variation, the change needs to be around 50% of the previous measurement. 132 This review was not cited by any of the eight guidelines subsequently published.

Recommendations on when to take action were based on consensus statements or retrospective case series, with little attention paid to variations in the definition of the threshold, the definition of clinical failure and the frequency and length of follow-up between studies, all of which can affect the accuracy of any given cut-off point. Sensitivity and specificity are also known to be affected by differences in patient case mix between studies. 133 This was not acknowledged by any of the identified guidelines. However, a 2005 review of PSA measurement for monitoring prostate cancer90 found it impossible to recommend any single definition of biochemical failure following either surgery or radiotherapy for the reasons listed above. This review was cited by only one of the nine guidelines,80 possibly because it was not a full systematic review. Given the lack of description of inclusion criteria used in the guidelines, it is difficult to reconcile why an individual study or review was included or not.

Reviews of guidelines in other areas have shown similar findings regarding the presentation of evidence for recommended monitoring schedules. 134,135 Reviews of treatment and diagnostic guidelines have identified a similar inconsistency in recommendations between guidelines and variation in evidence cited, with some referring to a substantial body of evidence and others presenting very little evidence. 136–140

A number of factors are likely to contribute to these findings. In the first instance, although this area is beginning to receive more attention, there is a lack of high-quality evidence, and indeed lack of clear methodological guidance, on what to consider when establishing monitoring strategies. 38 It is, therefore, perhaps not surprising that relevant evidence has not been used to inform guidelines.

Second, the various pieces of information needed to inform a monitoring strategy are not usually available from a single study. Ideally, one or more monitoring strategies should be evaluated in a RCT or some form of prospective comparative study. When there is high-quality evidence, greater consensus between guideline recommendations and stronger guideline recommendations have been found. 136 Randomised trials of monitoring, however, have their own challenges and are consequently relatively rare. Instead, evidence has to be gathered from various sources. Although the diversity of evidence needed to inform coherent monitoring strategies makes the identification of relevant pieces of evidence a challenge for guideline developers and likely adds to the inconsistency in recommendations between guidelines, guideline developers have a responsibility to highlight recommendations for which there is a lack of evidence or the evidence is inconsistent.

Efforts to improve the evidence base for monitoring are ongoing. For example, a Bayesian hierarchical change-point model has been used to simulate multiple post-radiotherapy PSA series from primary data; the sensitivity and specificity of different definitions of biochemical failure were then compared, allowing characteristics that might affect accuracy to be controlled for. 141 More pertinently, statistical models using estimates of mean change and variability in a measurement over time to suggest optimal monitoring intervals are being developed. A review142 of four case studies143–146 found that for each topic the results suggested overfrequent monitoring. There is clear potential for the extension of this work to monitoring in other settings.

Finally, general failings in the guideline development process are likely to contribute significantly to the variations between published guidelines. In a review of hypertension guidelines, Campbell et al. 138 found a lack of methodological rigour in the guideline development process. In our sample, NICE80 and the Australian Cancer Network76 were the only organisations to cite a published handbook on guideline development,147 which may explain their higher ratings on the evaluation instrument; those clearly based on expert consensus tended to score considerably lower. 83 Others suggest that the greater the involvement of clinical experts in the development process of the guideline, the less the recommendations reflect the research evidence. 137 It is likely that, in the absence of clear methodologies for assessing monitoring strategies, greater involvement of methodologists on guideline panels would be beneficial.

Strengths and limitations

Our research has some limitations. Our literature search was limited to one major medical database, supplemented with searches of more specialist resources, and records were limited to English-language studies only. We believe, however, that we have identified key guidelines that provide a good representation of the methodologies in use by well-known agencies. Although other guidelines may be available, they are unlikely to have used alternative methods or to have reported on evidence that the included guidelines have missed.

Second, our use of the original AGREE instrument may be criticised given that it was published in 2003; however, at the time that the framework was chosen, the update to the original instrument148 and other potentially useful frameworks were not yet available. 146,149,150 Nevertheless, our approach to assessing the development and content of the guidelines was systematic and provides a reasonable means of comparison between guidelines.

We were not able to comment fully on the state of the underlying literature cited in support of the monitoring schedules as we did not retrieve copies of all of the cited primary studies. Furthermore, our ‘generous’ approach to associating citations to recommendations may have inadvertently led to citations being incorrectly associated with recommendations. This may have led to some bias in favour of the guidelines. However, this could be avoided only by carrying out a full review of all of the evidence cited or by direct contact with the guideline authors to determine which aspect of the recommendations were supported by the citations given, both of which were outside the scope of this review.

Finally, our use of only one case study may limit the generalisability of our results to other topic areas. However, we have no reason to believe that the picture would be any better or any worse for other fields and, indeed, Moschetti et al. 135 found similar results for monitoring in cardiovascular disease.

Our systematic approach to assessing the development and content of the guidelines provides a valuable insight into how strategies for monitoring are developed and reported and we were able to present a general picture of the types of evidence that have been cited. The true picture may be even worse given our attempt to attribute citations to recommendations whenever possible.

Conclusions

Our findings highlight the lack of a scientific or systematic approach to the development of monitoring schedules for the use of PSA testing, as reported in clinical guidelines. This is the result of both inadequacies in the evidence base and inappropriate use of the available evidence, resulting in considerable inconsistencies between guidelines.

Guideline developers should be encouraged to adopt systematic approaches to guideline development, such as those developed in Australia149 and the USA,150 and should take care to explicitly consider each element of a recommended monitoring schedule (interval, threshold and action to be taken on crossing that threshold) and the standard of its evidence base.

Introduction

Clinical consultations between patient and clinician usually involve the use of tests, often starting with a general clinical assessment and physical examination but also including the application of specific tests, from tools assessing functional or psychological status, to blood or urine tests, physiological measurements of, for example, blood pressure, imaging tests or more invasive assessments such as a colonoscopy or biopsy. Testing can inform a diagnosis or can be used for monitoring whereby a test, or set of tests, is applied repeatedly over time to assist in the management of a disease or condition. Our particular interest is in monitoring individuals with (or at risk of) a disease or condition that is likely to progress or recur at some time in the future. This is distinct from monitoring in a treatment titration context, in which the aim is to keep a marker within predefined limits until treatment can be discontinued or an alternative treatment is required. Although monitoring for disease progression or recurrence can serve many purposes, including providing reassurance to patients or clinicians, it is usually undertaken to allow timely decisions to be made regarding patient management. Management decisions include the initiation of treatment to prevent some clinical outcome from occurring (e.g. variceal bleeding subsequent to cirrhosis of the liver or infertility as a result of Turner syndrome), delay a clinical event [e.g. progression to acquired immune deficiency syndrome (AIDS) in HIV infection] or otherwise improve outcome (e.g. through earlier treatment of cancer recurrence); additionally, the goal may be to avoid or delay treatment in those who may not need it (e.g. surveillance of mild hip dysplasia in infants).

Whatever the goal, monitoring is a central activity for patient and disease management and, just as for therapeutic interventions and for tests used in a diagnostic context, it is important to identify its impact on patient outcomes: ‘the primary purpose of using . . . tests should . . . be to prevent premature death and suffering and restore functional health’. 151 Given the advantages of the RCT design for the evaluation of therapeutic interventions, it is tempting to assume that the same approach must be the gold standard for the evaluation of all monitoring strategies.

Trials of monitoring regimes present considerable challenges, however. As for diagnostic tests, trials of monitoring evaluate a particular strategy, with tests applied at specific intervals, defined thresholds for changing patient management and prescription of effective interventions, all of which should be specified in advance and ideally supported by previous research. The complexity of such strategies, and in particular the serial nature of testing, and consequent potential for ‘interactions between tests, repeated tests, test results and the decisions based on these results’ may necessitate unfeasibly large sample sizes to detect an effect on important patient outcomes. 152 Furthermore, even with careful planning it may be difficult to capture in a RCT the wider patient impact of testing, whether it is used for diagnostic, screening or monitoring purposes. Ferrante di Ruffano et al. 68 have outlined a range of effects from testing in a diagnostic context, including emotional, cognitive and behavioural effects, which also have applications in monitoring.

We conducted a methodological review of RCTs of monitoring to gain some insight into how successfully the design has been used to identify patient benefit from monitoring.

Methods

Results

The CENTRAL search retrieved 4697 potentially eligible records (Figure 3), of which 119 titles were selected for further evaluation, along with nine trials identified from the Health Technology Assessment database and the Trials journal. Following full-text review, 49 trials published in 58 publications were selected for inclusion. Twenty trials identified from the ClinicalTrials.gov database were selected and the trial principal investigators were contacted. Documents related to 12 of these were successfully retrieved, of which seven met the inclusion criteria. Reference list screening identified a further two eligible trials. Of the 58 included trials, five were reported in two publications153,154 and 19 previous or related publications were also identified (for a total of 74 papers153–225). Figure 4 shows that there was a general upwards trend in the number of trials published per year.

FIGURE 3.

Flow diagram of the trial selection process. HTA, Health Technology Assessment.

FIGURE 4.

Number of trials published over time. The seven trials identified from ClinicalTrials.gov were published in 2007 (n = 1), 2009 (n = 1), 2010 (n = 3) and 2011 (n = 2).

Description of monitoring strategies

A total of 139 tests were applied in the control arms of 55 RCTs, excluding those with no formal surveillance or those in which all patients were treated (Table 8). After clinical examination, imaging tests were the most commonly used tests (39%). The 55 ‘new’ tests applied in the experimental arms included biochemical (35%), imaging (34%) and physiological (16%) tests and implanted devices (9%).

TABLE 8 - Details of monitoring schemes

Excludes one trial in which the control arm underwent immediate treatment and two trials in which no formal surveillance was standard practice.

Out of a total of 68 experimental arms, we excluded 13 arms in which no new test was introduced (test frequency varied), nine ‘treat all on recruitment’ arms (the remaining three of the treatment trials were included as randomisation was conditional on crossing a predefined threshold) and one study evaluating no formal surveillance. Of the 45 remaining experimental arms, eight added or replaced more than one test; hence, the total number of tests is 55.

Excludes 53 ‘clinical assessments’.

Includes one trial in which the experimental arm involved ‘treat all on recruitment’ but the confirmatory test was performed first.

Includes 12 trials in which the experimental arms received immediate treatment.

Details	Trial arm
	Control (N = 139 testsa)		Experimental (N = 55 testsb)
	n	%	n	%
Type of test
Biochemical	15	11	19	35
Clinical	53	38	0	0
Cytological	0	0	2	4
Histological	1	1	1	2
Imaging	46	33	16	29
Imaging (invasive)	8	6	3	5
Implanted device	4	3	5	9
Physiological	12	9	9	16
Type of threshold used (excluding clinical assessment)
Not reported	41	48	18	33
Simple threshold	30	35	21	38
Change from previous measurement	1	1	4	7
Algorithm	6	7	6	11
More than one threshold	8	9	6	11
	Trial arm
	Control (N = 86 testsc)		Experimental (N = 55 tests)
	n	%	n	%
Repeat measure taken to confirm abnormal result (excluding clinical assessments)
All abnormal	8	9	4	7
Indeterminate only	7	8	4	7
Not repeated	0	0	1	2
Not reported	71	83	46	84
Change in patient management following positive monitoring test (N = 58 trials)
Confirmatory testing (non-invasive)	10d	17	10	17
Confirmatory testing (invasive)	8	14	14	24
Treatment	26	45	26e	45
Treatment or further investigation	2	3	1	2
More intensive surveillance + treatment option	1	2	2	3
Not described	11	19	5	9

Figure 5 shows that, although the frequency of application of the tests was well reported, the method of application of the tests was provided for only two-thirds of the experimental arms and fewer than half of the control arms. The test thresholds, describing when a change is patient management is indicated, were reported for 79% and 51% of the experimental and control arms, respectively. A simple threshold approach (i.e. in which the patient crosses a predefined threshold on a single test measurement) was used to judge an abnormal test result in 35% of control tests and 38% of experimental tests (see Table 8). Few trials reported using test measurements over time to define an abnormal result, although the percentages were higher for the experimental tests: the change from the previous measurement was reported for 1% of control tests and 7% of experimental tests and a more complex algorithm taking account of more than one test result was reported for 7% and 11% of the control and experimental tests, respectively. There was also limited reporting of repeated testing to confirm abnormal or indeterminate test results [reported for < 20% of tests in both arms (see Table 8)].

FIGURE 5.

Adequate description of monitoring strategy elements.

The recommended change in patient management following a positive monitoring test was the same in the experimental and control groups for 69% of trials. The change in management prescribed was usually treatment (45% of both the control and the experimental arms). Some form of confirmatory testing was indicated in 31% and 41% of the control and experimental arms, respectively (see Table 8), with many studies also recommending the treatment to be used following a positive confirmatory test. Details of the actual intervention given and method of application of the intervention were provided for 74% and 28% of the experimental arms, respectively, compared with only 62% and 22% of the control arms, respectively (see Figure 5).

More evidence was cited to support the various elements of the monitoring strategies for the experimental arms than for the control arms (Figure 6); however, many of the previous studies were cited in the introduction or discussion sections of the papers rather than being cited to explicitly support particular test frequencies, test thresholds or interventions reported in the methods sections.

FIGURE 6.

Citation of evidence to support features of the monitoring strategies. NA, not applicable; Rx, treated.

What was the aim of the monitoring evaluation and what change in patient care was implemented?

In 78% (45/58) of the trials, usual care in the control arm was based on some form of clinical assessment (Table 9), often with a focus on one main test (12/58) or more commonly a battery of other tests (21/58). Across the 68 experimental arms, the most common change in monitoring was the addition of a new test to an existing monitoring strategy (n = 29) or as triage to a more invasive test (n = 3). In 12 experimental arms, there was no change in the tests used, but test frequency was increased (n = 5) or decreased (n = 7). In 53% of the experimental arms (36/68), the frequency of outpatient visits undertaken by patients was the same as for the control arm.

TABLE 9 - What change in patient care was evaluated?

58 control arms and 68 experimental arms because six trials were multiarm trials.

Includes one trial in which standard care was treatment on recruitment with subsequent regular follow-up; the experimental arm underwent treatment only if the surveillance tests continued to show abnormalities at later follow-up points.

Test frequency was also reduced in three arms.

Test frequency was increased in five experimental arms, reduced in seven and stayed the same in one (this was a trial of treatment vs. surveillance, but following treatment patients in the control arm underwent the same surveillance as those in the experimental arm158).

Includes three trials in which all patients were monitored with a new test following trial recruitment and were randomised only on crossing a given threshold. Participants in the control arms of these trials continued to undergo standard monitoring following randomisation and were treated according to usual criteria.

Control arms (n = 58a)	Total, n	% of total (n = 8)	Experimental arms (n = 68a)
			Change in testing strategy						Effect on frequency of office visits
			Addition of test	As triage to existing test	Replacement test	Same test(s) (change in test frequency)	Fewer tests (no monitoring)	Immediate treatment	More	Same	Less	Not reported
No formal monitoring	2	3	4	0	0	0	0	0	4	0	0	0
Monitoring focused on single test	11	19	2	2	5	5	0	0	3	6	5	0
Clinical assessment	12	21	10	0	0	0	0	2	1	8	1	2
Clinical assessment plus one main test	12	21	4	0	0	4	0	5	0	4	4	5
Clinical assessment plus multiple testsb	21	36	9	1	5	4	1	5	0	18	2	5
Total	58		29	3	10c	13d	1	12e	8	36	12	12
% of experimental arms (n = 68)			43	4	15	19	1	18	12	53	18	3

For most trials, the change in patient management was intended to improve patient outcomes (75%; 51/68), either through earlier initiation of treatment or better selection of patients requiring treatment (Figure 7). This was generally achieved through the addition of a new test to an existing monitoring strategy (55%; 28/51), although, in 12 (24%) studies, triallists evaluated earlier treatment by enrolling patients who had not yet reached the standard (implicit or explicit) threshold for treatment and randomising them to either an immediate treatment option or continued surveillance. In four studies,155–157,226 there was an explicit underlying evaluation of a monitoring strategy as patients in both groups were monitored with a specific test following recruitment and randomised only when their test result crossed a predefined threshold for intervention.

FIGURE 7.

Aim of monitoring evaluation (N = 68 experimental arms). Rx, treated.

For the remainder of the studies, the goal was to maintain the same patient outcomes (25%; 17/68) but to reduce the amount of testing undertaken, either by reducing the number of tests (n = 9) or reducing the amount of invasive testing carried out (n = 7), or introduce surveillance to avoid treatment (n = 1) (see Figure 7). This was achieved by replacing an existing test (n = 4), reducing the frequency of testing (n = 8), adding a new triage test (n = 3) to select patients for a more invasive test or reducing the number of tests performed at each visit (n = 1). The remaining study was a non-inferiority trial that aimed to demonstrate that additional testing did not improve the survival of patients with colorectal cancer. 211

In terms of study validity, sequence generation and allocation concealment were judged to be adequate in 48% and 45% of the studies, respectively (Figure 8). Blinding of study participants, study personnel and outcome assessment was rarely implemented; in the majority of the studies, blinding was not described and a judgement had to be made as to the likelihood of blinding being present or not. Uniform and unbiased outcome assessment (in which the primary outcome was assessed in the same way in both arms and was not determined by the monitoring test under evaluation) was carried out in 83% of studies. In eight trials, however, the presence of the primary outcome was clearly measured by the monitoring test, so that the outcome was defined differently between arms.

FIGURE 8.

Trial validity.

Primary outcomes and results

Approximately half of the trials chose to evaluate patient-related primary outcomes (Table 10), of which one-third (n = 9/31) aimed to assess the impact of monitoring on mortality and over half aimed to detect either new (n = 9/31) or recurrent (n = 7/31) disease. Process outcomes evaluated were primarily related to the numbers of patients treated between arms (n = 8/13) or the time to treatment (n = 4/13).

TABLE 10 - Description of the primary outcomes used and the main result

Full trial reports only.

Topic	n	%
Type of primary outcome measures
Patient	31	53
Process	13	22
Composite	10	17
Unclear/not described	4	7
Patient outcomes used
Clinical	4	13
Function	1	3
New disease rate	9	29
Recurrent disease rate	7	23
Mortality	9	29
Psychological morbidity	1	3
Process outcomes used
Diagnostic yield	1	8
Therapeutic yield	8	62
Timing of care	4	31
Was the effect on the primary outcome statistically significant? (n = 34a)
Yes	12	35
No	20	59
Not reported	2	6

Around one-third of the full trial reports (12/34) reported statistically significant effects on the primary outcome (Table 11). The 16 trials reporting power calculations and reporting results as risk differences generally found smaller effects on the primary outcome than were predicted (Figure 9).

TABLE 11 - Full trial reports according to statistical significance of the primary outcome

Study features assessed for full trial reports	Results		Chi-squared test
	Non-significant (N = 22)	Statistically significant (N = 12)
Mean sample size, n	405	778
Median sample size (range)	274 (64–1340)	225 (79–4439)
Industry funded, n/N (%)	4/22 (18)	4/12 (33)	p = 0.912
Median follow-up (months)	35	12
Adequate sequence generation, n/N (%)	18/22 (82)	2/12 (17)	p < 0.001
Adequate allocation concealment, n/N (%)	12/22 (55)	4/12 (33)	p = 0.24
Outcome assessment blinded, n/N (%)	3/22 (14)	3/12 (25)	p = 0.41
No uniform outcome assessment, n/N (%)	3/22 (14)	2/12 (17)	p = 0.81
Primary outcome – patient outcome, n/N (%)	16/22 (73)	5/12 (42)	p = 0.07
Primary outcome – process outcome, n/N (%)	4/22 (17)	5/12 (41)	p = 0.14
Evidence cited (methods) for experimental arm, n/N (%)	11/22 (50)	4/12 (33)	p = 0.35
Evidence cited (methods) for control arm, n/N (%)	6/22 (27)	3/12 (25)	p = 0.89
Evidence cited (methods) in either arm, n/N (%)	11/22 (50)	5/12 (42)	p = 0.64
Any evidence cited for experimental arm, n/N (%)	16/22 (73)	9/12 (75)	p = 0.89
Any evidence cited for control arm, n/N (%)	11/22 (50)	4/12 (33)	p = 0.35
Any evidence cited in either arm, n/N (%)	17/22 (77)	10/12 (83)	p = 0.67

FIGURE 9.

Comparison of observed vs. predicted effects.

Discussion

This is the first review that we know of to have examined the use of the randomised trial design to evaluate monitoring tests. We found that triallists have made valiant attempts to evaluate a wide range of monitoring strategies in various medical fields and clinical settings and the number of studies published per year appears to be increasing over time. Many of the strategies examined were relatively complex, involving the addition or replacement of a test within an existing battery of tests or changing the frequency of one or more tests, presenting considerable challenges for their evaluation. Only a small proportion of trials reported statistically significant results for the primary outcome; various possible reasons for this and the impact of other key features of the trials are worth exploring. From data presented in the Hopewell et al. 227 publication bias paper, between 55% and 75% of published trials reviewed in five studies demonstrated statistically significant effects.

In the first instance, a striking lack of scientific basis for the monitoring strategies that were evaluated was presented. Just as reviews of monitoring strategies specified in clinical guidelines have shown,69,134,135 any existing evidence base for the strategies was poorly cited. Potential issues with the tests, thresholds or interventions, which could have been avoided with a thorough evaluation of each component of the new monitoring strategies in the context in which they were to be applied, were identified only in retrospect. 228

Key to this is the consideration of relevant evidence related to the test(s) to be used (in terms of both accuracy and the ability to predate the appearance of clinically relevant disease), the interval between test applications (which should be influenced by the expected natural history of disease and by the degree of measurement variability associated with the test in the given setting) and the change in patient management that is to be implemented (particularly in terms of establishing the effectiveness of any intervention in the trial population). Providing evidence (or, at the very minimum, a rationale) for each of these components is key to the interpretation of trials of monitoring; as others have pointed out, ‘a mediocre test could improve outcomes when it is coupled with effective management; similarly a quality test could fail to improve outcomes in the absence of effective management’. 151

It appears, however, that insufficient attention was paid to establishing test properties and intervention effectiveness in the populations of interest before trials were undertaken. For example, although one might expect both the rate of disease progression and the degree of measurement variability associated with a given test to be taken into account when setting test frequency, test intervals were apparently determined by convenience or by fear of missing a key clinical event. 38 There was little acknowledgement of the potential for false-positive results, limited use of repeated testing to confirm abnormal or indeterminate test results and examples of ‘personal tailoring’ of decisions based on risk from previous tests, for example by changing the frequency of testing or altering thresholds for intervention on the basis of previous test results, were rare. A small number of studies did acknowledge problems with technical aspects of the tests evaluated, or with the test thresholds evaluated,159–162 with one reporting a change to the threshold in the trial protocol (to minimise the number of biopsies undertaken in the experimental arm). 162 Others found that, although the new monitoring strategies identified potential disease earlier in the disease pathway, as was intended, the disease was detected too early to warrant intervention, the intervention itself may have been ineffective or the lack of observed benefit from monitoring may have resulted from the effectiveness of treatment in the control group. 163–165 It is difficult to assess how widespread such issues might be without a more in-depth examination of each topic area; however, it is clear that, before a trial is undertaken, the specifics of the monitoring strategies to be evaluated should be established using appropriate methods and in a similar population to that eligible for the trial.

The second characteristic of the trials was the lack of detailed description of the protocols for monitoring, particularly for the control groups. Overall, the methods of application of the tests and interventions, and of how test results should be used to inform downstream management, were particularly poorly reported. Given the multiple components of a monitoring strategy and the multitude of ways in which these might interact to affect outcomes, a clearly defined protocol for testing and subsequent management is essential, not only so that the ‘intervention’ can be replicated in the future, but also so that the mechanisms by which outcomes are affected can be better understood and the generalisability of the results beyond the trial can be judged. 151,228 The reasons behind the lack of description are not clear. It may be partly down to poor reporting but could also be related to a lack of acknowledgement of testing strategies as complex interventions or an unwillingness to standardise a complex intervention. 229

Even for those elements of the intended monitoring strategies that were described, we found some evidence of lack of compliance, for example trials reporting the use of the experimental test or the use of extensive additional testing in the control arm; failure to administer treatment despite positive test results; and the prescribing of treatment despite negative test results. 160,163,166–168 This apparent lack of ‘buy in’ by clinicians is difficult to interpret. One explanation might be that a monitoring test is considered more of a guide to potential changes in management rather than as a definitive indication for a particular intervention. The often lengthy time frame of monitoring may also increase the likelihood of trial fatigue, making the intended intervention difficult to implement over long periods. Delaney et al. 228 advocate that a variety of methods might be used to ensure that complex interventions are delivered reliably over a period of time. Alternative designs might also be considered, particularly when monitoring relies primarily on a single test. For example, monitoring of all enrolled patients with randomisation to immediate treatment or a continued surveillance option on crossing a particular threshold leaves less to chance in terms of downstream management. 155–157,169

In terms of study validity, trials of tests should be subject to the same validity standards of randomisation and allocation concealment as the wider RCT literature; however, in many instances, the same standards of blinding of patients, and particularly clinicians, will be difficult to achieve (Box 1). In theory, patient or clinician blinding might be more easily implemented for simpler tests such as blood tests or when tests are applied by a non-treating clinician such as a radiographer, but in many contexts blinding could be more difficult and even inappropriate given that it is not just the test per se that is being evaluated but its interaction with other components of the overall monitoring process. Although we found two examples in which stringent attempts at blinding patients mitigated the potential benefit from monitoring (both using implanted devices),170,171 the full impact of not using blinding in a monitoring trial is as yet unclear. Nevertheless, its impact may be mitigated through the use of objective outcome measures and blinded and uniform outcome assessment. Blinded outcome assessment should be feasible for most trials, but its use was reported for less than one-fifth of our sample and in over half of these trials the blinding related to outcome adjudicators rather than those collecting the outcome data. Furthermore, we observed a small number of trials with a ‘fatal flaw’ in terms of the outcome assessment, whereby the presence of the primary outcome, for example recurrence of cancer, was defined by a different test in each arm, thereby introducing an additional source of bias. It is fundamentally important that the primary outcome is uniformly defined in the same way between groups, ideally at the same point in time.

A final characteristic of these monitoring trials was an apparent lack of power to detect significant effects. This is not a phenomenon limited to trials of tests; previous authors have found sample size calculations to be based on inaccurate assumptions for the control group, with others suggesting that up to 10–20% of trials might be inappropriately discontinued because of a perceived insufficient rate of recruitment. 232–234 However, statistically significant effects are easiest to achieve when two groups are allocated to different treatments; in trials of testing strategies all ‘test-positive’ patients usually undergo the same treatment, reducing the potential to demonstrate clear differences in outcome between groups. We have not yet conducted an in-depth look at the sample size calculations of trials in our sample and cannot yet comment on the assumptions made around the predicted benefit from monitoring; however, we did find evidence of lower than expected control group event rates both in trials that were stopped early and other trials, with some forced to revise their sample size calculations or change their primary outcome from overall survival (OS) to a surrogate outcome of number of recurrences treated surgically with curative intent. 156,170–173

Our research has some limitations. We are unlikely to have retrieved all of the available eligible trials. This is in part because of our focus on one main database, although this was supplemented with searches of more specialist resources, and in part because of a lack of standard terminology for trials of monitoring, which made searching for trials a challenge. However, an exhaustive search is not as important for a methodological review as for a systematic review of effectiveness. We did not aim to systematically identify all of the available trials but to retrieve a sample of trials that provide a good representation of those available. Although other trials may be available, it seems likely that our review has flagged up many of the key issues.

Our review has identified a range of problems with available randomised trials of monitoring, raising real questions regarding the feasibility and appropriateness of this approach for evaluating monitoring. Trial investigators have perhaps underestimated the complexity of the interventions that they were trying to evaluate and the multitude of ways in which the effect of an intended change in a monitoring strategy might be mediated by other factors. The recommendations in Box 2 provide guidance for future researchers evaluating monitoring strategies.

Introduction

Monitoring strategies used to direct the care of patients with potential recurrent or progressive disease are rarely evidence based. 69 Monitoring strategies specify the frequency of observations, the duration of monitoring, the decision rule and the threshold for a positive test result, with a positive result prompting a change in patient management. There is a need for monitoring strategies to be developed based on evidence of how a disease will progress and the performance of the monitoring test to be used. Too often test frequencies are based on routine care schedules with decision rules and thresholds chosen in an ad hoc manner.

Methods

Methodological information related to monitoring was first sought from the first edition of the book Evidence-Based Medical Monitoring,38 edited by Paul Glasziou, Les Irwig and Jeffrey Aronson. Key textwords related to monitoring methodology were identified and purposive searches of MEDLINE were undertaken from 2000 to 2010 (searches conducted on 26 March 2010). Reference tracking and citation tracking using Science Citation Index were used to identify additional relevant literature.

A variety of searches were performed to identify relevant literature using various combinations of the following text words:

• monitor*

• measure* or biomarker* or marker*

• serial or repeat* or periodic or longitudinal or trajectory*

• recurrence or progression

• rule* or threshold* or trigger

• statistical process control or control chart* or reference change value or critical difference

• screen* with frequenc* or intensit* or interval*

When necessary, the results were filtered to select:

• statistical or epidemiological journal titles (Biostatistics, Biometrics, Statistics in Medicine, Methods of Information in Medicine, Lifetime Data Analysis, Journal of Clinical Epidemiology, American Journal of Epidemiology and Annals of Epidemiology)

• RCTs (sensitive search).

Additional searches were undertaken to identify literature related to statistical process control, RCVs and the methodology of health screening.

The papers selected for review are summarised in Table 12 and in the following sections.

Results: development and evaluation of monitoring strategies

Limited methodological literature was identified that provides guidance on the design of studies to evaluate monitoring tests. Work has focused more on analytical techniques to assist with the design of monitoring strategies, primarily through analysis of existing data in order to make recommendations on monitoring frequency or decision rules, or simulation work, with both approaches being specific to the disease area researched.

Results

Summary and conclusions

This review has revealed that there is limited methodological literature on the design of monitoring strategies. Work has focused primarily on the analysis of data, allowing subsequent recommendations to be made for monitoring frequency or decision rules, or simulation work, with both approaches being specific to the disease area being researched. Methods have been developed in the area of screening, with the focus being on identifying the optimal frequency of screening, which could be used for designing monitoring strategies. Some work has been carried out on the design of biomarker development studies, which could potentially be adapted to allow for the evaluation of a monitoring strategy using previously collected specimens. It appears that thresholds are often developed by analysis of the variability of the test being used, identified by the literature describing signal-to-noise ratios, biomarker development studies, statistical process control and RCVs.

The study by Buclin et al. 146 shows an approach in which decision rules were devised by a review of the literature and, then, using an obtained data set and analysis of signal and noise the rules were refined to minimise false results. Following this, recommendations on the decision rule and frequency of monitoring could be made. Takahashi et al. 255,256 and Oke et al. 248 also used signal and noise methods when analysing data and subsequently recommendations could be made for future monitoring strategies.

A number of applications of the signal and noise approach were identified, largely in the area of treatment titration. 143,144,146,237–240,245,248,249,255,256,259 The limitations of this approach for monitoring disease progression or recurrence are that rules and thresholds are devised purely by analysing the variability of test measures and the minimisation of false findings rather than the detection of disease at the earliest point possible and the impact on patients.

The simulation approach proposed by Li and Gatsonis247 uses a joint latent class model that combines predictions from the model with a utility function to identify optimal monitoring frequencies. The results of a simulation study reported by Li and Gatsonis247 appear promising; however, the approach has not been widely adopted, perhaps because of the complex nature of the model. Other simulation approaches may also have potential under certain circumstances, particularly if measurement error and a link between biomarker values and true disease state can be included.

The biases that are well documented in the screening literature are also applicable to the area of monitoring. Length time bias and lead time bias should be considered when analysing monitoring data and when designing monitoring studies. There is also the issue of post-screening noise, which is again important to take into consideration when evaluating a monitoring strategy; the time point at which monitored and non-monitored patients are compared should be selected to minimise the issue of incidence after the final testing point and should also consider the number of likely events. Harm to patients is vitally important in screening and monitoring as this harm may occur at several time points and this must be thought of when designing strategies.

A further consideration in the analysis of monitoring data concerns the number of test measurements and the timing of test measurements: people with more results will contribute more data to the model but they may be very different from those with fewer results. 241 Measurement error and particularly biological variability should also be considered. Studies have shown that RCVs from biological variability studies of healthy participants are not necessarily reflective of the true RCV for a diseased population. As methods to derive test thresholds used in monitoring rely heavily on the variability of test results it is important that estimates from biological variability studies are accurate. In addition, the quality of studies undertaken when developing new biomarkers is not always rigorous; however, there is literature concerning the design of these studies268 and the evaluation of quality of these studies. 265

Methods

A monitoring care pathway was outlined to identify in simple terms the points at which monitoring might affect outcomes (Figure 10). The three identified frameworks68,317,318 were reviewed in terms of their relevance to this monitoring context and 58 trials from a review of RCTs in which monitoring was carried out in at least one arm of the trial (see Chapter 4) were used for illustration purposes. The trials were grouped into three main categories in terms of the change in patient care under evaluation and the intended impact on patient outcome:

1. A new monitoring strategy compared with an existing monitoring strategy, such that the current monitoring test might be replaced by a new and more accurate test, a new test might be added to the strategy or the currently used test might be applied at a different intensity or with an alternative threshold for intervention. Depending on the associated change in patient care, the new monitoring strategy may be intended to detect patients at an earlier stage of disease, to more accurately detect those in need of treatment or to reduce the invasiveness or frequency of testing.

2. A monitoring strategy compared with immediate treatment of all patients at risk of an adverse outcome, in which monitoring may be used to avoid or delay treatment in those who do not need it.

3. A monitoring strategy compared with no monitoring, in which patients are usually treated on the basis of clinical presentation only and the likely aim of monitoring is the detection and treatment of disease at an earlier stage.

FIGURE 10.

Monitoring care pathway: (a) pathway; and (b) detail of ongoing monitoring process. Adapted from Ferrante di Ruffano et al. 69 with permission. +ve, positive; –ve, negative.

In the following sections, we first consider the similarities and differences between monitoring, screening and diagnosis, before broadly outlining the potential for benefit and harm from monitoring and considering the ways in which patient outcomes can be mediated by particular aspects of the monitoring care pathway according to the aim of monitoring and the change in strategy under evaluation.

Potential benefits and harms from monitoring

Figure 11 uses the concept of a 2 × 2 contingency table to illustrate that assigning potential benefits and harms from a monitoring test is not as straightforward as might be imagined. For simplicity, the following is set mainly in the context of a new monitoring strategy to allow earlier detection and treatment.

FIGURE 11.

Summary of the potential harms and benefits from a new monitoring strategy. Harms and benefits are similar to those identified from a screening context by Adriaensen et al. 317 FN, false neagative; FP, false positive; TN, true negative; TP, true positive.

In general terms, for those patients with a ‘true’ result, benefits accrue both to those who would otherwise have experienced a poor clinical outcome but for the new test (A) and to those who would have been detected clinically and successfully treated but the new test allows this to happen at an earlier point in the disease process (B). Benefit also occurs for those with no disease and for whom a negative monitoring test result has a reassurance value, increasing a patient’s sense of control over the disease (G). Positive benefits might also be experienced by all patients, for example with the use of a less invasive test or less frequent testing (H).

Patients who have a ‘false’ monitoring test result will experience harm from the new testing strategy in a similar manner to that experienced after false-negative or false-positive diagnostic test results. False-negative test results can lead to a false feeling of security, delayed detection of disease and potentially a delay in effective treatment until the disease becomes clinically apparent (D). False-positive test results can lead to unnecessary further investigation and/or unnecessary treatment (E). For monitoring tests that aim to detect preclinical or very-early-stage disease, ‘early’ false-positive results (i.e. in patients whose disease would not have progressed to clinically overt disease within a clinically meaningful time frame) will lead to a longer period of time in a diseased state and potentially in overdiagnosis and unnecessary treatment (F). Similarly, ‘early’ positive monitoring tests in those with a true-positive test result (C) can cause harm for those patients who go on to experience a poor clinical outcome and who undergo a longer period of treatment (with associated side effects and a longer period of time in a ‘diseased’ state).

More direct harms can also be incurred by the testing experience, relating to repeated applications of the monitoring test, to any confirmatory testing and to any intervention that is implemented (I and J). These can be physical or psychological in nature and may result from either positive or negative test results, with the serial nature of testing in a monitoring context necessarily multiplying the potential impact. In particular, the ongoing monitoring process (even with repeated negative results) may raise general levels of anxiety and distress and can also have a ‘labelling’ effect that can have a negative influence on patients’ perceptions of themselves and their disease. 319,320

Conclusion

The impact of a monitoring strategy is driven not only by the properties and timing of testing and the effectiveness of treatment but also by patients’ responses to the type and frequency of testing and clinicians trust in, and willingness to comply with, the monitoring protocol. Reitsma et al. 323 advocate that what is more important for clinical decision-making than the level or change in a given marker is the confidence with which that marker can be used to inform patient management. A move towards a test validation paradigm is advocated, by which a number of methods (including establishing test properties) are used to determine whether or not the results of a test are meaningful in practice. In some circumstances, randomised evidence will be needed to fully assess the impact of a test, but this level of evidence will not be needed in every circumstance.

For example, the feasibility of testing and interpretability of test results can be estimated in the development phase of a test, as long as the technical properties of the test are established in clinically relevant populations rather than in laboratory-based studies alone. 316 Patients’ interaction with the testing experience and their likely adherence to monitoring or to subsequent management can be assessed in qualitative studies. Feasibility or pilot studies can help identify what a new test adds to current clinical practice, particularly in terms of clinicians’ interaction with and likely adherence to a new monitoring strategy, and can help identify potential barriers to implementation, as has been recommended for trials of complex interventions. 228 Estimates of key aspects of test performance can be obtained from non-randomised, preferably longitudinal studies comparing tests against a (delayed) reference standard or clinical outcome. 65 The efficacy of treatment is the only mechanism that requires evaluation in a RCT per se; however, the combined effect of the individual mechanisms that come into play may be fully assessable only in a RCT.

Any decision to undertake a RCT should be informed at a minimum by good evidence of the natural history of the disease, the establishment of test properties (in terms of clinical validity and estimation of long-term change in disease status) and evidence (or lack) of treatment efficacy in those patients who are identified by the new monitoring strategy.

Introduction

Tests are used in health care to monitor, and subsequently manage, a variety of chronic conditions. The focus of this research is the monitoring of progressive or recurrent conditions, in which the aim of monitoring is to identify early signs of recurrence or progression, prompting a change in management, typically the initiation of treatment or further testing.

Monitoring strategies are complex interventions combining a test, a schedule, a decision rule and further diagnostic or therapeutic action. Monitoring strategies stipulate the frequency of testing and the ‘monitoring rule’ used to identify when a change in patient management is necessary. A monitoring rule indicates the value or values that would trigger a change in management. Monitoring rules can be simple, such as when a single value above a threshold will prompt a change in management (a ‘snapshot rule’), or more complex, such as when a patient requires a series of test results observed at different time points that fit certain criteria (such as a relative increase from previous measures) to initiate a change in management (a ‘track-shot rule’). 146

Although patient monitoring is a fundamental function of health care, resulting in considerable costs to health-care providers, the area of monitoring is under-researched and there is an increased need for monitoring strategies to be systematically developed with knowledge of the likely progression of disease and the performance of the monitoring test to be used. 62,324 Dinnes et al. ,69 in Chapter 3, reviewed the evidence base for PSA monitoring to identify the recurrence of prostate cancer. The review identified the lack of a systematic approach to developing a monitoring strategy, with monitoring intervals based on standard follow-up schedules and limited evidence of consensus over the thresholds used to initiate treatment.

Stevens et al. 142 discussed various statistical models of the transition between the maintenance and the re-established control phases of monitoring (the process of detecting when a disease is out of control, leading to a change in management, e.g. treatment or more intensive monitoring) and identified a general statistical model for the evolution of monitoring data over time, outlining possible sources of variation. This general statistical model proposes the form of monitoring data based on the observed values of sequential monitoring tests, the values of measurement error and other sources of variability and the true disease state, which can be modelled based on epidemiological evidence but never observed.

This general model, along with existing data and evidence gathered from the literature, can be used to simulate monitoring data and allow for the evaluation of strategies for a given target condition. The potential effect of monitoring strategies can then be evaluated and ranked, prior to full-scale investigation. 325

The example presented here investigates the use of the ELF biomarker in monitoring patients with known liver fibrosis, alongside the ELUCIDATE trial,326 a prospective multicentre randomised trial. The ELUCIDATE trial is evaluating the ELF test for the early detection of progression from liver fibrosis to liver cirrhosis compared with routine care, with the aim of enabling earlier treatment and providing potentially improved patient outcomes. It is described fully in Chapters 16–24.

Aims and objectives

The aim of this study was to identify the optimal monitoring strategy, from candidate monitoring strategies, for patients known to have liver fibrosis receiving repeated testing using the ELF biomarker.

Candidate strategies were selected and evaluated to:

• compare the alternative frequencies of monitoring (6-month or 12-month intervals)

• evaluate the benefit of using targeted retesting compared with no retesting

• compare decision rules [positive results based on crossing a threshold determined by a single value (snapshot simple threshold rule) and positive results dependent on track-shot rules based on absolute or relative increases from the first test value, absolute or relative increases from the last test value and prediction from a linear regression model].

Methods

First, we describe the model used to generate the underlying and unobserved disease progression, incorporating estimates of disease progression and the variability of these estimates, for a cohort of simulated individuals. Then, the process of obtaining observed test result values using the true disease progression values and estimates of test performance is described. Finally, the methods used to evaluate and compare selected monitoring strategies, using both the observed test values and the true disease status, are given. An explanation of the notation used in the model is provided in Table 15.

Simulation of true disease progression

The model simulated true disease progression, generating a random slope and random intercept in terms of fibrosis stage for each patient. The model then converted the fibrosis stages to ELF values.

Fibrosis progression: random slope

The rate at which patients progress through the fibrosis stages was assumed to be constant throughout the stages of fibrosis and normally distributed:

⁽⁴⁾ p i ∼ N ( µ p , σ p 2 ) ,

where i = 1, . . ,n and n is the number of simulated individuals, µ_p is the mean fibrosis progression and σ_p is the SD of fibrosis progression. Fibrosis progression was restricted to only positive values, by fixing p_i at 0.01 if p_i ≤ 0, meaning that only increases in fibrosis stage were simulated, with no patients having decreasing fibrosis stages; however, as the increase is just 0.01 fibrosis units per year, this effectively means that these patients are in a stable fibrosis state.

Fibrosis stage at entry: random intercept

Patients recruited to a trial would be in varying stages of disease at entry. Using data on the likely distribution of fibrosis stage for a population with known liver fibrosis, a multinomial distribution was used to simulate a starting stage for each individual:

⁽⁵⁾ S i ∼ M n ( 0 , 1 , 2 , 3 , 4 ) ( ρ 0 , ρ 1 , ρ 2 , ρ 3 , ρ 4 ) ,

where ρ_s is the probability of starting in each stage, s is the fibrosis stage and s = 0, . . ,4.

Enhanced Liver Fibrosis score link to fibrosis stage

For each stage of fibrosis, the distribution of true ELF values within fibrosis stage was assumed to follow a normal distribution:

⁽⁶⁾ E s ∼ N ( µ s , σ s 2 ) ,

where s is the fibrosis stage and s = 0, . . ,4, µ_S is the mean value of ELF at each fibrosis stage and σ_s is the SD of the mean ELF value at each fibrosis stage.

Enhanced Liver Fibrosis progression between fibrosis stages

The model used fibrosis stage on a continuous scale rather than on a discrete scale. To generate ELF values for each patient at all stages of fibrosis, ELF progression between consecutive integer fibrosis stages was assumed to be linear. It was assumed that patients would have ELF values at the same point of the normal distribution for each fibrosis stage (patients would remain a given number of SDs from the mean). To randomly select the point of the normal distribution that patients would follow, a value from the standard normal distribution was generated for each patient, z_i∼N (0,1). The ELF value for each participant at each stage of fibrosis was E_is = µ_s + (z_iσ_s) (Figure 12a).

FIGURE 12.

(a) Fibrosis units linked to ELF value; (b) ELF value progression through time; and (c) starting stage adjusted ELF value progression through time.

Enhanced Liver Fibrosis progression: random slope

The ELF values at the beginning of each fibrosis stage for each individual (E_is) and the rate at which each simulated participant progresses through fibrosis (p_i) were combined to calculate the increase in ELF per year. The gradient of ELF progression was β_is = (E_i,s + 1 − E_i,s)p_i, for s = 0, . . ,3. The gradient of ELF progression after stage 4 was assumed to be the same as the gradient between stages 3 and 4 (β_i3 = β_i4). The time point signalling when each individual would progress to the next fibrosis stage was πis=spi. β_is is the random slope in terms of ELF progression.

The underlying and true ELF progression for each stage and all time points from the onset of fibrosis was then calculated as:

⁽⁷⁾ E i j s = E i s + β i s x j s ,

where x_js is time within stage s and:

⁽⁸⁾ 0 ≤ x j s < π i , s + 1 − π i , s for s = 0 , . . , 3 .

For stage 4, s = 4 and x_js = 0, . . , ∞. The true ELF values for each individual across time could also be expressed as:

⁽⁹⁾ E i t = { E i j 0 for 0 ≤ t < π i 1 E i j 1 for π i 1 ≤ t < π i 2 E i j 2 for π i 2 ≤ t < π i 3 E i j 3 for π i 3 ≤ t < π i 4 E i j 4 for π i 4 ≤ t ,

where t is time across all stages (t = 0, . . , ∞). This allowed the simulation of lifetime progression data for a cohort of patients (see Figure 12b). ELF values were truncated at 0 if a negative value was simulated.

Enhanced Liver Fibrosis value at entry: random intercept

The time at registration for each participant (τri) was a randomly selected time point from the time period when the individual was in his or her generated fibrosis stage at study entry (S_i), that is, a random value from the interval [πi,si,πi,si+1), where S_i is the starting stage for each individual and S_i = 0, . . ,3. If the participant was in stage 4 of fibrosis at entry, S_i = 4 and τri was generated by identifying a random value from the interval [π_i,4,π_i,4 + c₁], where c₁ is the longest given amount of time that a patient can be in stage 4 of liver fibrosis before entering the trial. τri also has a maximum value of c₂, where c₂ is the longest amount of time that a patient can have fibrosis on registration to the trial. If, for a simulated individual, τri>πi,4+c1 or τri>c2, the data for that individual were not used in the analysis. This was done to prevent patients being included when they would be confirmed as having cirrhosis or were at a point of fibrosis that they would not have reached in their lifetime because of their simulated progression rate. The c₁ time used means that participants registering in the trial who are in stage 4 of liver fibrosis have been in stage 4 for a maximum of c₁ years. The c₂ time used means that participants have liver fibrosis for a maximum of c₂ years before being registered in the trial.

In the ELUCIDATE trial, a registration ELF test was given to each patient to assess eligibility. The first ELF test included in the trial data was taken at the point of randomisation. If the start of the trial occurred c₃ time units after registration, then the time at randomisation was τti=τri+c3. The random intercept in terms of ELF is α_i = E_it for t=τti.

Random slope and random intercept model in terms of the Enhanced Liver Fibrosis test

The underlying disease progression for the simulated individuals over the time of the trial was U_it = E_it for τti≤t<τti+c4, where c₄ is the duration of the trial and i denotes simulated patients with an eligible registration ELF value (i = 1, . . ,N, where N is the number of simulated patients available for randomisation) (see Figure 12c).

Simulation of observed values

The true underlying ELF measurements were converted to observed ELF measures by the addition of error.

Error

The error at each observation point (ω_it) was formed of within-individual variation and analytical variation. Error was assumed to be normally distributed with a mean of zero:

⁽¹⁰⁾ ω i t ∼ N ( 0 , σ ω 2 ) .

The observed ELF measurement at any given time was:

⁽¹¹⁾ Y i t = U i t + ω i t .

Values were adjusted to equal 0 if a negative observed ELF value was simulated.

Entry criteria

To fulfil trial entry criteria the observed ELF measurement at registration had to be greater than the preset value of Y^*_r; thus, the equation:

⁽¹²⁾ Y i t > Y r * for t = τ r

had to be satisfied for each simulated participant to be included in the trial data. An example of simulated observed ELF measures can be seen in Figure 13.

FIGURE 13.

Observed ELF values.

Evaluation of a monitoring strategy

To evaluate each strategy, the decision made from implementing that monitoring strategy using the simulated observed values and the corresponding true underlying values was assessed. With knowledge of the true underlying disease state of each participant, the performance of a variety of monitoring strategies was evaluated.

Comparison of observed results with the underlying disease state

Participants had a positive or negative test result based on the simulated observed data and the decision rule employed. The test result was then found to be either true or false depending on the underlying disease state. The purpose of the ELF test was to identify when patients enter compensated cirrhosis, stage 4. From the simulation of underlying true ELF values, the time point when each individual enters stage 4 was π_i,4. As it may be beneficial to identify patients prior to time π_i,4, participants were classed as ‘diseased’ δ units of time prior to π_i,4 and onwards. If at a testing point a participant was diseased (≥ π_i,4 − δ), a positive result would be a true positive and a negative result would be a false negative. If at a testing point the patient was not diseased (< π_i,4 − δ), a negative result would be a true negative and a positive result would be a false positive. As a positive result (truly or falsely) caused a change in management and cessation of monitoring, patients who achieved a positive result did not have a test result at subsequent monitoring times. Figure 14 illustrates how a strategy with a simple threshold decision rule can be evaluated.

FIGURE 14.

Results of implementing a monitoring strategy.

Measuring the performance of a monitoring strategy

The performance of a monitoring strategy was assessed at each monitoring point by calculating the number of patients at each monitoring test visit and specifically the number of true-positive, false-positive, true-negative and false-negative test results. The number of tests carried out across the duration of the strategy was used to represent resource use. The positive predictive value (PPV) was used to investigate how likely it was for an individual with a positive result to be diseased. To measure patient harm the delay from onset of disease to the point of diagnosis was calculated; this was the time between the onset of compensated cirrhosis (π_i,4) and a patient having a positive test result.

When comparing strategies the number of tests per person for the duration of monitoring, the PPV (for all tests over the duration of monitoring) and the percentage of patients with a delayed diagnosis (delay from onset of disease to diagnosis of > 12 months) were used to measure performance. To allow for comparisons to be made between strategies in which only two of the three measures of performance ranged, thresholds used by monitoring strategies were varied to obtain a PPV of 25%. A PPV of 25% was chosen as this would be an acceptable PPV in practice.

Evaluation of strategies

The strategy evaluated first was the simple threshold strategy, with observations every 6 months and no retest component (reference strategy). Alternative strategies were evaluated in which individual components of the reference strategy were varied: the frequency of monitoring (θ), the decision rule and whether or not a retest value was used. The same simulated data were used when evaluating strategies A–H.

Sensitivity analyses were carried out to estimate the effect of inaccurate information regarding test performance and progression of liver disease. Estimates used in the simulation of data were altered (halved and doubled) and the reference strategy was evaluated with all aspects of the strategy kept constant (including the threshold value). Results for the reference strategy with the threshold varied to give a PPV of 25% using data with altered estimates were also obtained. Further sensitivity analyses were undertaken in which the fibrosis progression rate was adjusted based on expert opinion; these were analyses of strategies A–H as for the main analysis (with PPV held at 25%) and the reference strategy using varied estimates to generate monitoring data.

To assess the accuracy of the model, the mean randomisation ELF values and SDs were calculated and compared for the ELUCIDATE and simulated data sets. ANOVA was used to assess between-individual and within-individual variability in the ELF values recorded for patients in the trial and the simulated results. Multilevel models were fitted using the simulated observed values and the observed values from the ELUCIDATE trial (for participants with two or more ELF measures post registration) and the results from these models were compared. In the ELUCIDATE trial ELF measurements were not taken in the majority of cases after participants had an ELF value of ≥ 9.5. To allow for this, the ELUCIDATE and simulated data sets were modified so that each patient with an ELF value of ≥ 9.5 did not undergo any subsequent measurements.

Simulations were based on a cohort of 20,000 patients to give adequate precision. With 20,000 test results, if one of the performance measures gave an estimate of 15%, the corresponding 95% CI would range from 14.5% to 15.5%; for an estimate of 1.5% the 95% CI would range from 1.3% to 1.7%.

Results

The estimates from various data sources and details of the data used in the simulation model can be found in Table 16. In addition, the estimates regarding the trial used in the simulation are shown in Table 17. When evaluating strategies patients were considered truly positive if they received a positive result 3 months prior to entering compensated cirrhosis (δ = 0.25). When using targeted retesting, patients with initial values within ±1 of the threshold value have a retest (Δ = 1).

The same simulated data were used when evaluating strategies A–H. For the simulated cohort of 20,000 patients, 5314 (26.6%) would develop cirrhosis if there were no intervention during the period of the trial.

Comparing strategies with changes to individual components with the reference strategy

Table 19 and Figure 15 show the performance of various monitoring strategies. The results of each strategy by observation point are provided in Tables 20–26.

TABLE 19 - Results of strategies A–H

FP, false positive; TP, true positive.

Tests over the duration of monitoring.

Patients with a delayed diagnosis (delay from onset of disease to diagnosis of > 12 months).

Number of tests per person over the duration of monitoring.

% of all patients with a delay to diagnosis.

% of patients who would reach cirrhosis within the trial period with a delay to diagnosis.

The number of TP results per person over the duration of monitoring.

The number of FP results per person over the duration of monitoring.

218, 974 tests were carried out to generate 153,971 results because of retests being used.

Strategy	Monitoring strategy components					PPV, %	Testsa			Delay to diagnosisb			Test performance
	Decision rule	Threshold value	Observation interval (months)	Retest	Initial threshold		N	N per personc	Median IQR	N	% of alld	% of stage 4e	TP per personf	FP per persong	Positive, n (%)	Sensitivity, %
A	Simple threshold	10.715	6	False	–	25	152,724	7.64	11 (3, 11)	1220	6.10	22.96	0.12	0.37	9883 (6.47)	22.13
B	Simple threshold	10.58	6	True	–	25	218,974h	10.95	12 (6, 15)	1300	6.50	24.46	0.12	0.35	9406 (6.11)	21.15
C	Simple threshold	10.55	12	False	–	25	86,787	4.34	6 (2, 6)	1249	6.25	23.5	0.13	0.38	10,053 (11.58)	35.34
D	Absolute increase from initial value	1.295	6	False	10.715	25	155,648	7.78	11 (4, 11)	1536	7.68	28.9	0.13	0.40	10,598 (6.81)	19.85
E	Absolute increase from last value	1.46	6	False	10.715	25	172,363	8.62	11 (7, 11)	2085	10.42	39.24	0.08	0.24	6305 (3.66)	8.45
F	Relative increase from initial value	1.144	6	False	10.715	25	156,460	7.82	11 (4, 11)	1630	8.15	30.67	0.13	0.38	10,266 (6.56)	18.12
G	Relative increase from last value	1.1795	6	False	10.715	25	176,385	8.82	11 (9, 11)	2217	11.09	41.72	0.07	0.20	5338 (3.03)	6.68
H	Linear regression	10.495	6	False	10.715	25	150,478	7.52	11 (2, 11)	1126	5.63	21.19	0.12	0.35	9342 (6.21)	21.58

FIGURE 15.

Results of various monitoring strategies.

TABLE 20 - Results by observation point using the retest monitoring strategy (strategy B)

FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Tests performed/number of people who tests were performed on (number of results generated).

Tests performed when the patient was diseased.

Observation time (months)	Tests,a N/n	Results, n (%)					Diseased,b n (%)	PPV, %	Sensitivity, %
		TP	FP	FN	TN	Positive
0	28,357/20,000	1233 (6.2)	2447 (12.2)	700 (3.5)	15,620 (78.1)	3680 (18.4)	1933 (9.7)	33.5	63.8
6	22,924/16,320	144 (0.9)	796 (4.9)	711 (4.4)	14,669 (89.9)	940 (5.8)	855 (5.2)	15.3	16.8
12	21,549/15,380	125 (0.8)	568 (3.7)	734 (4.8)	13,953 (90.7)	693 (4.5)	859 (5.6)	18.0	14.6
18	20,617/14,687	97 (0.7)	494 (3.4)	775 (5.3)	13,321 (90.7)	591 (4.0)	872 (5.9)	16.4	11.1
24	19,913/14,096	117 (0.8)	438 (3.1)	776 (5.5)	12,765 (90.6)	555 (3.9)	893 (6.3)	21.1	13.1
30	19,167/13,541	103 (0.8)	408 (3.0)	803 (5.9)	12,227 (90.3)	511 (3.8)	906 (6.7)	20.2	11.4
36	18,555/13,030	105 (0.8)	413 (3.2)	816 (6.3)	11,696 (89.8)	518 (4.0)	921 (7.1)	20.3	11.4
42	17,927/12,512	105 (0.8)	422 (3.4)	849 (6.8)	11,136 (89.0)	527 (4.2)	954 (7.6)	19.9	11.0
48	17,272/11,985	108 (0.9)	408 (3.4)	858 (7.2)	10,611 (88.5)	516 (4.3)	966 (8.1)	20.9	11.2
54	16,674/11,469	126 (1.1)	392 (3.4)	855 (7.5)	10,096 (88.0)	518 (4.5)	981 (8.6)	24.3	12.8
60	16,019/10,951	88 (0.8)	269 (2.5)	887 (8.1)	9707 (88.6)	357 (3.3)	975 (8.9)	24.6	9.0
All	218,974/153,971	2351 (1.5)	7055 (4.6)	8764 (5.7)	13,5801 (88.2)	9406 (6.1)	11,115 (7.2)	25.0	21.2

TABLE 21 - Results by observation point using the reduced frequency of monitoring strategy (strategy C)

FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Tests performed when the patient was diseased.

Observation time (months)	Tests, N	Results, n (%)					Diseased,a n (%)	PPV, %	Sensitivity, %
		TP	FP	FN	TN	Positive
0	20,000	1247 (6.2)	2735 (13.7)	686 (3.4)	15,332 (76.7)	3982 (19.9)	1933 (9.7)	31.3	64.5
12	16,018	277 (1.7)	1253 (7.8)	739 (4.6)	13,749 (85.8)	1530 (9.6)	1016 (6.3)	18.1	27.3
24	14,488	243 (1.7)	983 (6.8)	775 (5.3)	12,487 (86.2)	1226 (8.5)	1018 (7.0)	19.8	23.9
36	13,262	247 (1.9)	952 (7.2)	782 (5.9)	11,281 (85.1)	1199 (9.0)	1029 (7.8)	20.6	24.0
48	12,063	246 (2.0)	861 (7.1)	806 (6.7)	10,150 (84.1)	1107 (9.2)	1052 (8.7)	22.2	23.4
60	10,956	252 (2.3)	757 (6.9)	809 (7.4)	9138 (83.4)	1009 (9.2)	1061 (9.7)	25.0	23.8
All	86,787	2512 (2.9)	7541 (8.7)	4597 (5.3)	72,137 (83.1)	10,053 (11.6)	7109 (8.2)	25.0	35.3

TABLE 22 - Results by observation point using the absolute increase from start value monitoring strategy (strategy D)

FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Tests performed when the patient was diseased.

Observation time (months)	Tests, N	Results, n (%)					Diseased,a n (%)	PPV, %	Sensitivity, %
		TP	FP	FN	TN	Positive
0	20,000	1162 (5.8)	2236 (11.2)	771 (3.9)	15,831 (79.2)	3398 (17.0)	1933 (9.7)	34.2	60.1
6	16,602	50 (0.3)	564 (3.4)	890 (5.4)	15,098 (90.9)	614 (3.7)	940 (5.7)	8.1	5.3
12	15,988	66 (0.4)	517 (3.2)	1014 (6.3)	14,391 (90.0)	583 (3.6)	1080 (6.8)	11.3	6.1
18	15,405	95 (0.6)	550 (3.6)	1085 (7.0)	13,675 (88.8)	645 (4.2)	1180 (7.7)	14.7	8.1
24	14,760	116 (0.8)	582 (3.9)	1132 (7.7)	12,930 (87.6)	698 (4.7)	1248 (8.5)	16.6	9.3
30	14,062	169 (1.2)	559 (4.0)	1123 (8.0)	12,211 (86.8)	728 (5.2)	1292 (9.2)	23.2	13.1
36	13,334	187 (1.4)	603 (4.5)	1066 (8.0)	11,478 (86.1)	790 (5.9)	1253 (9.4)	23.7	14.9
42	12,544	174 (1.4)	586 (4.7)	1048 (8.4)	10,736 (85.6)	760 (6.1)	1222 (9.7)	22.9	14.2
48	11,784	212 (1.8)	583 (4.9)	955 (8.1)	10,034 (85.1)	795 (6.7)	1167 (9.9)	26.7	18.2
54	10,989	218 (2.0)	591 (5.4)	865 (7.9)	9315 (84.8)	809 (7.4)	1083 (9.9)	26.9	20.1
60	10,180	204 (2.0)	574 (5.6)	765 (7.5)	8637 (84.8)	778 (7.6)	969 (9.5)	26.2	21.1
All	155,648	2653 (1.7)	7945 (5.1)	10,714 (6.9)	134,336 (86.3)	10,598 (6.8)	13,367 (8.6)	25.0	19.8

TABLE 23 - Results by observation point using the absolute increase from last value monitoring strategy (strategy E)

FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Tests performed when the patient was diseased.

Observation time (months)	Tests, N	Results, n (%)					Diseased,a n (%)	PPV, %	Sensitivity, %
		TP	FP	FN	TN	Positive
0	20,000	1162 (5.8)	2236 (11.2)	771 (3.9)	15,831 (79.2)	3398 (17.0)	1933 (9.7)	34.2	60.1
6	16,602	28 (0.2)	342 (2.1)	912 (5.5)	15,320 (92.3)	370 (2.2)	940 (5.7)	7.6	3.0
12	16,232	22 (0.1)	281 (1.7)	1086 (6.7)	14,843 (91.4)	303 (1.9)	1108 (6.8)	7.3	2.0
18	15,929	32 (0.2)	256 (1.6)	1230 (7.7)	14,411 (90.5)	288 (1.8)	1262 (7.9)	11.1	2.5
24	15,641	31 (0.2)	256 (1.6)	1378 (8.8)	13,976 (89.4)	287 (1.8)	1409 (9.0)	10.8	2.2
30	15,354	34 (0.2)	240 (1.6)	1538 (10.0)	13,542 (88.2)	274 (1.8)	1572 (10.2)	12.4	2.2
36	15,080	39 (0.3)	248 (1.6)	1682 (11.2)	13,111 (86.9)	287 (1.9)	1721 (11.4)	13.6	2.3
42	14,793	52 (0.4)	228 (1.5)	1860 (12.6)	12,653 (85.5)	280 (1.9)	1912 (12.9)	18.6	2.7
48	14,513	56 (0.4)	223 (1.5)	2014 (13.9)	12,220 (84.2)	279 (1.9)	2070 (14.3)	20.1	2.7
54	14,234	45 (0.3)	204 (1.4)	2212 (15.5)	11,773 (82.7)	249 (1.7)	2257 (15.9)	18.1	2.0
60	13,985	78 (0.6)	212 (1.5)	2423 (17.3)	11,272 (80.6)	290 (2.1)	2501 (17.9)	26.9	3.1
All	172,363	1579 (0.9)	4726 (2.7)	17,106 (9.9)	148,952 (86.4)	6305 (3.7)	18,685 (10.8)	25.0	8.5

TABLE 24 - Results by observation point using the relative increase from start value monitoring strategy (strategy F)

FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Tests performed when the patient was diseased.

Observation time (months)	Tests, N	Results, n (%)					Diseased,a n (%)	PPV, %	Sensitivity, %
		TP	FP	FN	TN	Positive
0	20,000	1162 (5.8)	2236 (11.2)	771 (3.9)	15,831 (79.2)	3398 (17.0)	1933 (9.7)	34.2	60.1
6	16,602	44 (0.3)	641 (3.9)	896 (5.4)	15,021 (90.5)	685 (4.1)	940 (5.7)	6.4	4.7
12	15,917	48 (0.3)	539 (3.4)	1041 (6.5)	14,289 (89.8)	587 (3.7)	1089 (6.8)	8.2	4.4
18	15,330	84 (0.5)	525 (3.4)	1125 (7.3)	13,596 (88.7)	609 (4.0)	1209 (7.9)	13.8	6.9
24	14,721	100 (0.7)	558 (3.8)	1193 (8.1)	12,870 (87.4)	658 (4.5)	1293 (8.8)	15.2	7.7
30	14,063	143 (1.0)	524 (3.7)	1217 (8.7)	12,179 (86.6)	667 (4.7)	1360 (9.7)	21.4	10.5
36	13,396	169 (1.3)	519 (3.9)	1184 (8.8)	11,524 (86.0)	688 (5.1)	1353 (10.1)	24.6	12.5
42	12,708	179 (1.4)	528 (4.2)	1164 (9.2)	10,837 (85.3)	707 (5.6)	1343 (10.6)	25.3	13.3
48	12,001	195 (1.6)	568 (4.7)	1099 (9.2)	10,139 (84.5)	763 (6.4)	1294 (10.8)	25.6	15.1
54	11,238	226 (2.0)	528 (4.7)	1009 (9.0)	9475 (84.3)	754 (6.7)	1235 (11.0)	30.0	18.3
60	10,484	220 (2.1)	530 (5.1)	918 (8.8)	8816 (84.1)	750 (7.2)	1138 (10.9)	29.3	19.3
All	156,460	2570 (1.6)	7696 (4.9)	11,617 (7.4)	134,577 (86.0)	10,266 (6.6)	14187 (9.1)	25.0	18.1

TABLE 25 - Results by observation point using the relative increase from last value monitoring strategy (strategy G)

FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Tests performed when the patient was diseased.

Observation time (months)	Tests, N	Results, n (%)					Diseased,a n (%)	PPV, %	Sensitivity, %
		TP	FP	FN	TN	Positive
0	20,000	1162 (5.8)	2236 (11.2)	771 (3.9)	15,831 (79.2)	3398 (17.0)	1933 (9.7)	34.2	60.1
6	16,602	16 (0.1)	263 (1.6)	924 (5.6)	15,399 (92.8)	279 (1.7)	940 (5.7)	5.7	1.7
12	16,323	9 (0.1)	206 (1.3)	1117 (6.8)	14,991 (91.8)	215 (1.3)	1126 (6.9)	4.2	0.8
18	16,108	18 (0.1)	199 (1.2)	1280 (7.9)	14,611 (90.7)	217 (1.3)	1298 (8.1)	8.3	1.4
24	15,891	13 (0.1)	184 (1.2)	1455 (9.2)	14,239 (89.6)	197 (1.2)	1468 (9.2)	6.6	0.9
30	15,694	16 (0.1)	170 (1.1)	1642 (10.5)	13,866 (88.4)	186 (1.2)	1658 (10.6)	8.6	1.0
36	15,508	13 (0.1)	172 (1.1)	1829 (11.8)	13,494 (87.0)	185 (1.2)	1842 (11.9)	7.0	0.7
42	15,323	22 (0.1)	161 (1.1)	2054 (13.4)	13,086 (85.4)	183 (1.2)	2076 (13.5)	12.0	1.1
48	15,140	25 (0.2)	140 (0.9)	2264 (15.0)	12,711 (84.0)	165 (1.1)	2289 (15.1)	15.2	1.1
54	14,975	20 (0.1)	134 (0.9)	2511 (16.8)	12,310 (82.2)	154 (1.0)	2531 (16.9)	13.0	0.8
60	14,821	21 (0.1)	138 (0.9)	2809 (19.0)	11,853 (80.0)	159 (1.1)	2830 (19.1)	13.2	0.7
All	176,385	1335 (0.8)	4003 (2.3)	18,656 (10.6)	152,391 (86.4)	5338 (3.0)	19,991 (11.3)	25.0	6.7

TABLE 26 - Results by observation point using the linear regression monitoring strategy (strategy H)

FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Tests performed when the patient was diseased.

Observation time (months)	Tests, N	Results, n (%)					Diseased,a n (%)	PPV, %	Sensitivity, %
		TP	FP	FN	TN	Positive
0	20,000	1162 (5.8)	2236 (11.2)	771 (3.9)	15,831 (79.2)	3398 (17.0)	1933 (9.7)	34.2	60.1
6	16,602	280 (1.7)	1488 (9.0)	660 (4.0)	14,174 (85.4)	1768 (10.6)	940 (5.7)	15.8	29.8
12	14,834	146 (1.0)	741 (5.0)	650 (4.4)	13,297 (89.6)	887 (6.0)	796 (5.4)	16.5	18.3
18	13,947	92 (0.7)	492 (3.5)	685 (4.9)	12,678 (90.9)	584 (4.2)	777 (5.6)	15.8	11.8
24	13,363	87 (0.7)	385 (2.9)	706 (5.3)	12,185 (91.2)	472 (3.5)	793 (5.9)	18.4	11.0
30	12,891	93 (0.7)	307 (2.4)	742 (5.8)	11,749 (91.1)	400 (3.1)	835 (6.5)	23.2	11.1
36	12,491	85 (0.7)	284 (2.3)	772 (6.2)	11,350 (90.9)	369 (3.0)	857 (6.9)	23.0	9.9
42	12,122	68 (0.6)	268 (2.2)	838 (6.9)	10,948 (90.3)	336 (2.8)	906 (7.5)	20.2	7.5
48	11,786	100 (0.8)	273 (2.3)	859 (7.3)	10,554 (89.5)	373 (3.2)	959 (8.1)	26.8	10.4
54	11,413	107 (0.9)	277 (2.4)	880 (7.7)	10,149 (88.9)	384 (3.4)	987 (8.6)	27.9	10.8
60	11,029	111 (1.0)	260 (2.4)	906 (8.2)	9752 (88.4)	371 (3.4)	1017 (9.2)	29.9	10.9
All	150,478	2331 (1.5)	7011 (4.7)	8469 (5.6)	132,667 (88.2)	9342 (6.2)	10,800 (7.2)	25.0	21.6

Inferior strategies

The retest strategy (strategy B) and the strategies with decision rules based on absolute and relative increases from the first and last recorded value (strategies D–G) were inferior to the reference strategy, requiring more tests and causing more patients who had progressed to liver cirrhosis to experience a delay to diagnosis.

Compared with the reference strategy, the main effect of the retest strategy was to increase the number of tests performed (increase of 3.30 tests per person), with also a small increase in the percentage of patients with a delay to diagnosis (absolute increase of 0.40 percentage points), whereas the strategies with decision rules based on absolute and relative increases from the initial value showed only small increases in the number of tests required (increases of 0.14 and 0.18 tests per person, respectively) but larger increases in the percentage of patients with a delay to diagnosis (absolute increases of 1.58 and 2.05 percentage points, respectively). The absolute and relative increase from last recorded value decision rules both increased the number of tests required (by 0.98 and 1.18 per person, respectively) and increased the percentage of patients with a delay to diagnosis (to 10.42% and 11.09%, respectively).

‘Trade-off’ strategies

The reduced monitoring frequency strategy (strategy C) showed a ‘trade-off’ between delay to diagnosis and the number of tests required compared with the reference strategy. The number of tests required decreased by 3.30 tests per person and the percentage of patients with a delay to diagnosis increased by 0.15 percentage points (absolute increase) compared with the reference strategy.

Superior strategies

The reference strategy was found to be inferior to the linear regression strategy. The linear regression strategy used fewer tests (decrease of 0.12 tests per person) and resulted in a lower percentage of patients with a delay to diagnosis (absolute decrease of 0.47 percentage points) compared with the reference strategy.