The clinical effectiveness and cost-effectiveness of the PROGENSA® prostate cancer antigen 3 assay and the Prostate Health Index in the diagnosis of prostate cancer: a systematic review and economic evaluation

Gleason score

A histopathologist reviews biopsy specimens. If cancerous cells are detected, the histopathology report includes the Gleason score;25 the Gleason score is a measure of the aggressiveness of the tumour. The Gleason score25 (range 2–10) describes the degree of abnormality of the tumour found in the biopsy. The higher the Gleason score, the more aggressive (and worse prognosis) the cancer.

The Gleason score25 is calculated by first assessing (using a microscope) the biopsy specimen for the degree of abnormality in the prostate tissue, which is categorised as one of five different Gleason patterns. Gleason pattern 1 is the most differentiated and therefore the most favourable, and pattern 5 is the most disrupted and aggressive. Pattern 3 is the most common. The Gleason score is obtained by adding together the number of the most widespread pattern (primary grade) and the number of the second most prevalent pattern (secondary grade). If a tumour has patterns 3 and 2, the score would be 5. If the tumour has only one pattern, or less than 5% of a secondary pattern, the single pattern is added to itself (e.g. 3 + 3 = 6). It is advised that the diagnosis of low-grade Gleason score 2–5 prostate carcinomas in the setting of needle biopsy should be made with extreme caution,25 as such a diagnosis on final radical prostatectomy is proved wrong most of the time. 26 Recent consensus is that diagnosed prostate cancer must have a minimum score of 6. 27,28 Cancers with a Gleason score higher than 7 are considered to be aggressive.

Other reported abnormalities

Apart from cancerous cells, other abnormalities which may be reported on histopathology reports include:

• HGPIN. This is a premalignant change in glands which has been shown to be associated with increased risk of invasive cancer elsewhere in the prostate.

• ASAP. Atypical changes are present in cells but the pathologist is uncertain of their significance.

Clinically insignificant prostate cancer

The prognosis and natural history of prostate cancer vary with the extent of spread and grade of cancer at diagnosis. Clinically insignificant prostate cancer can be defined as a cancer which will not affect the patient during the natural course of his lifetime, meaning that he is likely to die from other causes. 29 The detection of these potentially clinically insignificant cancers on either initial or second biopsy is an important issue and can lead to potentially invasive and unnecessary treatment as well as increased anxiety for men who live with a diagnosis of prostate cancer that may not affect their life expectancy.

There are a number of different definitions of the term ‘clinically insignificant prostate cancer’. The definitions are based on observed survival rates after radical prostatectomy. These pathology-based definitions require that the disease is restricted to the prostate, with a Gleason score of 6 or less. In addition, some definitions include limits on the total tumour volume and/or largest individual tumour volume. 30,31 However, in clinical practice the challenge is to correctly identify men with clinically insignificant disease before any treatment or surgery, that is at diagnosis. There are several systems for predicting the risk of localised prostate cancer progressing. 32–34 However, recent data have suggested that these tools may be inaccurate33 and the NICE guideline11 includes a research recommendation for further research in this area.

Lack of long-term evidence

Ideally, clinical utility would be assessed in ‘end-to-end’ or ‘test-to-treatment’ studies and it would be possible to follow men from early clinical investigation through to diagnosis, treatment and long-term follow-up for prostate cancer. Such end-to-end studies of clinical utility are often not available. Published studies of clinical validity frequently focus on the diagnostic process and assess the performance of the different tests. Thus, although available studies provide some information on the effectiveness of the intervention tests, data describing the long-term impact of using new tests are often scarce.

Lack of clinically relevant comparisons

Many clinical validity studies focus on the use of (1) a new test or (2) a new test that is a replacement for an existing test. However, usually the comparator and intervention pathways involve combining multiple tests.

Sampling bias

Study recruitment may be restricted to, for example, men in the PSA ‘grey zone’ (i.e. with a PSA of 4–10 ng/ml) or to men with abnormal DRE findings in addition to a negative or equivocal initial biopsy. This means that the range of clinical assessment variables is restricted in the study population and hence the observed diagnostic accuracy of these clinical variables will be reduced. This sampling bias affects the generalisability of study findings to the population of interest to this review (i.e. all men with a negative or equivocal first biopsy).

Verification bias

Studies that consist of opportunistic cohorts of patients presenting at referral centres will include biopsy results for men who have been referred for, and have accepted, a repeat biopsy. Acceptance of biopsy is likely to be related to PSA level or PCA3/phi score; it is more likely that men with higher PSA levels will accept a repeat biopsy. This leads to so-called differential verification bias, when the availability of the reference standard result is dependent on the result of the intervention or comparator test.

Imperfect reference standard

In clinical validity studies the diagnostic accuracy of a new or intervention test is assessed against a reference standard. The reference standard is the best test available, that is the current preferred method of diagnosing a disease. In the case of prostate cancer, the reference standard is a biopsy. The diagnostic capabilities of all new tests need to be compared against the diagnostic accuracy of a biopsy.

In prostate cancer the reference standard (biopsy) does not detect all cancers and is considered to be imperfect. Some men with a negative biopsy result do have an undetected cancer. These men are indicated by x and y in the Table 1.

Different types of biopsy have different cancer detection rates and the sensitivity and specificity of the intervention pathways may therefore differ depending on the type of second biopsy that is carried out (see Types of biopsy). A different biopsy sampling scheme (such as saturation or extended) might mean that x and y are moved to the biopsy-positive column, as shown in Table 2.

The estimate of diagnostic accuracy for PCA3/phi scores will alter for different biopsy types if the proportion of x/false positive (FP) is not the same as the y/true negative (TN) value, that is if men in whom cancers were missed on a standard biopsy but would have been detected on a saturation biopsy are more likely or less likely than men with cancer detected on a standard biopsy to have raised PCA3/phi scores. This is plausible. For instance, a standard biopsy is more likely to detect widespread, rather than localised, cancers. If widespread cancers are also associated with higher PCA3/phi scores than localised disease, then a ‘better’ biopsy scheme which picks up more localised disease might reduce the diagnostic accuracy of the use of the PCA3 assay or phi.

Imaging used with biopsy: incorporation bias

A separate issue relating to biopsy is the type of imaging used. Using ultrasound or MRI to guide the biopsy in effect incorporates another test into the reference standard. Men with lesions detectable on ultrasound or MRI often have additional biopsy cores taken which have come from the area surrounding the identified lesions. This may well increase the chance of a positive biopsy result and so increase the observed diagnostic accuracy of MRI. However, this means that the type of reference standard used differs according to the MRI test result; if MRI is positive, more cores would be taken than if MRI was negative.

Electronic databases

The following databases were searched on 28 April or 19 May 2014 for eligible studies:

• MEDLINE

• EMBASE

• Cochrane Central Register of Controlled Trials

• Health Technology Assessment (HTA) database

• Cochrane Database of Systematic Reviews

• Database of Abstracts of Reviews of Effectiveness

• ISI Web of Science

• Medion database for related diagnostic test accuracy reviews (www.mediondatabase.nl/)

• Aggressive Research Intelligence Facility database (www.birmingham.ac.uk/research/activity/mds/projects/HaPS/PHEB/ARIF/databases/index.aspx)

• PROSPERO systematic review register (www.crd.york.ac.uk/PROSPERO/).

No study design filters were applied and non-English-language reports were excluded. All databases were searched from 2000. The following types of report were excluded:

• editorials, opinion pieces and correspondence on journal articles

• conference abstracts.

Trial and research registers were searched on 24 July 2014 for ongoing trials and reviews including:

• ClinicalTrials.gov (http://clinicaltrials.gov/)

• metaRegister of Current Controlled Trials and International Standard Randomised Controlled Trial Number (ISRCTN) Register (www.controlled-trials.com/mrct/)

• WHO International Clinical Trials Registry Platform (http://apps.who.int/trialsearch/).

Details of the search strategies used can be found in Appendix 2.

Searching other resources

Backward citation searching was undertaken on key review articles. 1,15,66–70 As in all searches, the US FDA website was searched for the following terms: PCA3, phi and p2PSA.

Search results

The results of the searches undertaken are summarised in Figure 1. A total of 2249 unique records were identified by database searching and via the use of additional resources (e.g. trial registers and backward citation searching). Of these, 2021 records were excluded at the title and abstract screening stage. Overall, 228 studies were reviewed in full text and six papers were considered to be relevant for inclusion in the analytical validity review.

FIGURE 1.

Flow chart of search results. ARIF, Aggressive Research Intelligence Facility database; WoS, Web of Science.

Six papers48,71–75 reporting on analytical validity or pre-analytical effects were identified from the electronic databases: three related to the PCA3 assay48,71,72 and three related to the p2PSA assay. 73–75 In addition, the Summary of Safety and Effectiveness Data (SSED) report to the US FDA for each test50,58 was obtained. The manufacturers included a pack insert for the PCA3 assay48 in their submission and the draft pack insert for the p2PSA assay was obtained from the FDA website. 57

Data from two of the identified studies73,74 for the p2PSA assay appear to be described in the SSED report58 but, as citations were not stated, it was not possible to confirm this potential double use of data; the results from different data sources have been reported separately in the review and potential overlaps have been highlighted. The draft pack insert for p2PSA57 does not appear to include any data that have not already been described in the SSED report58 and so full data extraction using the draft pack insert57 has not been undertaken; however, relevant data are reported in the study characteristics table for completeness.

A search of electronic trial databases found two trials76,77 that were potentially relevant to the review of analytical validity of the PCA3 assay: one76 was ongoing and the status of the other was unclear (Table 4). No relevant ongoing trials on phi or p2PSA were identified. 77

PROGENSA prostate cancer antigen 3 assay

All of the PCA3 studies48,50,51,71,72 were conducted in the USA and reported data for clinical validity and analytical validity. All studies48,50,51,71,72 measured precision. Four studies48,50,51,71 measured accuracy, but only the SSED report50 and pack insert51 reported on five or more different outcomes that were all relevant to analytical validity. Pre-analytical effects were considered by all but one study. 72 None of the studies compared the PCA3 assay with a ‘gold standard’, as such a reference test does not exist for analytical validity. However, the PCA3 assay analyte quantitation was compared with in vitro transcripts which had been value-assigned using ultra violet spectroscopy in three studies. 48,50,71 Four of the studies50,51,71,72 provided adequate descriptions of the test under study, that is reported specific methods/platforms evaluated and information on quality assurance measures. Although all studies used clinical samples, it was unclear whether or not the same population was used for both the analytical validity and clinical validity studies. Only in one study72 did it appear that specimens represented routinely analysed clinical specimens in all aspects (e.g. collection, transport, processing). None of the studies provided sample size/power calculations, and the number of samples analysed varied by outcome both within and across studies (see Appendix 4, Table 71).

[–2]pro-prostate-specific antigen assay

Studies of p2PSA were conducted in Germany73,75 and in the USA;74 one study58 described results from studies that had been conducted in both of these countries. The manufacturers have confirmed that the assay that was research use only used in the study by Stephan et al. 75 is the same assay that is now commercially available. All studies reported analytical sensitivity, specificity, accuracy, precision, linearity and range. Pre-analytical effects were considered only in the SSED report to the FDA58 and by Semjonow et al. 73 None of the studies compared the p2PSA test to a ‘gold standard’ as such a reference test does not exist for analytical validity. However, in the SSED report58 recovery (a measure of accuracy) used internal reference preparation of p2PSA. Stephan et al. 75 was the only study that did not adequately describe the test under study, that is the study did not report the specific methods/platforms evaluated or present sufficient information on quality assurance measures. As with PCA3 studies, precise details about the population from which the samples were derived was not provided. The number of samples varied by outcome within and across studies (see Appendix 4, Table 71). None of the studies provided sample size/power calculations. In all instances, it was unclear if the specimens represented routinely analysed clinical specimens in all aspects (e.g. collection, transport, processing).

Impact of digital rectal examination

Sokoll et al. ,48 in a sample of 179 patients, found that 74.4% of urine samples taken before a DRE were informative, compared with 95.5% of urine samples that were taken after a DRE. First-morning void urine samples (n = 56) had an informative rate of 80.4%. The number of strokes per lobe in the DRE did not affect the informative rate of tests (98.7% for three strokes and 94.4% for eight strokes per lobe). There were no significant differences in the reported PCA3 score for those tests which were informative, regardless of whether or not the men had a prior DRE.

Storage of unprocessed urine samples

The SSED report50 and pack insert51 both described the effect of time spent at 30 °C and 2–8 °C on urine samples before processing into the transport tubes. The SSED report50 included 12 specimens and the pack insert51 reports on 10 specimens, and it is not clear whether or not these are the same samples. At 30 °C, the PCA3 score showed a 5% drift over 4 hours and at 2–8 °C a 2% drift over 4 hours. Estimates of drift are not presented for more than 4 hours’ storage. 50

Storage of processed urine samples

In Groskopf et al. ,71 three previously frozen processed urine specimens were thawed and held at 4 °C or 30 °C for 14 days. Degradation of mRNA was noted from day 1 at 30 °C; the PCA3 score remained within 20% of initial value for 14 days. The SSED50 reported drift for 12 processed samples held for 6 days at varying ambient conditions between 30 °C and –70 °C and between 30 °C and 55 °C; both temperature ranges had a drift of 8%. The pack insert50 reported that 12 specimens were stable for 21 days at 4 °C, for 5 days at 30 °C and for 90 days between –20 °C and –70 °C; no raw data were presented.

Groskopf et al. 71 and the pack insert51 reported stable results in processed urine after five and six freeze–thaw cycles, respectively; neither of the studies presented raw data.

Analytical sensitivity

Limit of blank (LoB), limit of detection (LoD) and limit of quantitation (LoQ) were reported as shown in Table 5. The LoQ of both analytes (PCA3 and PSA) were the same as the corresponding LoD in Sokoll et al. 48 and in the SSED report. 50

Analytical specificity

The assay did not detect unspliced PCA3 RNA. 50,51 No assay interference was recorded in the SSED report,50 with either 10 listed endogenous compounds or six micro-organisms; out of 27 exogenous compounds, only selenium and raw palmetto were reported to cause interference . 50 However, in the pack insert51 it was reported that none of the 35 therapeutic substances tested (which included selenium and raw palmetto) interfered with the assay. In addition, the SSED report50 states the effects of medications such as finasteride and dutasteride which are known to affect serum PSA levels were not evaluated. However, in the FDA pack insert51 (p. 34) but not in the SSED report, these two drugs are clearly listed among the therapeutic substances tested. Nevertheless, the pack insert51 states that ‘The PROGENSA PCA3 assay should not be used for patients who are taking medications known to affect serum PSA levels such as finasteride (Proscar, Propecia), dutasteride (Avodart), and anti-androgen therapy (Lupron). The effect of these medications on PCA3 gene expression has not yet been evaluated’ (p. 31). Urine samples from men after a prostatectomy and from female participants were below the assay limit for PCA3. 51,71 RNA from 10 tissue types throughout the male urogenital tract was tested, and only prostate tissue RNA was detected in the assay. 51 The SSED report50 included carryover studies with a 0% FP rate for negative samples interspersed with high-titre samples. 58

Accuracy

No gold standard is available, and without a gold standard that offers 100% specificity and 100% sensitivity it is difficult to confidently assess the accuracy of competing diagnostic strategies. Four studies48,50,51,71 assessed accuracy by calculating the percentage recovery of measured PCA3 or PSA RNA copies/ml compared with ultra violet-determined copies/ml of female urine samples spiked with varying concentration of in vitro transcripts or with control samples. Across four studies,48,50,51,71 accuracy varied from 90% to 118% for PCA3 and from 85% to 121% for PSA (Table 6).

Precision

Precision was assessed in four papers48,50,51,71 by including only within-laboratory variation (including intra-run and inter-run variance, reagent, observer) and in two papers50,71 by including both within- and between-laboratory variation. Multiple results were reported in some papers. 50,71 In six studies48,50,51,71 the within-laboratory total coefficient of variation (CV)% ranged from 4% to 27% for PCA3 and from 7% to 18.7% for PSA; in two studies50,51 the PCA3 score varied from 12% to 28% (Table 7).

Within- and between-laboratory total CV% was reported by two studies48,50 and ranged from 5.9% to 17.2% for PCA3 and from 10.1% to 19.3% for PSA (Table 8). Only one study50 reported within- and between-laboratory total CV% for the PCA3 score, which ranged from 12.3% to 25.0%. Most variation occurs within laboratory, with assays on different sites adding little extra variability.

Shappell et al. 72 reported between-laboratory precision from 50 clinical samples sent to two different laboratories in terms of concordance of PCA3 scores. When the PCA3 score was divided into three categories (indeterminate, < 35, ≥ 35), results were concordant in 47 out of 50 samples (94%). Correlation was reported to be good for 48 informative samples (r = 0.85). When three outliers were omitted, this improved further (r = 0.96). The mean percentage difference in test values was 13.6% [standard deviation (SD) 42.5%].

The SSED report50 also compared spiked female urine versus clinical samples (Table 9). Maximum variation in total CV% for PCA3, PSA and the PCA3 score appeared to be slightly less for clinical samples than for control samples. The sample precision in clinical samples was therefore reported to be comparable with that in control samples.

Linearity

The SSED report50 assessed linearity using 11 samples of PCA3 and PSA in in vitro transcripts in processed female urine. Here the deviation from linearity for PCA3 was < 9% and for PSA deviation was < 7%. Linearity studies using 10 clinical specimens in specimen diluent or processed female urine were also reported. Deviation from linearity for PCA3 in specimen diluent and processed female urine was less than 6%. However, for PSA in specimen diluent, linearity was < 23% and in processed female urine deviation was < 30%. The higher than expected variance in PSA, although remaining within study acceptance criteria, may have been caused by variation in linearity panel preparation. The pack insert51 reported a direct proportional relationship between dilutions tested and analyte copies/ml.

Stability of [–2]pro-prostate-specific antigen in blood and serum

Semjonow et al. 73 examined the stability of 22 clinical samples stored as clotted blood at 21 °C or as serum at 4 °C and 21 °C and then frozen at –20 °C or –70 °C. Percentage recovery of the samples over time from each baseline measurement was reported. The stability criterion used was that mean change in recovery did not exceed 10%.

In clotted blood, the mean recovery at 1 hour 9 minutes was 105.6% [95% confidence interval (CI) 103.2% to 107.9%], compared with 100% at the 37-minute baseline. At 3 hours 1 minute, the mean recovery was 112.7% (95% CI 109.7% to 115.6%). These data show that, by 3 hours after drawing the blood sample, the stability criterion is not met. No data are available for storage times between 1 hour 9 minutes and 3 hours 1 minute and so it is not clear precisely when the stability criterion is breached. These data are the basis for the recommendation stipulated in SSED,58 that is specimens should be spun and refrigerated within 3 hours. A regression equation extrapolated results to a baseline at 97% of time of specimen collection. The increase in value is considered to be because of proteolytic activity in the clot.

Samples in serum were within the stability criterion after 48 hours at either 4 °C or 21 °C and at least 12 months at –20 °C or –70 °C. Two freeze–thaw cycles did not result in < 10% variation compared with 21 °C.

Stability of reagents and calibration materials

The SSED report58 included data confirming stability of reagents and calibration products, both as sealed packs and once opened.

Thermal sensitivity of assay

The effect of change in ambient temperature (18 °C, 23 °C and 31 °C) on assay performance was investigated for three different analysers (Access 2, DxI 800 and DxI 600; Beckman Coulter Inc., Brea, CA, USA) and reported in the SSED. 58 Results were compared with results at the centre-point ambient temperature. A thermal effect was noted, with 1.84–2.82% change in p2pSA per 1 °C ambient temperature. This suggests a maximum of 16.9% variation in p2PSA result for ± 6 °C change in ambient temperature compared with temperature at which the calibration curve performed.

Analytical sensitivity

Limit of blank, LoD and LoQ were reported as shown in Table 10. The results reported in the SSED58 appear to be the same as the results reported in Sokoll et al. 74

Analytical specificity

Potential interference with seven endogenous compounds was investigated by comparing test mean (with added compound) and control mean (without added compound) for three different concentrations of p2PSA. 74 Most recoveries were within 10% of the target 100%, with a mean of 93%, although the addition of 8.4 g/dl total protein reduced one recovery to 88.4%. The same seven compounds at the same concentrations were also reported to be analysed for interference. 58 The raw data (mean recoveries) were not reported, although a warning was given that protein levels greater than 8 g/dl may interfere with p2PSA measurement. It is unclear if the analyses reported in the SEED report58 are the same as those reported by Sokoll et al. 74 Forty-nine commonly encountered medications and therapeutic drugs were also tested and the SEED report58 concluded that they did not interfere with assay performance, although no raw data were presented.

Crossreactivity with other PSA isoforms, including α₁-antichymotrypsin-PSA, benign PSA, fPSA, (–4) PSA and (–5/–7) PSA, was tested in three studies. 58,74,75 Minimal cross-reactivity was detected (recovered test dose < 5%58,74 or < 2.5%75 of expected dose).

Carryover was reported by only one study58 with no evidence of carryover from high concentration samples.

Accuracy

No gold standard is available and the reference material used is based on purified p2PSA. Accuracy was reported in three studies58,74,75 by calculating the per cent recovery of measured p2PSA pg/ml in male serum samples spiked with varying concentration of purified p2PSA (Table 11). The data reported in SSED58 and Sokoll et al. 74 appear to be from the same study.

Precision

Precision was assessed by including only within-laboratory variation (including intra-run and inter-run variance, reagent, observer) in three studies58,74,75 and by including both within- and between-laboratory variation in one study. 58

All studies reported CV% for p2PSA, but only the SSED report58 included CV% for phi. Within-laboratory precision as measured by total CV% varied from 2.91% to 13.05% for p2PSA and from 8.5% to 12.0% for phi (Table 12). Within- and between-laboratory precision for p2PSA was reported as being between 5.39% and 9.39% for p2PSA and between 4.9% and 7.3% for phi (Table 13). These maximum estimates are lower than those for within-laboratory only precision, and it is likely that this reflects the different populations used. There appears to be an overlap in data for p2PSA variability between Sokoll et al. 74 and the SSED report. 58 Sokoll et al. 74 reported within-laboratory precision data from four sites, but one of these had higher than expected variability and no total variance across all was reported in this paper (see Table 12). The variance for p2PSA across three sites reported in the SSED report58 may be from the three lower variance sites (see Table 13).

Linearity

The SSED58 reported linearity studies using diluted known concentrations of p2PSA in 12 serum samples. Eleven out of the 12 samples had a slope of 1.0 ± 0.15. A linear range was confirmed to 4922 pg/ml. Sokoll et al. 74 assessed dilutions of three samples and Stephan et al. 75 assessed six samples; both confirmed a linear range to 5000 pg/ml and 4500 pg/ml, respectively. No hook effect to 15,000 pg/ml was found in the two studies. 58,74

[–2]pro-prostate-specific antigen/Prostate Health Index assay

Practical issues relating to the use of the p2PSA assay that may be important to consider are sample handling and thermal sensitivity. The draft pack insert57 states that blood should be centrifuged and serum separated within 3 hours of the blood sample being taken; this guidance is based on the work of Semjonow et al. 73 However, the data in this paper suggest that by 3 hours 95% of samples will have breached the stability criterion of a 10% increase in p2PSA level. As neither the manufacturer nor Semjonow et al. 73 present a rationale for the use of the 10% stability criterion or a time period of 3 hours, the consequences of breaching the 3-hour time period are not clear. In addition, whether or not sample handing can be carried out in routine clinical practice as per the instructions set out in the draft pack insert57 is not yet known; in particular, given the 3-hour time limit, only hospitals with on-site laboratory facilities may be able to offer this test.

Studies of the thermal sensitivity of the p2PSA assay indicated that there is a 16.9% variation in p2PSA result for a ± 6 °C change in ambient laboratory temperature. 58 However, the SSED58 report suggests that any differences in results because of temperature change would not affect clinical validity results.

Within-study (direct) comparisons

Owing to uncertainty about the diagnostic pathways used in NHS clinical practice and the limited availability of MRI facilities, the EAG initially included all studies with a direct comparison of the PCA3 assay or the phi with any one or more of following component comparator tests:

• individual clinical risk factors such as age, a DRE

• standard clinical judgement/nomograms

• PSA levels

• MRI results: T2-weighted magnetic resonance imaging (T2-MRI)/DW-MRI.

As the intervention tests (PCA3 assay or phi) can be used as replacement, add-on or triage tests to the comparator tests, studies that have directly compared the clinical validity of the PCA3 assay with the clinical validity of phi, with or without other comparators, were also included. The inclusion criteria used to select eligible within-study comparisons are presented in Table 14.

Systematic reviews for use in between-study (indirect) comparisons

In the absence of any available within-study comparisons, the EAG would have considered carrying out between-study (indirect) comparisons of the intervention tests versus comparator tests. Given the probable large number of studies evaluating each of the intervention and comparator tests, estimates of the clinical validity of the intervention and comparator tests from good-quality systematic reviews with meta-analyses were sought to provide data for any between-study (indirect) comparisons undertaken. The inclusion criteria used to select eligible systematic reviews are presented in Table 14.

Within-study (direct) comparisons

Limited data [e.g. details relating to the comparator interventions, study population (including inclusion and exclusion criteria) and author conclusions] were extracted from studies that were eligible for inclusion but did not report data from a clinically relevant comparison, that is limited data were extracted from studies that reported the results of univariate PCA3 or univariate phi versus univariate PSA.

Complete data were extracted from all other eligible studies. Particular attention was paid to:

• how the intervention and comparator tests were used (replacement, add-on, triage or not stated)

• definition of positive biopsy, including grade and stage of tumour detected

• threshold values used for intervention tests.

The available data on all reported clinical validity outcomes were recorded including:

• 2 × 2 tables of true positive (TP), FP, false negative (FN) and TN values

• sensitivity, specificity, positive predictive value, negative predictive value, positive and negative likelihood ratios

• area under the curve (AUC) and sensitivity and specificity values derived from receiver operator characteristics (ROC) curves

• multivariate odds ratios (ORs) for logistic regression.

Outcomes were recorded for every reported:

• threshold value

• combinations or sequence of tests

• grade of cancer.

Systematic reviews for use in between-study comparisons

Study extraction was limited to:

• details relating to the comparator interventions

• study population (including inclusion and exclusion criteria)

• number of studies and participants included in meta-analyses

• author conclusions.

Clinical trials search results

A search of electronic trial databases found one ongoing randomised trial84 that was possibly relevant for the clinical validity review of the PCA3 assay; summary details of this trial are shown in Table 15. No relevant ongoing trials that included phi were identified.

Excluded studies

The studies excluded from the review at stage 2 (including any listed in the manufacturer submissions which were not eligible for inclusion in the review) are listed in Appendix 5 with the reasons for their exclusion. This list contains both excluded primary studies and excluded systematic reviews and meta-analysis papers. The most frequent reason for exclusion was ineligible or unclear study population.

Intervention pathways

Three intervention pathways are of interest to this review; these pathways are repeated here from Box 2.

Of the three intervention pathways described, data are available from the included studies to address only the first pathway. As the results of tests are most often presented as outputs from logistic regression models, it is not possible to determine from the data available whether or not carrying out the diagnostic tests in one order is better than carrying out the tests in a different order. Nor is it possible to determine whether diagnostic accuracy is improved if the PCA3 assay (or phi) test is carried out before or after a MRI. Therefore, there are no included studies which explicitly address the second or third pathways.

Study designs and populations

Fourteen studies45,46,85,86,89–92,94–97,99,102–106 were observational cohort studies and one was a randomised controlled trial (RCT). 103 Eleven studies were of a prospective cohort design,45,46,85,86,89,90,92,94,97,99,102,103 three studies91,95,96,106 were of a retrospective cohort design and one study104 was of mixed design. None of the studies was based in the UK; the relevance of the included study results to UK clinical practice is, therefore, uncertain.

In all but one trial, the study population was made up of men who had been referred for repeat prostatic biopsy for clinical indications; the exception was the REDUCE study,86,105 in which men were participants in the placebo arm of a clinical trial.

The criteria for referral for repeat biopsy were often unclear and differed across studies. Some studies were restricted to men with normal89,94,103 or abnormal DREs. 96 The terms ‘positive DRE’ and ‘negative DRE’ were often used, and we assumed that ‘positive’ meant abnormal and ‘negative’ meant normal. The percentage of men with abnormal DRE scores in the studies therefore varied from 0% to 100%. 47,96 The mean (or median) age of study populations was between 6094 and 67 years. 45,89,102 Mean or median PSA, when stated, ranged from 4.891 to 11.0 ng/ml. 106 Seven studies recruited only men with PSA scores within the grey zone of PSA (PSA levels between 4 and 10 ng/ml). 47,86,90,94,96,97,105,107 The prevalence of cancer detected on repeat biopsy varied from 11.4%76 to 68.3%. 94

Recruitment to the placebo arm of the REDUCE trial86,105 did not rely on referral for repeat biopsy. Men aged 50–75 years were recruited to the main REDUCE trial on the basis of increased PSA levels (2.5–10 ng/ml) and a negative initial biopsy. 86,105 Participants were then scheduled to receive a repeat biopsy at 2 and 4 years regardless of clinical indications. A subsample of these men, based on whether or not the trial centre was able to process urine samples for PCA3 testing, were included in this study. 86 However, this study excluded all biopsy results that were indicated by abnormal clinical assessment, such as rising PSA or an abnormal DRE, and used only the results from the biopsies that were mandated by the trial protocol. This study population is, in effect, the reverse selection of the clinically selected population seen in most studies and represents a low-risk population.

The details of the reference standards used in studies were poorly reported. An added complication was the fact that the number of biopsy cores taken often differed across patients within a study. Among cases for which details were provided, 10- or 12-core biopsies were the most common. Three studies96,99,102 exploring the efficacy of the PCA3 score and all four studies92,99,102,104 exploring the efficacy of phi used saturation biopsies or reported that the mean or median number of cores taken was 20 or more. The repeat biopsy was usually performed transrectally under ultrasonography guidance, taking 10–20 cores. Two studies described the repeat biopsy as saturation biopsies,96,102 and in one of these the biopsy route was transperineal96 and in the other transrectal. 102

Patient selection

Risk of bias from patient selection was assessed as being unclear for 10 studies,45,46,85,86,90–92,97,99,102,105,106 as the type of clinical pre-selection operating was not explicit, and, therefore, it was impossible to assess how this might have biased the assessment of the clinical variables within the diagnostic models. Four studies89,94,96,103 were assessed as having a high risk of bias because of pre-selection on a DRE or other clinical variables, and these studies89,94,96,103 also had high concerns regarding applicability of patient selection. The study by Stephan et al. 104 was assessed as having a high risk of bias from patient selection as, in this mixed prospective and retrospective study, 29% of patients were excluded from the analyses because it was unclear whether the biopsy was initial or repeat.

Intervention tests

The majority of studies provided no details of whether the intervention tests (PCA3 assay or phi) were conducted with or without knowledge of other important considerations, for example results of comparator tests or the reference standard. Studies that did record this information included those by Gittelman et al. 45 and Goode et al. ,91 the REDUCE trial placebo arm86,105 and the study by Stephan et al. 104 However, given the objective nature of the intervention tests, all studies were assessed as having a low risk of bias. As all study authors reported using the PROGENSA PCA3 assay (or, in the case of four studies,85,86,105,106 this was confirmed by the manufacturer) and/or the Beckman Coulter assay systems, we had little concern regarding the applicability of the intervention tests.

Comparison tests

The risk of bias and concerns regarding applicability arising from the comparison tests were considered separately for clinical assessment, including PSA levels, and for MRI. Clinical assessment, including PSA levels, was often poorly described with no criteria given for an abnormal DRE. It was often not clear when the data used for clinical assessment had been recorded in relation to the timing of the biopsy or the intervention tests and if these had been collected or recorded by the analyst without knowledge of intervention tests or reference standard results. In multicentre studies there was no description of how clinical variables were standardised across centres. All studies were assessed as having an unclear risk of bias from the clinical comparator tests.

Concern regarding applicability of clinical assessment was assessed on the variables used in the multivariate models, algorithm or nomogram. Studies which pre-selected patients who had undergone DREs89,94,96,103 (which meant that a DRE was not included in the model) were marked as being of high concern, as this does not reflect clinical assessment in routine practice. Most studies, including studies of phi, included standard PSA measures such as tPSA or fPSA in the clinical comparator model and added p2PSA to the intervention test model. However, Porpiglia et al. 99 and Goode et al. 91 did not include PSA in the clinical comparator model, and so we had a high degree of concern about the applicability of these study results to UK clinical practice.

Four studies90,94,99,103 which compared the PCA3 assay or phi with MRI were all assessed as having a low risk of bias and concerns regarding applicability of the MRI were low. MRI was either performed before repeat biopsy90,94,103 or the radiologist was blinded to the biopsy result. 99 Diagnostic criteria were described in Porpiglia et al. 99 and Panebianco et al. 94 mpMRI with T2-imaging, DW imaging and DCE-MRI were performed in Porpiglia et al. 99 and mpMRI with T2-imaging, MRS, DW imaging and DCE-MRI were performed in the other studies. 90,94,103

Reference standard

The reference standard involves two procedures, both of which are prone to bias: the targeting and selection of the biopsy cores and the pathology reporting of the cores taken. In many studies,45,46,85,86,91,97,105,106 the reporting of the details of type and pattern of repeat biopsy performed was poor. In seven studies,46,85,89,96,97,102,104,106 the number of cores taken varied, indicating that the number and locations of cores taken were affected by clinical findings such as a DRE or TRUS. In two studies,92,97 although the methods specified a set number of cores to be taken, the number of cores actually taken was reported as a range. In addition to variation in the number and site of biopsy cores taken, pathology reporting is a potential source of bias. In three multicentre studies,45,46,85,104 pathology reporting was not centralised, with potential for differences in biopsy processing protocols and pathology reporting. By contrast, in the REDUCE study,86,105 all cores were processed at a single central laboratory. Four studies indicated that the pathologists were blinded to the clinical status of the patient,45,91,92,99 and one study reported blinding to MRI. 103 Owing to these uncertainties, eight studies45,46,85,86,89,96,97,102,104,105 were assessed as having an unclear risk of bias. If it was clear that additional cores were taken because of abnormalities identified on MRI90,94,103 or TRUS,106 the study was assessed as having a high risk of bias. The study by Porpiglia et al. 99 had a low risk of bias, as it was stated that the biopsies taken were not affected by MRI results or biomarker results and that a constant number of cores (depending on prostate volume) and pattern were taken.

In all studies, the applicability of the reference standard used was an area of high concern. Although the TRUS prostate biopsy is the usual method of diagnosing prostate cancer, this type of biopsy is inaccurate and often misses cancers.

Five studies had funding from, or financial links to, companies which produced the assays. 45,46,85,86,102,104,105 In another study, the manufacturer had supplied reagents. 92

Overall, the results of the quality assessment exercise revealed that none of the studies was free from the risk of bias. The main areas for concern were related to the applicability of the study populations, variation in clinical assessment and whether or not choice and use of the reference standard were linked to previous clinical results. None of the studies was carried out in the UK and so the results of the studies were not directly relevant to NHS decision-makers.

Comparison 1: clinical assessment versus clinical assessment + PROGENSA prostate cancer antigen 3 assay

Comparison 2: clinical assessment versus clinical assessment + Prostate Health Index

Comparison 3: clinical assessment + magnetic resonance imaging versus clinical assessment + magnetic resonance imaging + PROGENSA prostate cancer antigen 3 assay

Comparison 4: clinical assessment + magnetic resonance imaging versus clinical assessment + magnetic resonance imaging + Prostate Health Index

Comparison 5: clinical assessment + PROGENSA prostate cancer antigen 3 assay versus clinical assessment + Prostate Health Index

Comparison 6: clinical assessment + magnetic resonance imaging + PROGENSA prostate cancer antigen 3 assay versus clinical assessment + magnetic resonance imaging + Prostate Health Index

Comparison 7: clinical assessment versus clinical assessment + PROGENSA prostate cancer antigen 3 assay + Prostate Health Index

Comparison 8: clinical assessment + PROGENSA prostate cancer antigen 3 assay versus clinical assessment + magnetic resonance imaging

Comparison 9: clinical assessment + PROGENSA prostate cancer antigen 3 assay versus clinical assessment + magnetic resonance imaging + PROGENSA prostate cancer antigen 3 assay

Comparison 10: clinical assessment + Prostate Health Index versus clinical assessment + magnetic resonance imaging

Generalisability of study population

The clinical validity review addressed the potential use of the PCA3 assay and the phi, in combination with clinical assessment, to assess the need for a second biopsy in men suspected of having prostate cancer whose initial biopsy result was negative or equivocal. This population can be considered to be made up of three different groups of men: first, men who have signs which are strongly suggestive of prostate cancer (such as abnormal DRE findings and/or abnormal histopathology) and who would be referred for second biopsy by most, if not all, clinicians; second, men who no longer display signs of prostate cancer, for example men whose PSA levels have returned to normal and who have no other evident risk factors – many clinicians would not refer this group of men for a second biopsy; and, third, men who fall between these two positions, that is men who have some signs of prostate cancer. In the case of this last group of men, referral to second biopsy may vary between clinicians. The use of PCA3 scores and phi may contribute to the diagnostic process in all three of these populations by providing clinicians with an additional source of clinical information which they can consider before the decision regarding referral to a second biopsy is made.

All but two78,79 of the populations described in the included studies comprise men who were referred for a second biopsy because, following a negative initial biopsy result, clinicians still suspected that malignant prostate cancer was present. In these cases, the effect of adding the PCA3 assay or phi to clinical assessment was tested in populations of men whose clinician had already made the decision that a second biopsy was necessary. The study entry criteria differ across the included studies, suggesting that the disease characteristics of the study populations may be heterogeneous. Therefore, some of these study populations include patients for whom the reason for referral for a second biopsy is clear, while in other studies the reason is unclear.

The two papers78,79 in which the populations had not been referred for a second biopsy report diagnostic accuracy outcomes for participants in the placebo arm of the REDUCE trial. Participants in this trial constitute a low-risk population of men who do not exhibit any clinical signs to suggest that a second biopsy might be appropriate.

These differences in patient selection criteria mean that it may not make clinical sense to apply the results of this review, without clearly stated caveats, to all men with negative or equivocal biopsy results. Furthermore, none of the included studies was conducted in the UK.

Variation in clinical assessment

An additional issue that makes it difficult to draw firm conclusions from the data is that the representation of clinical assessment varies in the included studies. Although clinical assessment is not standardised in practice, it is difficult to meaningfully compare the results of studies which have markedly different representations of clinical assessment. For example, it may be inappropriate to compare the results of clinical assessment (a DRE, age) + phi versus clinical assessment (a DRE, age, family history, prostate volume, ethnicity and PSA level) + PCA3.

Reference standard

How the reference standard was used was unclearly reported in some of the studies, and this is an indication of a poor-quality study. As the choice and conduct of the reference standard may be affected by the results of the comparator or intervention test, studies that stated explicitly that the MRI results influenced the choice of reference standard were assessed as having a high risk of bias. Finally, the reference standard (prostate biopsy) is an imperfect diagnostic tool as it does not detect all cancers. Without a gold standard that offers 100% specificity and 100% sensitivity, it is difficult to confidently assess the accuracy of competing diagnostic strategies. The applicability of all of the included studies was, therefore, assessed as being of high concern and so the quality of the studies was inevitably low.

Addition of PROGENSA prostate cancer antigen 3 assay score to clinical assessment

Study findings varied depending on the outcome metric used in the analysis.

Eight45,46,85,86,90,99,102 efficacy comparisons of clinical assessment versus clinical assessment + PCA3 using AUC data demonstrated that the addition of PCA3 score to clinical assessment led to improvements (1–19%) in diagnostic accuracy. Two studies45,86 used the PCA3 score as a dichotomous variable using thresholds of 25 and 35; the remainder used PCA3 score as a continuous variable. The representation of clinical assessment varied across the studies.

Seven45,86,89,99,106 efficacy comparisons using multivariate ORs also showed that the addition of PCA3 score to clinical assessment increased diagnostic accuracy compared with clinical assessment alone; six of the seven ORs for PCA3 score were statistically significant and one OR was borderline. Two studies86,106 reported ORs for unit increase in PCA3 score, four studies45,86,89 used the PCA3 score as a dichotomous variable (25, 35, 39 and 50), and in one study99 the threshold used was unclear. The representation of clinical assessment varied across the studies.

Diagnostic performance was assessed in terms of sensitivity and specificity in only one study. 105 The study by Tombal et al. 105 showed that the addition of PCA3 score to clinical assessment led to a small decrease in sensitivity (from 75% to 66%) but led to a marked increase in specificity (from 26% to 71%).

Studies45,99,102 which fixed sensitivity at 80% or 90% and derived specificity from logistic regression models also reported mixed results. The results from studies90,99,102 that assessed efficacy using decision curve analyses were also mixed, with no clear benefit associated with adding PCA3 score to clinical assessment; increased net benefit was shown in two studies90,99 when the risk threshold was set at 25%. The implications of adding the PCA3 score to clinical assessment are not clear and it is not possible to identify a single-threshold value for use in a clinical setting.

Addition of Prostate Health Index to clinical assessment

Four studies92,99,102,104 compared clinical assessment versus clinical assessment plus phi, and demonstrated higher AUC estimates (2–6%) when phi was included. All of the studies used the phi result as a continuous variable. The representation of clinical assessment varied across studies. Two studies92,99 reported multivariate ORs for clinical assessment + phi and the results indicated that the addition of phi to clinical assessment led to a small improvement in diagnostic accuracy (OR > 1). No studies reported sensitivity and/or specificity and the studies92,99,102 reporting results for derived sensitivity and specificity or decision curve analysis have conflicting results. The implications of adding phi to clinical assessment are not clear and it is not possible to identify threshold values for use in a clinical setting.

The addition of PCA3 score to clinical assessment + MRI to the diagnostic process did not have a noticeable impact on discrimination.

Addition of PROGENSA prostate cancer antigen 3 assay score to clinical assessment + magnetic resonance imaging

Two studies90,99 assessed the incremental gain in diagnostic accuracy resulting from adding PCA3 score to clinical assessment + MRI using AUC estimates; the addition of PCA3 score in both studies had negligible impact. Both studies used PCA3 score as a continuous variable. The study by Busetto et al. 90 employed a MRI-targeted biopsy, whereas the study by Porpiglia et al. 99 did not. The OR for PCA3 score when added to clinical assessment + MRI was not statistically significant.

No studies reported sensitivity or specificity. Porpiglia et al. 99 reported results at set levels of specificity and showed that adding PCA3 score had minimal effect on derived sensitivity; Porpiglia et al. 99 also demonstrated that adding PCA3 score to clinical assessment + MRI at set levels of sensitivity had minimal effect on derived specificity (–5.9% to 0.8%). In these two studies,90,99 the results of decision curve analyses showed that the addition of the PCA3 score to clinical assessment + MRI did not improve diagnostic accuracy when added to clinical assessment + MRI at threshold probabilities between 10% and 50%.

Addition of Prostate Health Index to clinical assessment + magnetic resonance imaging

Only the study by Porpiglia et al. 99 assessed the gain associated with adding phi to clinical assessment + MRI. Adding phi to clinical assessment + MRI had no effect on AUC. The OR for phi when added to clinical assessment + MRI was not statistically significant. At set levels of sensitivity and specificity, the addition of phi had a minor effect on derived specificity and sensitivity. In this study,90,99 the results of decision curve analyses showed that the addition of phi to clinical assessment + MRI did not improve diagnostic accuracy at threshold probabilities between 10% and 60%.

The addition of phi to clinical assessment + MRI to the diagnostic process did not have a noticeable impact on discrimination.

Systematic reviews

None of the systematic reviews identified for inclusion in the clinical validity review included comparisons that assessed the addition of the PCA3 assay or phi to clinical assessment with or without MRI.

Modelling effectiveness

A number of different measures are used by researchers to show the relative efficacy of different diagnostic strategies being considered in this assessment (including sensitivity, specificity, AUC, multivariate ORs and decision curve analyses results). Of these measures, those that are the most readily useable in an economic model are sensitivity and specificity. This is because, in combination, these metrics allow a simple comparison to be made of the number of cancers that are correctly identified and the number of unnecessary biopsies that are undertaken when using competing diagnostic strategies.

The differences in benefits and costs arising from the diagnostic strategies can, therefore, be separated into the benefits and costs arising from differences in:

• undetected, untreated cancers (the higher the sensitivity, the higher the rate of detected cancer)

• unnecessary repeat biopsies for patients without cancer (the higher the specificity, the lower the rate of unnecessary biopsies for those without cancer).

Only one of the studies included in the review of clinical validity reported sensitivity and specificity estimates; it was more common for the studies to report derived sensitivity and derived specificity values for a range of different intervention and comparator diagnostic strategies. It was, therefore, not possible for the EAG to undertake a meta-analysis of sensitivity and specificity across trials as the data were unavailable, as explained in Chapter 2, Within-study comparisons: additional data analyses.

In the review of clinical validity, the included studies present sensitivity and specificity results either for specific test thresholds or, more often, as estimates that are derived from logistic regression models.

The EAG’s de novo economic model uses the derived specificities for stated sensitivity levels. The questions that this approach address are ‘Given a desired cancer detection rate for the target population, what proportion of the population would need a second biopsy?’ and ‘What proportion of these second biopsies would be unnecessary?’.

The EAG acknowledges that this approach to modelling may not precisely reflect clinical practice in the NHS in England and Wales. As stated in Appendix 1, derived sensitivity or derived specificity estimates are calculated from ROC curves, and it is often not possible to associate a stated sensitivity/specificity combination with a particular threshold of the intervention test.

As sensitivity and specificity are required to populate the model, the EAG also considered using sensitivity values for the tests that were based on the estimated cancer rates associated with the different threshold levels recommended by the manufacturers. However, to translate these values into estimates of sensitivity and specificity, several other pieces of information would be required:

• the proportion of patients that would be at each threshold

• how clinical assessment of other patient information (PSA level, DRE findings, etc.) would influence whether or not a biopsy would be recommended at different thresholds

• the proportion of patients who, on being recommended to receive a second biopsy, choose to receive one.

As this information is not readily available, the values for the above would have had to be assumed to generate sensitivity–specificity combinations. The EAG considered that such an approach would generate considerable uncertainty and that a more robust approach would be to focus on the available evidence on derived sensitivity–specificity combinations that is underpinned by findings from clinical studies, if not from clinical practice.

The use of derived specificity at stated sensitivity levels allows a fair comparison to be made between different testing strategies. Using this approach, the percentage of cancers that are detected is always the same regardless of the strategy chosen, but the number of biopsies required to detect these cancers differs. This simplifies the decision problem, negating issues such as which test threshold values to use in the model and how test results interplay with patient and clinician risk preferences.

As the percentage of detected underlying cancers is the same for all diagnostic strategies in the EAG model, the proportion of patients with treated and untreated cancers is also the same for all diagnostic strategies. Consequently, patient benefits and costs from cancer detection and treatment are the same for all diagnostic strategies. Therefore, as specificity levels for a given level of sensitivity differ across the comparator diagnostic strategies, the differences in patients’ benefits and costs between strategies are driven only by the difference in unnecessary biopsies carried out on patients without cancer. Although there is some evidence that biopsies may be linked to increased mortality in the short term, this is as yet unproven. 117 The EAG model, therefore, only considers the short-term impact of a biopsy on QoL and associated complications.

Time horizon

The NICE reference case118 states that the time horizon of economic models should be:

Long enough to reflect all important differences in costs or outcomes between the technologies being compared.

p. 57118

In the EAG economic assessment, the differences in costs and outcome are limited to:

• the differences in costs and complication-related outcomes from the additional biopsies indicated by the testing strategies

• the costs and outcomes of any monitoring of patients who are either indicated as negative for cancer by the testing strategy or who have a negative repeat biopsy.

As a PSA monitoring strategy can run for several years, the time horizon of the model is limited to the time that patients spend within any such strategy. The monitoring strategy is independent of the diagnostic strategies assessed in the model, so unless there is a lifetime PSA monitoring strategy the model does not require a lifetime horizon. In the base case, the PSA monitoring strategy runs for 3 years so the time horizon is also 3 years. The time horizons for the scenario analyses exploring the impact of the PSA monitoring strategies used in the CG17511 MRI model and the Mowatt et al. 14 model are 6 years and 1 year, respectively.

There is currently no unequivocal evidence that the biopsy procedure increases mortality. In the EAG model, the proportion of cancers identified and treated is assumed to be identical regardless of testing strategy. Thus, overall mortality rates will also be identical across testing strategies and so were not included in the model. Mortality could influence costs during the monitoring phase, but Bill-Axelson et al. ,119 who collected data on 348 patients with localised cancer being monitored by watchful waiting, found very low mortality rates over 3 years (under 5%). As almost all of the population in the EAG model who enter PSA monitoring do not have prostate cancer, and those who do are picked up and treated in a relatively short period of time, mortality rates would be even lower for patients in the EAG model than the levels reported by Bill-Axelson et al. 119 Given this, the impact on cost from introducing mortality into the model would be negligible and so has been excluded.

In addition, the following two scenario analyses were undertaken by the EAG:

• the monitoring and second biopsy strategy used in the CG17511 MRI model

• the monitoring strategy used in the Mowatt et al. model. 14

These two scenarios can be considered to represent the ‘least costly’ (Mowatt et al. 14) and ‘most costly’ (CG175 MRI model11) PSA monitoring scenarios.

A sensitivity analysis which involved varying the percentage of patients without cancer who had persistently elevated PSA levels, and so would require re-biopsy while under PSA monitoring, was considered. However, the Mowatt et al. 14 scenario considers the case that no men without cancer continued to have an elevated PSA level and the CG17511 MRI model scenario considered the case where all men without cancer continued to have an elevated PSA level. These two scenarios represent the two extremes of the percentage of men with persistently elevated PSA levels. Thus, the EAG felt that the inclusion of a sensitivity analysis varying the percentage of patients without cancer who had a persistently elevated PSA level would be uninformative and so such an analysis was not undertaken.

Clinical effectiveness

The clinical effectiveness estimates for different diagnostic testing strategies have been taken from the available published clinical evidence (see Chapter 2). As it has not been possible to carry out between-trial analysis and pool effectiveness data, the data in each study have been considered independently. Three studies provide information on derived specificity at differing sensitivity levels: Porpiglia et al. ,99 Scattoni et al. 102 and Gittelman et al. 45 Of these, Porpiglia et al. 99 provide data for all of the diagnostic testing strategies considered by the other two studies102,45 plus additional strategies that include mpMRI. Therefore, results from Porpiglia et al. 99 have been used in the base case, while data from the other two studies102,45 have been used in scenario analyses to explore the effect that different levels of effectiveness (and elements of clinical assessment) might have on conclusions.

Clinical validity data reported by Tombal et al. 105 were considered for incorporation into the model. However, while sensitivity and specificity values are reported they are only reported for a specific PCA3 threshold value. Reported results, therefore, do not allow comparisons of specificity rates (at stated sensitivity levels) for the PCA3 assay against alternative strategies and so have not been used in the model.

Clinical advice to the EAG is that it is very difficult to pinpoint a precise sensitivity estimate that most clinicians use in clinical practice. Furthermore, clinical decisions regarding biopsy referral are made with sensitivity implicitly in mind but not explicitly stated. Choosing a sensitivity level to use in the EAG base case was necessarily arbitrary; 90% sensitivity was chosen as it is the middle estimate of the three levels of sensitivity data that were provided in the study reported by Porpiglia et al. 99 The impact of using sensitivity levels of 80% and 95% was explored in scenario analyses. Only the Gittelman et al. 45 paper included data on the variance, or range, of the derived specificity estimates and, therefore, it has not been possible to vary these values in the probabilistic sensitivity analysis.

Studies reporting clinical validity data use a biopsy as the reference standard despite biopsies not being 100% sensitive or specific. To check the impact that this assumption has on model findings, sensitivity analyses were undertaken in which the proportion of cancers detected at biopsy were set at 50% and 100%.

As stated in CG175,11 mpMRI-targeted biopsy has greater sensitivity and specificity than TRUS biopsy alone. However, in the Porpiglia et al. study,99 which includes data on the sensitivity and specificity of mpMRI in combination with PCA3 assay or phi, the urologists were blinded to the mpMRI results before a biopsy was taken. Although mpMRI can influence biopsy sensitivity and specificity, the EAG has assumed that a biopsy after mpMRI does not influence the final diagnostic accuracy of the second biopsy. This assumption will put downwards pressure on the efficacy of mpMRI, but the decision question is ultimately about the addition of PCA3 assay or phi with or without MRI. Thus, this assumption will influence only the comparison of mpMRI with strategies without mpMRI, biasing the findings against mpMRI.

The sensitivity/derived specificity values used for the different diagnostic strategies are presented in Table 31.

Biopsy complications

The CG17511 MRI model provides a detailed description of biopsy complication rates identified by a literature review. As CG17511 was published in the same year as the EAG model was constructed, it was postulated that it might be appropriate to use the complication rates used in CG17511 in the EAG model. Citation searches were carried on the relevant studies. 120,121 These searches failed to identify more up-to-date rates and so the complication rates used in the CG17511 MRI model were also used in the EAG model.

Biopsy complication rates are shown in Table 32. The costs associated with biopsy complications that are included in the model should be considered as conservative, as literature searches failed to identify any published studies reporting the costs associated with sepsis or antibiotic resistance. To establish whether or not this omission could affect findings, a sensitivity analysis was undertaken in which all complication costs were increased by 100%.

Values for the upper and lower CIs have been used to model pessimistic and optimistic resource use scenarios.

Cost year

Unless otherwise stated, the costs are in 2014 GBP.

Cost of clinical assessment

The diagnostic strategies included in the EAG model comprise one or more of four separate components: clinical assessment, phi, PCA3 assay and mpMRI. While the nature of clinical assessment varies between studies, within studies it is the same for all participants. As the model does not pool data but looks at evidence from studies individually, and clinical assessment is required in all diagnostic strategies, there is no requirement to model the cost of clinical assessment, as it will make no difference to costs between strategies.

Cost of PROGENSA prostate cancer antigen 3 assay

The PCA3 assay costs provided by the manufacturer have been calculated by applying UK costs to resource use obtained from a US study. (Commercial-in-confidence information has been removed.) The estimated cost of the PCA3 testing kit was given as £164.67 including value-added tax (VAT) and (commercial-in-confidence information has been removed). This higher cost of £175.11 has been used in a scenario analysis.

The cost of the PCA3 assay has not been varied in the probabilistic sensitivity analysis.

Cost of Prostate Health Index

The manufacturer provided the cost of a single phi test. This was £89.83 including VAT. (Commercial-in-confidence information has been removed.) Deterministic sensitivity analysis has been used to explore the impact of the number of tests being 50% lower and 50% higher than this figure, effectively changing the cost of the test by ± 50%.

With no evidence available on the distribution of tests conducted in a year, the cost of phi testing has not been varied in the probabilistic sensitivity analysis.

Cost of multiparametric magnetic resonance imaging

The CG17511 report provided detailed costings of mpMRI and these costings were used in the EAG model, updated where necessary with up-to-date unit or NHS Reference Costs (2012/13). 18

The unit costs for staff time and equipment costs used in the CG17511 MRI model were taken directly from Mowatt et al. 14 However, staffing costs, which were based on bottom-up calculations of staff time, were increased in the CG17511 MRI model, as the NICE Guidelines Development Group considered that they were an underestimate. Resource use and costs of mpMRI are provided in Table 33.

As no measures of dispersion were available on the cost per hour, or time, per patient no sensitivity analyses were undertaken around this cost.

Costs of biopsy

Biopsy costs are dependent on the type of biopsy undertaken. In the base case, the EAG has assumed that the second biopsy will be a TRUS biopsy carried out as an outpatient appointment.

The assumption used in the CG17511 MRI model is that mpMRI results influence whether a TRUS or a transperineal biopsy is performed. Incorporating this assumption into the EAG model is difficult as it contradicts the model assumption that mpMRI does not influence the nature, or accuracy, of the second biopsy, an assumption made because the evidence99 from which efficacy data were drawn explicitly blinded clinicians performing the biopsy to the mpMRI results. Therefore, while in the base case the model assumption is that all patients have a TRUS biopsy as an outpatient procedure, a scenario analysis has explored the impact on results of a situation in which 50% of second biopsies are transperineal biopsies carried out as day-case procedures.

The CG17511 MRI model uses NHS reference costs as the basis for costing the different biopsy procedures. The clinical experts advising on the CG17511 model considered that NHS reference costs did not take adequate account of pathology costs and, therefore, that they underestimate the true cost of biopsy. The developers of the CG17511 MRI model therefore increased the cost of the HRG for histopathology by adding an estimate provided by a NHS Pathology Department in Bristol. A saturation biopsy was assumed to have a higher cost than a routine biopsy, as the former procedure generates a greater number of cores for analysis. The biopsy costs used in the CG17511 MRI model were based on NHS reference costs from 2011/12. These have been updated to 2012/13 prices by the EAG (Table 34).

Costs of biopsy complications

In line with the CG17511 MRI model, the model developed by the EAG uses the costs of biopsy complications reported by Mowatt et al. 14 (updated to 2012/13 prices). However, some HRG codes have changed slightly since the Mowatt et al. 14 model was developed; where appropriate, the HRG codes that appear most relevant to the codes reported by Mowatt et al. 14 have been used in the EAG model. Biopsy complication costs are presented in Table 35.

The costs associated with biopsy complications that are included in the model should be considered as conservative, as literature searches failed to identify any published studies reporting the costs associated with sepsis or antibiotic resistance. To establish if this omission could affect findings, a sensitivity analysis was undertaken in which all complication costs were increased by 100%.

Costs of prostate-specific antigen monitoring

In CG17511 and in Mowatt et al. 14 PSA monitoring was assumed to occur twice a year and to be carried out by a GP practice nurse. A targeted literature search was undertaken by the EAG to identify any additional information that could indicate alternative PSA monitoring strategies, but no information was found that invalidated this assumption.

The cost, estimated to be £19.60, is based on a PSA test cost provided by Newcastle upon Tyne Hospitals NHS Foundation Trust (reported in CG17511) and the cost of a consultation with a practice nurse reported by Curtis. 122

Utility values

The only utility values required in the model are the disutilities associated with a biopsy. A targeted search of the literature was undertaken but no primary studies collecting disutility values specifically associated with prostate biopsy were identified. Neither the CG17511 MRI model nor the Mowatt et al. 14 model considers any disutility from the actual biopsy. Both studies include a discussion of the impact of including the disutility associated with urinary incontinence, should this occur as a biopsy complication or as a result of treatment; however, it is not clear how this is applied, as urinary tract infection, urinary bleeding and urinary obstruction do not necessarily lead to urinary incontinence.

The literature search identified one study (Heijnsdijk et al. 117) that investigated the utility values associated with a PSA screening programme. This study reported a utility decrement of 0.1 that lasted 3 weeks following a biopsy. This utility value was taken from an earlier study123 that focused on breast cancer biopsy and the duration of decrement was an assumption based on clinical opinion. Although this utility value is not ideal, in the absence of any other evidence, it has been incorporated into the EAG base-case model as a quality-adjusted life-year (QALY) loss of 0.0058 from a prostate biopsy.

It is not clear if the decrement used in the Heijnsdilk et al. 117 study is an average value that includes disutility as a result of complications of biopsy [including those listed in the table presenting the costs of biopsy complications (see Table 35) and additional complications such as sepsis]. In the absence of evidence to the contrary, the EAG has assumed that there was no additional QALY loss as a result of biopsy-associated complications. This assumption favours less specific strategies and therefore it is likely that it would bias results against the proposed tests.

Heijnsdilk et al. 117 also report lower and upper bounds of biopsy-related disutility of 0.06 and 0.13, respectively (QALY losses of 0.00346 and 0.0075, respectively), based on minimum and maximum values reported in the breast biopsy study. 123 As no measure of dispersion of disutility was provided in the published paper beyond these minimum and maximum values, the disutility has not been varied in the probabilistic sensitivity analysis. However, as suggested by the NICE Decision Support Unit report,124 the uncertainty has been explored using sensitivity analysis.

Discount rates

Both costs and benefits have been discounted at 3.5% per annum, as suggested in the NICE guide to the methods of technology appraisal. 118 A scenario analysis exploring the impact of a discount rates of 0% and 5% per annum was considered. However, as the model was based on a decision tree with linear transition through a pathway, changing the discount rate would change the costs and QALYs in each arm by exactly the same proportion and so leave the incremental cost-effectiveness ratio per QALY gained unchanged.

Incremental cost-effectiveness ratios

The results of cost-effectiveness analysis are presented as incremental cost-effectiveness ratios per QALY gained. These are calculated by dividing the difference in costs associated with two alternative strategies by the difference in QALYs:

Where more than two strategies are being compared, the incremental cost-effectiveness ratio is calculated according to the following process:

• The strategies are ranked in terms of cost, from least to most expensive.

• If a strategy is more expensive and less effective than the preceding strategy it is said to be ‘dominated’ and is excluded from further analysis.

• Incremental cost-effectiveness ratios are then calculated for each strategy compared with the next most expensive non-dominated option. If the incremental cost-effectiveness ratio for a strategy is higher than that of the next most effective strategy, then it is ruled out by ‘extended dominance’.

• Incremental cost-effectiveness ratios are recalculated excluding any strategy subject to dominance or extended dominance.

• The non-dominated strategies form an ‘efficiency frontier’ of strategies that are cost-effective and can then be judged against the value of an incremental cost-effectiveness ratio that is generally considered cost-effective by NICE, that is £20,000–30,000 per QALY gained.

Total number of biopsies

The different number of biopsies under each diagnostic strategy drives the different patient outcomes in the model. In the base case, the total number of biopsies is split into second biopsies recommended by the testing strategy and biopsies undertaken during PSA monitoring (Table 36).

Under the base-case PSA monitoring scenario, all patients without a second biopsy, or with a negative second biopsy, enter PSA monitoring. The total number of these patients is the same regardless of the strategy; therefore, the number of patients undergoing a repeat biopsy during PSA testing is independent of the strategy chosen and is always the same.

Mean costs and benefits

Costs and QALYs generated using the base-case parameter values are shown in Table 37.

Incremental analysis

The incremental results from the base-case analysis are presented in Table 38.

Varying derived sensitivity

The total numbers of biopsies performed if sensitivity is set at 80% or 95%, using data from the Porpiglia et al. study,99 are shown in Table 39.

Mean costs and benefits

Costs and QALYs generated by varying sensitivity values are shown in Tables 40 and 41.

Incremental analysis

The incremental results from varying the sensitivity level are presented in Tables 42 and 43.

Different prostate-specific antigen monitoring assumptions

The total numbers of biopsies performed when the PSA monitoring strategies assumed in CG17511 and Mowatt et al. 14 are adopted are shown in Table 44.

Mean costs and benefits

Costs and QALYs generated when the PSA monitoring strategies assumed in CG17511 and Mowatt et al. 14 are adopted are shown in Tables 45 and 46, respectively.

Incremental analysis

The incremental results when the PSA monitoring strategies assumed in CG17511 and Mowatt et al. 14 are adopted are presented in Tables 47 and 48, respectively.

As the Mowatt et al. 14 PSA monitoring scenario results in an identical reduction in biopsy numbers (and therefore cost and QALY loss) across the testing strategies, this scenario does not affect the base-case incremental costs and QALYs between strategies. Therefore, the resultant incremental cost-effectiveness ratios per QALY gained for this scenario are the same as those for the base case.

Alternative effectiveness data sources

The total number of biopsies performed if the derived sensitivity values presented in Scattoni et al. 102 (80% and 90%) or Gittelman et al. 45 (90%) are employed in the model are shown in Tables 49 and 50, respectively.

Mean costs and benefits

Costs and QALYs generated if the derived sensitivity values presented in Scattoni et al. 102 (80% and 90%) or Gittelman et al. 45 (90%) are employed in the model are shown in Tables 51–53, respectively.

Incremental analysis

The incremental results if the derived sensitivity values presented in Scattoni et al. 102 (80% and 90%) or Gittelman et al. 45 (90%) are employed in the model are shown in Tables 54–56, respectively.

Upper bound of biopsy complication rates

Table 57 shows the incremental analysis if the upper bound of complication rates from Table 32 are used.

Lower price of Prostate Health Index

Table 58 shows the incremental analysis if the cost of phi test decreased by 50% (i.e. £44.92 as opposed to £89.83).

Quality-adjusted life-year loss from biopsy

Table 59 shows the incremental analysis if the QALY loss from biopsy was at the upper bound suggested in the literature117 (i.e. 0.0075 as opposed to 0.0058).

One hundred per cent increase in biopsy complication costs

Table 60 shows the incremental analysis if the costs of biopsy complications are increased by 100%.

Fifty per cent of cancers are missed on second biopsy

Table 61 shows the incremental analysis if the sensitivity of second biopsy indicated by a testing strategy is 50% rather than 100% (as assumed in the base case).

As was the case with the Mowatt et al. 14 PSA monitoring scenario, reducing the sensitivity of biopsy by 50% changes biopsy numbers by the same amount for all strategies. Thus, it alters overall costs for strategies, but incremental costs and incremental cost-effectiveness ratios remain the same as in the base case.

Variation in the proportion of patients with negative second biopsies entering prostate-specific antigen monitoring

Reducing the percentage of patients with negative second biopsies entering PSA monitoring should favour those testing strategies with lower specificity. This is because such testing strategies result in more second biopsies and under this sensitivity analysis fewer patients receive PSA monitoring. Therefore, the results of only the extreme end of the sensitivity analysis are shown, that is where 0% of patients with negative second biopsies enter PSA monitoring. The different numbers of biopsies associated with each diagnostic strategy, assuming 0% of patients with negative second biopsy enter PSA monitoring, are shown in Table 62.

This sensitivity analysis shows that testing strategies with higher specificity result in more biopsies being undertaken during PSA monitoring than testing strategies with lower specificity.

Mean costs and benefits

Costs and QALYs generated by each diagnostic strategy, assuming that 0% of patients with negative second biopsy enter PSA monitoring, are shown in Table 63.

Incremental analysis

The incremental results, assuming that 0% of men with a negative result from a second biopsy enter PSA monitoring, are shown in Table 64.

Base-case analysis

The CEAC for the base-case analysis is shown in Figure 6. It demonstrates that the most cost-effective strategy at £20,000 per QALY gained is clinical assessment alone in 100% of model iterations. At about £33,500 per QALY gained approximately half of the iterations suggest that clinical assessment alone is the most cost-effective strategy, whereas the remaining iterations suggest that it is clinical assessment + mpMRI that is the most cost-effective strategy. At a threshold of £37,000 per QALY gained, all iterations suggest that clinical assessment + mpMRI dominates all other strategies.

FIGURE 6.

Cost-effectiveness acceptability curve (base case).

Figures 7 and 8 show the CEACs for the base case using 80% and 95% sensitivity estimates, respectively. As with sensitivity at 90%, both CEACs show that no testing strategy other than clinical assessment was cost-effective in any model iterations at threshold values below £30,000 per QALY gained.

FIGURE 7.

Cost-effectiveness acceptability curve (base case: 80% sensitivity).

FIGURE 8.

Cost-effectiveness acceptability curve (base case: 95% sensitivity).

Alternative effectiveness data sources

Cost-effectiveness acceptability curves using alternative effectiveness data reported by Scattoni et al. 102 and Gittelman et al. 45 are shown in Figures 9–11. All three CEACs show that there were no model iterations in which the PCA3 assay or phi was cost-effective compared with clinical assessment alone at threshold values at, or below, £30,000 per QALY gained.

FIGURE 9.

Cost-effectiveness acceptability curve (Scattoni et al. :102 80% sensitivity).

FIGURE 10.

Cost-effectiveness acceptability curve (Scattoni et al. :102 90% sensitivity).

FIGURE 11.

Cost-effectiveness acceptability curve (Gittelman et al. :45 90% sensitivity).

Predominance of one study

Of the 10 clinically relevant comparisons described in the 17 papers,45,46,85,86,89–92,94,96,97,99,102–106 data from the study by Porpiglia et al. 99 are used in nine comparisons. Data from Scattoni et al. 102 are used in four comparisons and the remaining 13 populations provide data for a single comparison. Clearly, relying heavily on the study by Porpiglia et al. 99 is a limitation of both the clinical validity review and the results of the economic modelling undertaken by the EAG. The EAG acknowledges that clinical assessment in this study is not representative of clinical practice in the NHS and that MRI results were not considered before the biopsy because of the design of the study protocol. However, the EAG considers the Porpiglia et al. study99 to be the only published study that reports data that could be used to inform the de novo economic model.

Clinical relevance of reported outcome measures

Only one of the included studies105 reported sensitivity and specificity estimates and the EAG considers this to be a substantial weakness in the available data. Logistic regression models offer the potential to study multiple tests. However, although outcome measures such as AUC and multivariate diagnostic ORs may show improved accuracy, they lack clinical relevance as it is not clear whether improvements in overall accuracy are because of improved sensitivity or improved specificity. 127 Derived sensitivity and derived specificity values offer clearer advice but it is not possible to identify individual threshold values of the risk factors included in the model which are associated with a particular achieved level of sensitivity or specificity.

Decision curve analysis results were reported in several publications90,92,97,99,102,106 included in the review of clinical validity. Neither the PCA3 assay nor phi when added to clinical assessment (or clinical assessment plus MRI) showed increased net benefit below a threshold risk of approximately 15% (i.e. if the assessed risk from the model is below 15% then the addition of a new test does help decision-making).

The EAG noted the following limitations in the use of decision curve methodology as used in the included studies:

• None of the reviewed studies used the option of adding a variable for the harm associated with a test (i.e. complications).

• The reviewed studies weighted the benefit of diagnosed cases as 1 but it is not clear whether or not this approach is appropriate when considering the identification of clinically insignificant cancers.

• The method does not consider the harm arising from missing cancers.

Clinical assessment

The process of clinical decision-making is difficult to capture, standardise and evaluate within a study population. In the reviewed studies, descriptions of clinical assessment varied widely. When definitions of clinical assessment are unclear or are very different across studies, it may not be clinically meaningful to compare results. In two studies85,97 previously validated nomograms were used to reflect clinical assessment. Another study105 used a clinical decision algorithm that had been developed in conjunction with 12 European urologists. The EAG considers that this type of decision tool may be the best representation of clinical assessment in the included studies.

The EAG notes that the inclusion of PSA in logistic regression models used to assess the efficacy of phi is inconsistent and gives rise to concerns about the validity of the model results as the phi result already includes a measure of PSA.

Use of different thresholds across the studies

The manufacturer of the PCA3 assay proposed a threshold value of 25 to differentiate between the presence and absence of prostate cancer. However, results using this threshold value were reported in only one study. 45 Other studies used 35,86 3989 and 5089 or used the PCA3 score as a continuous variable. 46,86,90,97,99,102

The manufacturer of phi proposed using three groups: low risk of cancer (score of 0 to 20.9), moderate risk of cancer (score of 21 to 39.9) and high risk of cancer (score of 40 and above). However, the four studies90,97,100,103 that used phi used the phi test results as a continuous variable.

It was difficult for the EAG to draw conclusions from the limited data available, as the included studies used a range of threshold values for the PCA3 assay and none of the studies used the phi tests results recommended by the manufacturer.

Confidence intervals and statistical significance of clinical validity results

Many of the reported results for the clinical validity outcomes do not include either standard errors or CIs. It has, therefore, not been possible for the EAG to assess whether or not the differences between groups were statistically significant. Values for derived sensitivity and derived specificity reported in Porpiglia et al. 99 were similar for several models that involved different combinations of diagnostic tests. This may have been because there were small numbers of participants above or below the required threshold associated with a given level of sensitivity and specificity.

Lack of generalisable clinical validity data to inform the economic model

As is the case with all economic models, the results are limited by the generalisability of the available evidence data used to populate the model. In the study99 used to inform the base-case analysis in the EAG model, the analysis undertaken is appropriate; however, there are differences in clinical practice between the study99 and the NHS in England and Wales. To explore the impact that these differences may have had on the incremental cost-effectiveness ratios, data from other studies with alternative clinical assessments were modelled. The EAG is confident that using alternative assumptions did not change the model findings regarding the probable cost-effectiveness of adding the PCA3 assay or phi into a testing strategy.

Limited incorporation of utility values

Although the model attempted to capture all the important clinical and cost events, it was not possible to capture and/or value all the key factors that might influence cost-effectiveness. The main area where information is lacking is in relation to utility decrements associated with prostate biopsies. It was necessary to use a proxy value, based on the findings from a study117 that focused on breast cancer biopsy, to represent pain and short-term complications. Any utility decrements associated with anxiety prior to a biopsy were omitted from the model because of lack of information. Inclusion of specific utility values would require a study that assessed utility across the testing process and would need to take into account anxiety not just from a second biopsy or waiting for mpMRI, but also from any change in anxiety associated with the change in risk information that different testing strategies may offer patients.

Detection of clinically insignificant cancers

The management of men who have been diagnosed with prostate cancer varies depending on the grade and extent of the cancer at diagnosis. Clinically insignificant cancers are monitored with active surveillance or watchful waiting. Many clinicians are concerned that, rather than improve health, increased diagnosis of clinically insignificant cancers may lead to an increase in morbidity because of anxiety.

One aim of the clinical validity review was to assess the ability of the PCA3 assay and the phi to improve the detection of more aggressive cancers. A lack of evidence meant that it was not possible for the EAG to draw any conclusions about the impact of the PCA3 assay or phi on the detection of clinically insignificant cancers.

Evidence for the relationship between the aggressiveness of tumours detected at prostate biopsy and the PCA3 assay or the phi has been reported in previous reviews. 128,129 These reviews128,129 were not restricted to studies of repeat biopsy populations and do not consider the intervention test results used in combination with other diagnostic tests. Filella et al. 128 in a narrative review, highlighted inconsistencies in the evidence linking higher PCA3 scores to various markers of tumour aggressiveness. Wang et al. ,129 in a meta-analysis of four studies, found that the AUC of phi for discriminating cancers with a Gleason score of above or below 7 was 0.67 (95% CI 0.57 to 0.77), with a sensitivity of 90% (95% CI 87% to 92%) and a specificity of 17% (95% CI 14% to 19%). The authors of the review129 comment on the need for further research.

If the PCA3 assay or phi has higher sensitivity for detecting aggressive cancer than clinical assessment alone (or clinical assessment plus MRI), the use of these tests may be cost-effective. Unfortunately, the studies that provided data that could be included in the EAG model did not report this type of result.

Order of tests

In the included studies, the results of tests were often presented as outputs from logistic regression models with all tests entered into one model. This approach meant that it was not possible to determine from the available data whether or not carrying out the diagnostic tests in one order was better than carrying out the tests in a different order. For example, it was not possible to determine whether or not diagnostic accuracy was improved if the PCA3 assay (or phi) was carried out before or after MRI. However, in clinical practice the order of the tests is important and has substantial cost and benefit implication, as the costs of MRI are higher than either the PCA3 assay or the phi and the order of the tests may result in different sensitivity and specificity estimates. There is no clinical, and therefore no economic, evidence on using the PCA3 assay or the phi to indicate whether or not mpMRI should be performed before a second biopsy. However, the economic model results provide evidence that if mpMRI is performed then the added information from the PCA3 assay or phi is minimal and incorporation of either into a testing strategy that will include mpMRI will, therefore, not be cost-effective.

Effect of different type of reference standard (prostate biopsy)

It has been shown that, in a given population, biopsy schemes that take a large number of cores spread widely across the prostate, such as saturation schemes, result in a higher prevalence of detected cancer than schemes that involve taking only a few cores. 1 An important clinical point is, therefore, to question if any incremental gain associated with the addition of the PCA3 assay or the phi to clinical assessment would vary depending on the biopsy scheme used to confirm the presence or absence of cancer. As discussed in Chapter 1, Potential sources of bias, any advantage gained by adding the PCA3 assay, or the phi, to clinical assessment could be reduced if a more extensive biopsy scheme were used. The EAG planned to assess this issue in the review of clinical validity by comparing results in studies which used different types of reference standard; however, the details of reference standards used in studies were poorly reported. An added complication was the fact that the number of biopsy cores taken often differed between patients within a single study. Where details were provided, 10- or 12-core biopsies were the most common.

Extent to which the model reflects NHS clinical practice

The EAG economic model addresses clear questions:

Given a desired cancer detection rate for the target population, what proportion of the population would need a second biopsy and what proportion of these second biopsies would be necessary?

However, the extent to which these questions reflect clinical practice is unclear. It is hypothesised that clinicians are more likely to think in terms of individual patients rather than in terms of desired cancer detection rates for the whole population of men suspected of having prostate cancer.

The actual achieved sensitivity and specificity of incorporating the PCA3 assay and the phi into a diagnostic strategy are unknown. The clinical evidence available does not address all of the factors that influence diagnostic practice, such as patient preferences for second biopsies given previous biopsy experience and increased/decreased anxiety levels resulting from the findings of additional tests that place patients in different risk categories. Ideally the cost-effectiveness model would be populated using ‘real-world’ data. In the absence of real-world data the EAG model has been constructed in such a way as to allow the tests to be compared fairly but at arbitrary levels of sensitivity.

Prostate-specific antigen monitoring strategy after a negative or equivocal biopsy

A key area of uncertainty is related to the best representation of the PSA monitoring strategy within the economic model. No published information was found by the EAG that described NHS monitoring practice in England and Wales. As a consequence, this parameter was varied in the sensitivity analyses and the incremental cost-effectiveness ratio per QALY gained fell below £20,000 only if the PSA monitoring regime employed in the CG17511 MRI model was used. In the PSA monitoring strategy described in CG175,11 all men with negative or equivocal results from an initial biopsy go on to receive at least one further biopsy and up to 6 years of PSA monitoring. When used in the EAG model, this assumption would mean that the optimal strategy would be to immediately carry out a further biopsy on everyone shown to be negative or equivocal on the initial biopsy and undertake no PSA monitoring.

Unclear clinical priorities

Improvements in diagnostic test accuracy are often a balance between a gain in sensitivity at the expense of lower specificity, or vice versa. Clinical priorities determine whether sensitivity or specificity is the most important outcome in any particular diagnostic situation. To understand the clinical implications of the findings of the clinical validity review, and to inform the design of the economic model, the EAG surveyed a convenience sample of five clinicians. The clinicians were asked whether, for men undergoing repeat prostatic biopsy, sensitivity or specificity was the most important and whether it was possible to identify a minimum level of sensitivity or specificity that that a test should achieve. Disparate views were expressed, with some clinicians favouring high sensitivity so that all cancers were identified, while others expressed a desire for a test that only identified the more aggressive cancers. No minimum level of sensitivity or specificity was suggested. The EAG, therefore, took the pragmatic approach of using sensitivity estimates of 85%, 90% and 95% in the economic model.

Unclear target population

The precise target patient population for the new tests is also unclear. As discussed in Chapter 2, Quality assessment, men with a negative result following an initial biopsy can be categorised into three groups: those with clear risk factors for a repeat biopsy (at high assessed risk), those with no remaining risk factors (at low assessed risk) and those where clinicians are uncertain (intermediate assessed risk). Most of the eligible studies included only men who had been referred for a repeat biopsy and had, therefore, presumably been assessed as at high or intermediate risk. It is not clear whether or not clinicians would wish to use the PCA3 assay and/or phi in all men, including those currently assessed as at low risk.

False-negative results

The impact of a FN result at repeat biopsy on the length of time to final diagnosis (and the impact that that delay might have on disease progression) is also an issue. However, recent data suggest that risk reductions associated with radical treatment for low-risk patients (and even moderate-risk patients) may be small and insignificant. 130 If this is the case, it might undermine the cost-effectiveness of strategies that increase cancer detection rates and costs over standard practice, unless those strategies are able to discriminate by grade of tumour. Furthermore, there appears to be no published information on the rate of FPs and overtreatment, although the EAG modelling approach means that these would be the same for all testing strategies and therefore should not impact on results.

Equality and diversity

The incidence of aggressive prostate cancer is greater in people with obesity, which can lead to the positive predictive value of a DRE being higher; DRE can be more difficult to perform in people with obesity. 131 The economic results rely on the results from clinical studies in which a DRE is part of a clinical assessment. It may be that cost-effectiveness results for the PCA3 assay and the phi differ depending on whether or not clinical assessment includes a DRE; this should be considered against the possibility that for some people a DRE may not be possible or may be more difficult to undertake.