Optimal surveillance strategies for Patients with stage 1 cutaneous melanoma post primary tumour excision: three systematic reviews and an economic model

Staging of disease

There have been great advances in the earlier detection of primary melanoma through increased public awareness, the adoption of dermatoscopic examinations and a rapid ‘2-week wait’ referral system in the UK. 16 There is also widespread belief that earlier detection of metastatic disease results in improved patient outcomes. 17 However, at present, there is no internationally accepted standardised model of follow-up of patients diagnosed with cutaneous melanoma, with wide variations in care across North America, Australia, Europe and the UK. 18

When it comes to follow-up of those treated for melanoma, disease staging and judgements on the risk of spread (metastasis) are based on the microscopic appearance and depth of the original tumour. Currently, this is based on the American Joint Committee on Cancer (AJCC) eighth edition staging criteria, which were published in 2016 and formally implemented on 1 January 2018. 19 However, as this is an evidence synthesis project, the definitions used in the 2010 seventh edition20 are also pertinent, as existing data would have based decisions on staging using this edition or earlier editions. The seventh edition included mitotic count (the number of actively dividing cells in the tumour), as this was thought to be an important prognostic feature for thin melanomas,21 but this has been dropped from the eighth edition staging guidelines because of a lack of evidence supporting its prognostic significance. The key definitions of stage I and II disease for both the seventh and eighth editions are set out in Table 1.

Specific changes of note between the seventh and eighth AJCC staging criteria affecting stage I melanoma criteria are as follows:

• T1a has had the Breslow depth reduced to 0.8 mm when non-ulcerated

• T1b is now any ulcerated tumour of < 0.8 mm Breslow depth or between 0.8 and 1.00 mm Breslow depth regardless of ulceration status

• mitotic count has no role in the defining stage.

One further change of note is the distinction between clinical and pathological staging for non-ulcerated tumours that are between 0.8 and 1 mm. These tumours are clinical stage IB, but if they undergo a sentinel lymph node biopsy (SLNB) that is negative, the tumour is ‘downgraded’ to pathological stage IA, with associated changes in overall prognostication. This subcohort of patients is likely to represent a tiny proportion of UK patients, given that, under the National Institute for Health and Care Excellence (NICE) guidelines,16 they would not be routinely offered SLNB if their tumour is of < 1 mm Breslow depth.

The AJCC stage I disease encompasses both stages IA and IB disease and represents the thinnest tumours. At initial diagnosis, 70% of melanomas are classified as AJCC stage I. Early-stage tumours are treated by surgical excision. However, the 5-year overall survival of patients with stage I disease is only 95%. 22 Stage II disease encompasses thicker, but still localised, tumours. Stages III/IV patients have evidence of local and distant metastases; 2-year mortality is up to 82% in stage IV disease, although, with the introduction of new systemic agents, this is now falling.

Surveillance strategies

Given the relatively low rates of local or systemic recurrence, those with AJCC stage I melanoma may not need the same level of clinician follow-up as is generally recommended,23 whereas patients at higher risk following surgical treatment may benefit from more intensive future surveillance to detect recurrent or metastatic disease early. However, balanced against this is the fact that approximately 10% of patients with AJCC stage I disease develop metastases and the prognosis for these people remains poor. Currently, this results in rigorous, routine follow-up for all melanoma patients.

The potential interventions and investigations used as part of a post-surgical treatment surveillance strategy also varies in AJCC stage I patients. An important element of surveillance is education of the patient to allow them to identify any new lesions of concern or signs of recurrence. In one study by Hofmann et al. ,24 30 (24%) of the 127 patients who had a first relapse were not being formally followed up at the time that the relapse was detected. This is because they had never been in a follow-up programme, had dropped out of follow-up, or had completed the formal follow-up process. These data demonstrate the often erratic and unpredictable course of the disease. In the same study, 68% of first relapses were detected by follow-up activity.

Nevertheless, regular clinical history and examination is the mainstay of most surveillance guidelines. Again, which type of health-care practitioner (e.g. nurse, surgeon, dermatologist) undertakes the examinations varies, as does the setting of these reviews, with recommendations for either primary care-based or secondary care-based (in-hospital) appointments. Specific radiological examination of patients may also be recommended for follow-up of stage I melanoma. Routine use of imaging modalities aims to detect the development of regional and distant metastases as early as possible, even before these become clinically apparent. However, if a patient is found to have clinical evidence of metastases, a further set of imaging modalities such as ultrasonography, computerised tomography (CT) and positron emission tomography-computerised tomography (PET-CT) may be used. These methods allow targeted biopsy, when possible, of the relevant melanoma deposits to allow histopathological assessment of the tissue. In the UK, modalities such as CT and PET-CT are not currently advocated as part of the routine management and follow-up of AJCC stage I disease, and it is felt to be unlikely that this situation will change in the next 5–7 years. As others have noted,25 and described in more detail in Chapter 2, there is considerable variability of surveillance practices worldwide.

The British Association of Dermatologists (BAD) revised UK guidelines for the management of cutaneous melanoma26,27 and the more recent NICE Melanoma: Assessment and Management guideline16 advise that patients who have stage I melanoma are followed up to detect signs of recurrence after history and examination. This surveillance is undertaken as follows:

• Patients with stage IA melanoma should be seen two to four times over a period of up to 12 months, and then discharged.

• Patients with stage IB melanoma should be seen every 3 months for 3 years, and then every 6 months for a further 2 years.

There are no recommendations for the routine use of any radiological modality, only guidance that these can be implemented if required in symptomatic patients.

There are currently no biomarkers in routine use in any guidelines for stage I disease. Lactate dehydrogenase blood levels are validated for use in patients with evidence of metastases only. 28 There is also increased application of serum S100B, but, once again, in patients with evidence of metastatic disease only. 29

Any changes to current recommended practice would need to consider multiple components, each of which would determine the costs, effectiveness, feasibility and acceptability of an alternative strategy (Box 1).

Current treatment options

The NICE guideline for melanoma,16 published in July 2015, recommends excision of melanomas of stages 0–II with a 0.5- to 2-cm total margin, depending on stage and histopathological assessment of the biopsy (Figure 1). Risk of disease progression is estimated based on the AJCC eighth edition staging criteria. 19 As described in the previous subsection, patients deemed to be at a low risk of disease progression are followed up at regular intervals for a period of 5 years following diagnosis, undergoing visual and physical examinations.

FIGURE 1.

The National Collaborating Centre for Cancer’s melanoma staging algorithm, including recommended use of SNLB.

Current draft NICE guidance does not suggest using SNLB for AJCC stage IA or IB melanomas with a Breslow thickness of < 1 mm, and acknowledges that a proportion of those patients with a negative SLNB will still experience melanoma recurrence. 16 This stance is further supported by a UK consensus statement made through a multidisciplinary meeting held by the Melanoma Focus group in 2018. 30

Since 2011, there have been rapid developments in the therapeutic options available for metastatic disease, with accompanying improvements in patient-related outcomes. These developments have been so rapid that we are currently in the follow-up period for many drug trials.

Metastatic disease encompasses the following:

• Satellite lesions – skin or subcutaneous deposits within 2 cm of the primary tumour.

• In-transit metastases – these occur > 2 cm from the primary tumour, but before the regional lymph node.

• Nodal micrometastases – metastatic deposits evident only following histopathological analysis of SLNB tissue or regional lymph node dissection.

• Nodal macrometastases – metastatic deposits in regional lymph nodes that are either clinically apparent or found on histopathological assessment of regional lymph node dissection.

• Metastases to distant skin, subcutaneous tissue, lymph nodes or other visceral sites/organs.

Localised metastatic disease is broadly distinguished based on the distance of spread and the total metastatic tumour bulk (and is based on AJCC staging criteria):

• IIIA

  • one to three local lymph nodes with micrometastases (diagnosed on SLNB or node dissection)

• IIIB

  • one to three local lymph nodes with macrometastases (clinically palpable lymph node involvement or within-node dissection)

  • in-transit metastases/satellite lesions with no metastatic lymph node involvement

• IIIC

  • four or more local lymph nodes involved

  • in-transit metastases/satellite lesions with frank metastatic lymph node involvement.

‘In-transit metastases’ covers a wide range of clinical presentations, ranging from localised, small melanoma deposits that are easily amenable to further surgery to > 100 deposits of bulky melanoma tissue. In such cases, the clinical decisions are made based on the extent and technical feasibility of treatment.

Among the most established therapies for in-transit metastases are isolated limb perfusion (ILP) and isolated limb infusion (ILI). Both of these therapies involve the isolation of a limb’s vasculature, with the addition of an anti-tumour agent into this closed system. The aim of therapy is to allow anti-tumour concentrations of the chemotherapeutic agent, without the associated systemic side effects. Traditionally, ILP and ILI have been carried out using melphalan, but, recently, they have been carried out with the addition of tumour necrosis factor. Overall, although tumour response rates range from 64% to 93%,31,32 the median survival time post treatment is still only 2 years. 33 There is currently no suggestion that ILP/ILI can be used in localised melanomas without any evidence of frank metastatic disease.

For metastatic deposits in lymph nodes (following detection by either SLNB or nodal biopsy), the most common therapy is for a lymphadenectomy (with or without post-operative radiotherapy34) of the involved lymph node basin. This has significant morbidity attached to the procedure and it is debatable whether or not there is any benefit for patients in terms of overall melanoma survival; it is currently not recommended routinely in SLNB-positive patients. 30,35,36

Distant metastases, encompassing stage IV disease, rely on systemic therapeutic options. In recent years, a raft of new therapeutic agents have been introduced. The current standard of care in the UK is in constant flux, but remains based on NICE guidance distilled from the continually changing evidence base for systemic therapies; however, there is still variation in local practice. It is generally accepted that adjuvant therapy with immune modulators should be made available to patients with frank stage II disease, or high-risk stage IIIA disease (a deposit of melanoma of > 1 mm² in the lymph node following SLNB), and that this is also a first-line treatment for stage IV disease. Combination therapies are also preferable for first-line use in stage IV or unresectable stage III disease.

The newer systemic agents can be categorised by their mode of action, either targeting the mitogen-activated protein kinase signalling pathway, or via immune checkpoint blockade. A multitude of clinical trials have been undertaken assessing the benefits of each group as first-line systemic therapy in patients with metastatic disease (usually AJCC IIIB and above), either as monotherapy or combined with another agent affecting the same pathway. Table 2 outlines the most influential recent clinical trials.

The vast majority of systemic agents are aimed at patients with evidence of distant disease progression. However, with the long-standing hypothesis that earlier introduction of systemic therapies may result in better response outcomes, studies have shown a benefit in introducing systemic agents before there is clinical evidence of metastasis. The 2019 NICE guidelines47 for treating stage III melanoma recommend that consideration be given to the use of two adjuvant therapies, nivolumab and pembrolizumab, in resected melanoma with evidence of lymph node involvement, including stage IIIA disease identified following SLNB. Similarly, dabrafenib and trametenib are also licensed and approved for resected stage III disease in BRAF-positive patients.

Such adjuvant regimes continue to be studied,48 but there is a need for better risk prediction of the prognosis of people treated for melanoma, especially for those with an earlier-stage disease, a significant minority of whom will experience progression. However, even for later-stage disease, a considerable number of people may receive these newer systemic regimens, but with only limited prospect of any gain. Hence, they are potentially being unnecessarily exposed to the side effects of systemic therapy with little or no overall benefit.

National Institute for Health and Care Excellence: Melanoma: Assessment and Management

This NICE guideline16 was published in 2015 to guide clinical practice for the management of melanoma in England. The process of the guideline formulation began in 2013 and the full draft was published in July 2015. There is an ongoing peer review of the guideline; the most recent report was in 2019,54 and suggested significant section updates to the guidelines.

British Association of Dermatologists revised UK guidelines for the management of cutaneous melanoma

The 2010 revised UK guidelines for the management of cutaneous melanoma27 are the current guidelines from the BAD, which have not been updated because of ongoing updates by NICE on melanoma management (see National Institute for Health and Care Excellence: Melanoma: Assessment and Management16). The BAD’s methodology for guideline formulation was also accredited by NHS evidence.

National Comprehensive Cancer Network guidelines, version 2.2019: Cutaneous Melanoma

This is the most recent (2019) version of the cutaneous melanoma guideline produced by the National Comprehensive Cancer Network^® (NCCN). 55 The NCCN is a not-for-profit alliance of 31 member cancer centres in the USA.

European Society of Medical Oncologists clinical practice guidelines for cutaneous melanoma

The European Society of Medical Oncologists (ESMO) produced a guideline on cutaneous melanoma in 2015. 56 ESMO comprises expert members from all over the world, although they are predominantly European. Hence, this guideline is more clinician focused than the NICE guidelines, which consider other audiences.

American Academy of Dermatology: guidelines of care for the management of primary cutaneous melanoma

Cancer Council Australia Melanoma Guidelines Working Party

As of September 2019, the clinical practice guidance for melanoma from the Cancer Council Australia Melanoma Guidelines Working Party53 is undergoing revision, with sections being released as they are completed. The current revision began in 2014, using a ‘wiki’ web-based platform to enable rapid updates, necessitated by the rapid turnover of new evidence, and to allow for sharing of information and ease of contribution among panel/working group members. It also allowed for collaboration with the German Dermatologic Cooperative Oncology Group, which provided access to some of its systematic reviews. Most of the guideline has now been published, with only two questions under development: ‘How should melanoma in children be managed?’ and ‘How should melanocytic tumour of unknown malignant potential be managed?’ Neither of these questions are directly relevant to the aims of the research reported in this health technology assessment.

Dutch Working Group on Melanoma

This is the 2012 national guideline produced by the Dutch Working Group on Melanoma. 58 The guideline addresses 19 questions. The appendices showing the list of questions covered are not available in English. However, a systematic review was conducted for three of the questions in the entire guideline; the remaining 16 questions had their recommendations based on studies put forward by guideline committee members.

A further revision was completed between 2014 and 2016 to answer three key questions: ‘The role of ¹⁸F-FDG-PET[fluorodeoxyglucose PET]/CT at diagnosis’,58 ‘the role of ¹⁸F-FDG-PET/CT in follow-up’58 and ‘the role of the sentinel node biopsy’58 (reproduced with permission from the Dutch Working Group on Melanoma58). This additional review was carried out by the Nederlandse Vereniging voor Nucleaire Geneeskunde (Dutch Society for Nuclear Medicine) and the Nederlandse Vereniging voor Pathologie (Dutch Association for Pathology).

German Guideline Program in Oncology

The S3-Guideline ‘Diagnosis, Therapy and Follow-up of Melanoma’ – Short Version is the most recent (2013) English-language version of the short version of the German guideline for cutaneous melanoma. 59,73 However, there have been updates in 2015 and in 2016/17, owing to the rapid developments in the field. These updated versions are available online in German only. 73 The ongoing stated plan will be to have a live system that allows for regular updates. A new version of this guideline is planned by the end of 2019.

Swiss Cancer League

The updated Swiss guidelines 2016 for the treatment and follow-up of cutaneous melanoma60 are an update of a 2006 guideline and were initiated as a result of advances in diagnostic capabilities and treatment options. 60

Brazilian guidelines for diagnosis, treatment and follow-up of primary cutaneous melanoma – Part II

This guideline, from 2016, is a follow-up to the 2002 Brazilian guideline and is the second part of the full guideline. 61 There was a need for the update as a result of recent developments in diagnosis and treatment. This update covered 10 questions; five of these questions are covered in this second part of the guideline. The five questions were on follow-up for stage 0 and I melanoma, the role of body mapping in follow-up, the benefit of sentinel lymph node in primary melanoma and the benefits of preventative excisions of acral naevi and giant congenital naevi.

Search strategy

The search strategy was designed by an experienced information specialist in collaboration with the project team. The search was designed in MEDLINE [via Ovid^® (Wolters Kluwer, Alphen aan den Rijn, the Netherlands)] according to the following main concepts: [melanoma] AND [surveillance OR screening]. Database-specific thesaurus headings were used, together with title and abstract keywords. The strategy was translated to other databases (Box 2), altering thesaurus headings and search syntax as appropriate. The databases listed in Box 2 were searched during the first week of May 2018 and the search was updated on 2 July 2019.

The search was then limited to studies published from 2011 onwards, the search date of a previously published systematic review of surveillance strategies for melanoma. 25 There were no restrictions according to language or publication status. The search strategy used in MEDLINE can be found in Appendix 1.

We did not update the systematic review authored by Cromwell et al. ;25 instead, we used it to identify publications prior to 2011. To reduce the screening burden of systematically searching from database inception, we screened all studies included in this review. 25 Full references of the review were also screened, in addition to the results of the systematic search limited to studies published from 2011 onwards.

A grey literature search plan was developed to complement this search by exploring (1) grey literature databases, (2) targeted websites and (3) reference leads from (1) and (2), with a focused attempt to locate international and national guidelines. By checking the references of international guidelines and the studies cited in a review by Cromwell et al. 25 extended out search to before the 2011 limit described above. The sources described in Box 3 were searched between 20 July 2018 and 10 September 2018.

Titles and abstracts of search results were imported into EndNote [Clarivate Analytics (formerly Thomson Reuters), Philadelphia, PA, USA] and deduplicated.

Inclusion and exclusion criteria

We based inclusion and exclusion criteria on the population, intervention, comparator, outcomes, timing and setting (PICOTS) formula, as outlined in the following sections. 80

Data collection

Risk-of-bias assessment in included studies

We used the Cochrane Collaboration’s ‘Risk-of-bias’ tool, RoB 2.0, which uses signalling questions to assess risk-of-bias judgements. 83 The tool examines five domains graded as being at a low risk-of-bias, of some concern or at a high risk of bias, from which an overall risk-of-bias judgement can be made. The domains considered are biases resulting from the randomisation process, deviations from the intended intervention, missing outcome data, measurement of outcomes, and selection of reported results.

Measures of effect

We planned to extract the following reported measures of effects of surveillance:

• dichotomous or binary data, odds ratios (ORs) or RRs or percentages

• time-to-event data, HRs or Kaplan–Meier plots.

Confidence intervals for all estimates missing data

We set out to report the number (per cent) of missing data for all variables/outcomes. We did not impute missing outcome data for any of our specified outcomes.

Data analysis

We summarised the characteristics of the surveillance strategies from included studies and guidelines in a table and provided a narrative summary of these.

We planned to conduct random-effects meta-analyses to pool data for each outcome in the review. In the absence of sufficiently robust or similar studies for a meta-analysis, we carried out a summary of studies, rather than a more formal narrative synthesis, owing to a lack of evidence.

Quality of the evidence using the GRADE approach

The GRADEpro tool was used to assess the overall certainty in the body of evidence for key outcomes. 84 The GRADE approach uses the risk of bias of individual studies, along with characteristics such as the imprecision and inconsistency of their results, to produce an overall estimate in terms of whether there is high, moderate, low or very low confidence that the systematic review estimates the true effect. These data are presented as a summary of findings table (see Table 6).

Number of studies identified

The searches retrieved 10,723 citations in total; 10,592 were retrieved from the electronic databases, 104 from the published systematic review25 and 27 from the grey literature and guidelines search (Figure 3). After deduplication, 6205 references remained. Following a title and abstract sift by one reviewer and two clinicians, 6134 references were excluded, resulting in 33 citations of articles and conference abstracts for full-text assessment.

FIGURE 3.

The PRISMA flow diagram for the review of surveillance strategies’ clinical effectiveness.

Following this, we excluded 31 articles. The reasons for excluding the full-text papers were as follows: < 80% of participants at stage I or wrong stage (58%), and studies identified as prognostic studies (26%) diagnostic studies (10%) or prevention studies (6%). In addition, all included studies from the Cromwell et al. 25 systematic review were excluded because they were not surveillance strategies among individuals post resection of a stage I melanoma, and most included studies had a non-comparative design. 25 A list of excluded full-text articles retrieved from the literature search, with reasons for exclusion, is provided in Appendix 2.

Authors of relevant studies presenting aggregated data were contacted to provide data stratified by stage. Correspondence was received from Robinson et al. 85 and Damude et al. ;23 however, only Robinson et al. 85 provided data that fulfilled our inclusion criteria.

Characteristics of included studies

One RCT met our inclusion criteria85,86 after provision of further data by the authors. Robinson et al. 85 assessed the frequency of SSE by patient–partner dyads. The study was conducted in the USA. 85 The study included a total of 494 participants with a mean age of 55 years [standard deviation (SD) ±10 years, range 18 to > 70 years). Descriptive information of the study is presented in Table 4.

Patients with stage 0–IIB melanoma participated in the trial from June 2011 to April 2015. This was a continuation of the trial initially reported by Turrisi et al. 87 Patients in the intervention arm received a structured skills training intervention, whereas patients in the control arm received customary care.

A total of 494 patients and their partners were randomised to one of four groups. Three of the dyad groups received a structured skills training intervention in SSE, either in person, from a written workbook or via a tablet. The fourth group served as control and received treatment as usual87 and customary education. 85 Patients were seen by a dermatologist every 4 months. The primary outcome was frequency of SSE by patient–partner dyads, and the follow-up period and end point of the trial was 24 months. The secondary outcome was detection of a new or recurrent melanoma by the dyad or physician.

Patients at stages 0–IIB receiving the intervention had significantly increased SSEs with their partners at 4 months, compared with controls [mean difference 1.57, 95% confidence interval (CI) 1.29 to 1.85]. Mean differences at 12 and 24 months were lower (mean difference 0.72, 95% CI 0.39 to 1.06, and mean difference 0.94, 95% CI 0.58 to 1.30, respectively). Overall, data reported for stages 0–IIB showed that the intervention was successful in increasing SSE by patient–partner dyads, compared with controls at 24 months (mean difference 0.94, 95% CI 0.58 to 1.30; p < 0.001). We contacted the authors for data by disease stage, which were provided by them for our analyses.

The individuals undertaking surevillance (through SSE) of the outcome of interest to our review (detection of a new primary tumour, recurrent melanoma or metastases) were both the dyad and physician, with the site of surveillance being either the home or the care setting of the dermatologist (predominantly a secondary care setting). The interval timing of the review appointments with the dermatologist were 4 months; however, for surveillance by the dyad, the interval timing was dependent on their own timeline of use of SSE. The duration of follow-up for the surveillance in the trial, and thus by dermatologists, was 2 years. However, the SSE by the dyads was intended to last for life. There was no routine imaging involved in the surveillance strategy; rather, the strategy was based on a structured skills training intervention on how to self-identify plausible new or recurrent melanoma. Given that the study reported that SSE increased, it has been assumed that this surveillance stratgey is well accepted by melanoma patients. There is no large burden on health-care providers due to surveillance by SSE.

Risk-of-bias assessment of included studies

It was judged that there were ‘some concerns’ regarding the risk of bias in the study by Robinson et al. 85 Two allied papers were used to identify data pertinent to the risk-of-bias assessment. 86,87 The results are discussed in the following sections, and the judgements made using the Cochrane risk-of-bias tool, RoB 2.0,83 are shown in Table 5.

Assessment of clinical effectiveness

Although three publications (including a conference abstract) reporting two RCTs were eligible for this systematic review,23,85,88 they did not report data for stage I patients separately. Authors were contacted to provide further data, which we obtained for stage I melanoma patients from the authors of the study by Robinson et al. 85

We were unable to conduct a meta-analysis as only one RCT met the inclusion criteria. We were unable to assess reporting biases using funnel plots or to conduct any subgroup or sensitivity analyses. Thus, data on effectiveness and safety from the included RCT were tabulated and presented in the summary of findings table (Table 6) and narratively summarised. For outcomes of interest, we have calculated and reported the magnitude of effect.

The primary outcome of the study by Robinson et al. 85 was the frequency of SSE by patient–partner dyads. The secondary outcome (among those post resection of stage 0–IIB melanoma) was detection of a new or recurrent melanoma by the dyad or physician. For those post resection of stage I melanoma, the population of interest in this review, data were provided by the study’s author. New primaries, recurrences or metastases were detected in 49 out of 258 (19%) patients with stage IA or IB melanomas post resection of a primary melanoma followed up for up to 24 months. Data were not split by whether the disease was a new primary or recurrence, and recurrences could be at different stages from the original primaries. The types of melanomas identified were melanoma in situ, stage IA, superficial spreading, lentigo maligna and melanomas of ≥ 0.1 mm.

There was no evidence of a difference between intervention and control arms in the proportion of patients with stage IA or IB melanomas in which a new primary or recurrence was detected in this subset (RR 0.75, 95% CI 0.43 to 1.31). However, imprecision affects our certainty of this finding and more evidence is needed to draw any conclusions.

The authors85 concluded that patients with melanoma and their partners reliably performed SSE after participating in a structured skills training programme lasting approximately 30 minutes, with reinforcement every 4 months by the study dermatologist. No conclusions were drawn by the study authors85 about how new primaries or recurrences were detected.

Summary of review of different surveillance strategies

This review sought evidence about the relative effectiveness of surveillance and follow-up strategies to identify melanoma recurrence, new primary tumours and metastases in stage I cutaneous melanoma patients following surgical excision of the primary tumour. Only two RCTs (reported in three papers)23,85,88 were eligible and we could obtain data on stage I patients from only one of them. 85 This study suggested that an educational intervention for patients and their partners improved self-identification of new primaries, regardless of whether it was delivered in person, through a workbook or via a tablet. However, among the subset of author-provided data on patients post resection of a stage IA or IB melanoma, there was no evidence of a difference in detection of a new primary tumour, recurrence or metastases between those undergoing SSE surveillance and those receiving usual follow-up.

This evidence is of low certainty according to GRADE because of the small number of studies and limited number of available relevant outcome data;86 it is probable that the results of this review would change with the addition of new eligible studies. The certainty of the evidence was downgraded for imprecision, sparse data and a low event rate and for concerns regarding risk of bias. At present, evidence is based on just a single study85 (n = 258 participants) and the evidence is incomplete and offers only internal validity as it was set in the USA. Only one of the prespecified outcomes (new primary or tumour recurrence) in our review was reported, meaning that there are complete gaps in the evidence in this area in terms of overall survival, progression-/recurrence-free survival and detection of recurrence (see Table 6).

As stage I is the most common stage at melanoma diagnosis, it is critical to understand the most effective method of surveillance following treatment. This review demonstrates that current evidence is insufficient and uncertain, so further robust RCTs are required, measuring recurrence and metastases, in addition to overall survival, as outcomes, to establish the most effective surveillance strategy. No assessment of surveillance strategies among those post resection of a stage I melanoma using clinical review, imaging, or diagnostic biopsy as the main component of the strategy were identified.

Strengths and limitations

To our knowledge, this is the first systematic review of surveillance or follow-up strategies for AJCC stage I melanoma. The review followed procedures set out by the Cochrane Collaboration for conducting systematic reviews of RCTs and non-randomised studies, and was robust. 82 We conducted comprehensive searches of bibliographic databases and grey literature. All stages of the review, involving screening, data extraction and assessment of risk of bias, were conducted by at least two researchers, either in duplicate or by one researcher with checks by a second researcher. We assessed the risk of bias using the Cochrane Collaborations’s RoB 2.0 tool. 83 We contacted authors from both studies to provide further information on participants’ staging of melanoma and obtained data for stage I patients from one study. 85 We excluded the majority of potentially relevant primary full-text articles because they were non-comparative or did not assess surveillance strategies.

Because the single included study85 was conducted in a single country, the USA, the findings could be limited in applicability and generalisability. The assessment of risk of bias revealed an overall judgement of ‘some concerns’ due to attrition in the trial at 24 months. 85 Publication bias could not be investigated because of the number of studies identified, but the possibility should be considered as there may be studies that did not find positive results and remain unpublished.

Impact and implementations

Evidence for the effectiveness of surveillance and follow-up strategies for stage I melanoma is limited. A previous systematic review sought to identify the range of stage-specific surveillance practices for melanoma patients (any stage) and concluded that surveillance strategies vary around the world during the first 5 years post treatment. 25 Our review had narrower inclusion criteria with respect to study design and staging of melanoma. The paucity of this evidence in our review makes it difficult to make recommendations regarding the effectiveness of surveillance and follow-up strategies for stage I melanoma in the UK.

Search strategy

The search strategy was designed by an experienced information specialist, in collaboration with the project team. The search was designed in MEDLINE (via Ovid) according to the following concepts: [melanoma] AND [risk models] AND [prognosis]. A published and validated prognostic study filter was used. 99 The strategy used database-specific thesaurus headings, along with title and abstract keywords, with appropriate use of stemming for alternative word endings, alternative spelling and plurals. The search strategy was translated to the databases listed in Box 4. No restrictions were applied according to language or country.

An example of the search strategy used in MEDLINE can be found in Appendix 3.

Supplementary searches were limited based on the development and uptake in practice of SLNB in melanoma as a new way of diagnosing and managing patients. The first evidence of its routine use in some of the larger melanoma centres in the UK, as well as in routine practice in the USA, was around 2000. 100 We, therefore, limited the subsidiary search criteria to this date.

In addition to the databases and resources reported in Box 5, guidelines and other subsidiary journal content were handsearched and references of relevant publications were searched. This supplemented the structured documented searches, aiming to ensure that relevant studies were not overlooked as a result of selective, poorly or inaccurately indexed, or unindexed content.

Inclusion and exclusion criteria

We based inclusion and exclusion criteria on the PICOTS formula, as outlined in the following sections. 80

Data collection

Risk-of-bias assessment in included studies

Working in pairs, the risk of bias of each included paper was assessed independently by one reviewer and was checked by a second reviewer. Disagreements were resolved through discussion or with arbitration from a third member of the study team. Studies were assessed using the Prediction model Risk Of Bias ASsessment Tool (PROBAST), which addresses four domains that may influence the applicability of the prediction models (participants, predictors, outcome and analysis). 103

Missing data

We set out to report the number (per cent) of missing data for all variables/outcomes. We did not impute missing outcome data for any of our specified outcomes.

Data analysis and synthesis

We narratively synthesised the predictive performance of the prediction models by analysing the statistical measures of predictive performance. Models were assessed for discrimination, calibration and overall performance from studies providing sufficient data. The review aimed to pool the evidence of model performance by performing a meta-analysis when possible.

Quality of the evidence using GRADE approach

We had planned to assess the overall quality of evidence for key outcomes using the GRADEpro tool;84 however, it was not developed for, and the evidence suggests it performs poorly for, prediction modelling studies. 104 Given this, we decided not to use the GRADE approach.

Number of studies identified

Our search identified 25,251 records from the electronic databases. After deduplication, 20,878 records remained; following screening of the titles and abstracts, 112 full texts of potentially relevant articles were retrieved for examination. The PRISMA flow diagram outlines the study selection process and the reasons for exclusion (Figure 4). A total of 11 articles reporting 11 different risk prediction models met the full inclusion criteria of this review. 105–115 A total of 101 studies were reviewed fully and excluded for the following reasons: used single prognostic factors for model development (21%), combined stages of the disease (28%) or were not validated (51%). Details of the excluded studies are presented in Appendix 4.

FIGURE 4.

The PRISMA flow diagram for the risk prediction model review.

Characteristics of included studies

A summary of the studies and patient characteristics is presented in Table 7. Eight of the studies were conducted in the USA,106–109,111,113–115 one in the UK,112 one in Australia105 and one in Italy. 110 Seven studies used a retrospective cohort design,105–109,112,115 three used a prospective design111,113,114 and one used a retrospective cohort of prospectively collected data. 110 Patient data for development and validation of these models were taken from cancer registries, AJCC melanoma databases, clinical data from patients diagnosed and treated for melanoma or a combination of one of these sources.

Each study presented a model that could be used to predict either recurrence, metastasis or survival. The studies concerned the development of a risk or prognostic score,107,115 a nomogram,110 the Melanoma Severity Index model,105 a novel histopathological classifier,111 an electronic prediction tool,113 a classification tree,114 prognostic trees,108,109 a model focused on adding a new predictor to an established model112 and a model used to validate AJCC staging. 106

Outcome definitions and follow-up times varied across studies. The median time of follow-up ranged from 42.5 months107 to 10.3 years. 110 All studies reported outcome measures of survival. Seven studies defined survival as patients who were alive at last follow-up or who died without evidence of melanoma. 105,106,108–110,112,113 Two studies defined survival as the number of patients who are alive after diagnosis. 107,111 The other studies did not provide a definition of survival. 114,115 Risk stratification for predicting overall survival was reported in nine studies. 106–114 Patients were reportedly grouped according to tumour thickness,106,113 ulceration status,114 melanoma-specific death or survival,107,108,110,111 growth phase lesions109 or Breslow depth. 112 Two studies measured the risk of recurrence, defined and stratified either as recurrence for individual melanoma patients at different points after initial treatment107 or as local recurrence (a recurrence in the scar at the primary site). 109 Gimotty et al. 109 classified and stratified patients with local recurrences as:

• type 1 – with radial growth phase (RGP)

• type 2 – with RGP and vertical growth phase

• type 3 – without RGP.

Another study111 measured recurrence-free survival, defined as the time from diagnosis to the first recorded date of regional or distant metastases. Two studies109,110 provided outcome measures for metastasis. Gimotty et al. 109 defined metastasis as regional metastasis (in-transit dermal or subcutaneous metastases and/or nodal involvement). Maurichi et al. 110 did not provide a definition of metastasis.

Characteristics of included models

Most of the studies used regression methods for building the models. Characteristics of the models are presented in Table 8. Seven studies used the Cox proportional hazards method. 106,107,110–113,115 Two studies used a recursive partitioning algorithm used in classification and regression tree analysis. 108,109 Tsai et al. 114 used the survival tree analysis method. Baade et al. 105 used the probit regression method.

At model development stage, various methods were used across the studies to choose the variables used in the final model. Two studies used the backward procedure based on the Akaike information criterion, an estimate of the measure of the quality of available models as they relate to one another for a certain set of data. 110,112 The Akaike information criterion is used to determine what variables influence the prediction of an outcome of interest and how these variables influence the outcome by estimating several different regression models to balance the trade-offs between the complexity of a given model and its goodness of fit. 118 Two studies108,109 selected predictors based on evidence of previous validation studies and two studies107,111 performed a univariate analysis to select only the variables for which evidence of statistical significance was generated. The remaining studies did not report on the criteria used to select the candidate predictors. 105,106,113–115 The number of predictors in the studies ranged from 3115 to 11 predictors. 109 Various possible risk factors were identified, the most common being age, tumour site, tumour thickness, sex and ulceration. Other predictors identified included metastasis, mitotic rate, positive lymph node, Clark’s level, growth phase, RGP regression, microsatellites, anatomical level, presence or absence of lymphovascular invasion, tumour-infiltrating lymphocytes, histological subtype, conformation status and digital or manual area.

Risk-of-bias assessment of included studies

Table 9 summarises the risk of bias and concerns regarding applicability to the intended population and setting of the included studies. Assessments were conducted using the PROBAST. Overall, eight studies105,107,109–112,114,115 were judged to be at high risk of bias and three106,108,113 were rated as having an unclear risk of bias. Five studies106,108–110,113 were deemed to have a low risk of bias regarding applicability, another five105,111,112,114,115 were deemed to have unclear risk and one study107 was deemed to have high risk. Bias was introduced by various methods.

Concerns regarding applicability

Performance of prediction models

Model evaluation methods

All studies in the review were validated either internally or externally. Internal validation refers to the efficiency of a model developed and evaluated from the same underlying population. 128 External validation refers to how well a model predicts an outcome in a new data set, different from the development population. 129

Eight studies105–107,109–111,114,115 reported that they internally validated their models. Four studies105,111,114,115 used four different cross-validation methods. Baade et al. 105 reported that they used the internal–external method (assessing consistency across a variety of different geographical areas in the state). However, the model was not validated against an external, independent data set. Rosenbaum et al. 111 used the 10-fold method (whereby the original sample is randomly partitioned into 10 equal-sized subsamples). Tsai et al. 114 used the fivefold method (whereby the original sample is randomly partitioned into five equal-sized subsamples). Vollmer and Seigler115 used simple cross-validation methods (splitting the data sets into training/development samples and validation samples). Cochran et al. 107 used the random split-sample method, whereas Maurichi et al. 110 validated their model based on a calibration plot by assessing the congruence of expected outcomes (predicted from the model) and observed outcomes. Gimotty et al. 109 validated their model on new patients meeting study eligibility and Balch et al. 106 validated the model using the melanoma patient data used to validate the proposed AJCC staging system.

The rest of the models108,109,112 were externally validated using the geographical validation method. Predictions were calculated from the previously developed model using the training/development population and tested in new data sets different from this population in a different geographical area. Soong et al. 113 developed their model in the USA and validated it by testing it on a data set comprising patients treated at a Sydney Melanoma Unit in Australia. Gimotty et al. 108 developed their model using data from a US population-based Surveillance, Epidemiology, and End Results cancer registry (1988–2001) and validated it using patients seen by the University of Pennsylvania’s Pigmented Lesion Group (1972–2001). Saldanha et al. 112 developed their model using data from Leicester cases diagnosed between 2004 and 2011 and validated in set of cases from Nottingham University diagnosed from 2003 to 2005 and from 2008 to 2010. All three models reported measures of model performance in both the original and validation samples.

Summary of model performance assessment

This systematic review identified studies describing 11 different models developed for the prediction of recurrence, new primary tumours or metastasis in patients with AJCC stage I cutaneous melanoma following excision. The models differed in the predictors used depending on the outcome of interest and statistical measures used to assess model performances; therefore, it was inappropriate to quantitatively synthesise their results. The lack of consensus in the approach used to select predictors is reflected in the model development methods. Only six studies reported the criteria used. Two studies used the backward procedure, which starts by including all predictors at the beginning and then subsequently removing predictors based on predefined criteria. 110,112 This is a preferred method because it has the ability to eliminate redundant predictors. 130 Two studies used the univariate analysis method to screen variables. 107,111 This is the simplest method as it analyses one predictor at a time. However, this is likely to overestimate regression coefficients and overfit models. 131 Two studies108,109 used evidence from previous study reports to identify predictors, and the rest of the studies did not report on how predictors were selected; therefore, it is difficult to determine whether or not the methods used were appropriate.

Model performance measures were available for assessing discrimination in six studies,105,108–112 assessing calibration in three studies110,112,113 and assessing overall performance in two studies. 105,114 The area under the curve (AUC) of the six studies ranged from 0.59 to 0.88. The discriminative performance of the models is considered acceptable when the AUROC statistics and their equivalent are ≥ 0.7. 123 Not all studies reported the variability statistics for the AUC. However, those that did report variability statistics reported a value below this estimate; therefore, it is unclear if all the models could accurately discriminate between those with defined outcomes and those without. The three studies that assessed calibration measures all reported values ≥ 0.7, which is closer to 1 (perfect calibration). 110,112,113 This suggests that all three models have the ability to accurately generate predictions that are close to the observed outcomes. Two studies105,114 measured overall performance. Baade et al. 105 assessed this by assessing how well the model fits the data using the R² statistic. Higher R² values represent smaller differences between the observed data and the fitted values; the model by Baade et al. 105 reported an R² of 0.47 (95% CI 0.45 to 0.49), indicating that the model explains an estimated 47% of the variation. Another study114 measured the overall performance by assessing the Brier score, a statistical test used to examine the goodness of fit. This score is useful because it simultaneously captures discrimination and calibration, and summarises the magnitude of error in the probability forecast; values range from 0.0 (total accuracy) to 1.0 (total inaccuracy). 132 The score tested the performance of an integrated tree approach, as opposed to the AJCC scheme, over a period of 15 months. Although there was little difference between the scores (all were < 0.25%), the new model presented slightly better scores, suggesting that it was the preferred model to accurately predict survival rates for individual patients with localised melanoma.

Most of the studies validated their prediction models internally using data from their development set. 105–107,109–111,114,115 Research shows that, although models validated internally may show acceptable performance, it is not guaranteed that they will produce the same results in a different group of participants. 133 A few of the models were validated using external populations from other institutes. 108,112,113

All models were rated as having a high risk of bias. The main source of bias related to the inclusion of patient data from existing databases or cancer registries. Although using existing data can be beneficial in that they are cheaper and quicker to obtain, there is a risk that information may have been collected for a particular purpose, thereby including irrelevant items and perhaps not recording outcomes of interest. Some of the studies identified as such included patients with advanced stages of melanoma; this could have distorted the predictive ability of the models in their favour.

Another source of bias was the omission of statistical analysis in the estimated predictive performance of the models. There was often insufficient or no information given regarding sample size determination. Sample size is often based on the ratio of the number of individuals with the outcome event to the number of candidate predictors (EPV). Models developed from data sets with a low EPV are likely to produce biased estimates. 120 Three studies did not provide definitions for outcome measures. 110,114,115 Having a predefined outcome reflecting a clinically significant and relevant health state limits the potential of bias. 134

Strengths and limitations

The main strength of this systematic review is that, to our knowledge, it is the first to summarise the evidence presented by prediction models for AJCC stage I melanoma. The review followed procedures documented by the Cochrane Collaboration for conducting systematic reviews, the CHARMS for extracting data, the PRISMA guidelines for reporting and the PROBAST for assessing risk of bias; therefore, it is robust. We had planned to assess the overall quality of evidence using GRADE. Although the tool has been adapted for assessing overall quality in prognostic studies, it was not found to be suitable for prediction models. 104 We conducted comprehensive searches of bibliographic databases and grey literature. All stages of the review, including screening, data extraction and the risk-of-bias assessment, were conducted by either two researchers in duplicate or by one researcher with checks by a second researcher.

As with most systematic reviews, the main limitation was the quality of the published studies. None of the studies reported having followed the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD). 135 First, however, eight of the studies in the review were published between 2001 and 2014, before TRIPOD was published; therefore, they could not follow these guidelines. Second, only a small number of models were externally validated, making it difficult to determine external validity. Third, model comparisons and meta-analyses were problematic because of the variety in the predictors and statistical measures used for model performances.

Impact and implementation

The results of this systematic review highlight the relative lack of appropriate evidence levels underpinning current melanoma follow-up practice in AJCC stage I disease. As all of the studies were judged to be at high risk of bias, they cannot be recommended for estimating the probability or risk of recurrence, new primary tumours or metastases in routine clinical practice. This review identifies the need for ongoing development of risk prediction models that encompass known patient and tumoral variables in conjunction with new and developing prognostic biomarkers. These models must be developed in a way that follows biomarker development and reporting guidelines, such as TRIPOD, thus enabling appropriate critical appraisal and further assessment in the general AJCC stage I melanoma population.

The results of this review clearly outline a need for ongoing biomarker and prognostication studies and, therefore, should act as an evidence base and catalyse project development and funding. The review also demonstrates the potential impact of such studies on future follow-up guidelines and management of patients, given the relative scarcity of evidence-based practice at present.

Search strategy

The search strategy was designed by an experienced information specialist, in consultation with the project team, and was based on previous scoping of the literature. The search was designed on MEDLINE (via Ovid), using a combination of controlled medical subject heading thesaurus terms, relevant keywords and text words to identify studies of early-stage, stage I and stage II cutaneous melanoma patients and diagnostic tests. The search strategy contains appropriate use of stemming for alternative word endings, alternative spellings and plurals, and was translated to conform to other bibliographic databases using the following topic outline: [melanoma] AND [ultrasound OR biopsy] AND [surveillance]. Database-specific thesaurus headings, along with title and abstract keywords, were used and translated to other databases, altering the thesaurus headings and search syntax as appropriate. Terms relating to ocular melanoma were excluded. Identification of a suitable diagnostic filter on which to limit database results was researched seeking reliability and consistency in performance. 149 The diagnostic search filter chosen and used was published, validated and adapted for use in other databases as necessary [the study developed three search strategies (A–C), from these strategy A was selected as this was the most sensitive strategy identified]. 150 The searches of the databases (see Box 5) were run on 4 April 2019 and updated on 3 July 2019. The search strategy used in MEDLINE can be found in Appendix 5. Following deduplication, publications were limited from 1998 to July 2019, as this is when SLNB became clinically utilised as a prognostic indicator. 100

A grey literature and guidelines search was conducted to identify further relevant material not retrieved by the database searches (Box 6).

A range of guidelines on melanoma from a range of countries were also identified, and the evidence supporting relevant recommendations relating to the diagnosis of recurrent disease was checked.

Inclusion and exclusion criteria

Data collection

Risk of bias

The QUADAS-2 tool was used to assess the risk of bias of diagnostic accuracy studies. 153 This tool covered four domains: summarising the review question, tailoring the tool, constructing a flow diagram for the primary study, and judging bias and applicability.

Data analysis and synthesis

Data from the 2 × 2 tables were used to calculate sensitivity and specificity for each study. Individual study results were presented graphically by plotting estimates of sensitivities and specificities in forest plots. If more than one threshold was reported, data from one threshold were to be chosen to be incorporated into a meta-analysis. Meta-analysis of pairs of sensitivity and specificity values was planned using a bivariate random-effects approach. This approach would enable the calculation of summary estimates of sensitivity and specificity, while correctly dealing with the different sources of variation. 154 The planned meta-analysis was not conducted, as each study used different index tests and would be used at a different stage in the diagnostic work-up.

Sources of heterogeneity in studies

Risk factors for melanoma progression and prognosis obtained from demographic data in the studies were to be included as potential sources of patient heterogeneity between studies. A range of potential factors influencing heterogeneity may include sex, age of participants, tumour characteristics, presence of ulceration, stage of disease, site of primary tumour (trunk, lower limbs, upper limbs, head or neck, hand or foot), clinical node status at follow-up (post operative), sentinel lymph node status at follow-up (post operative) and comorbidities. Potential sources of heterogeneity in reference tests may include the experience of surgeons, the method of sampling (either cytology or biopsy) and preparation techniques of the tissue sample [e.g. fresh-frozen section (cryosection) or paraffin section]. Heterogeneity of the index tests may arise from variations in the clinical pathway, including disease stage; professionals involved (e.g. radiographers, radiologists or clinicians); experience of operators performing index and reference tests; differing frequencies of ultrasonography instruments, manufacturers and models; and tumour depth and spread, ulceration, regions of interest for ultrasonography and anatomical sites of lesions.

Heterogeneity was to be investigated in the first instance through visual examination of forest plots of sensitivities and specificities and through visual examination of the ROC plot of the raw data. Heterogeneity was also to be assessed statistically using the I² statistic. 155 Had suitable data to meta-analyse been identified, sensitivity analyses exploring heterogeneity would have been conducted. Likewise, subgroup analyses by sample size of the study would have been conducted, as sample size is known to influence sensitivity and specificity. 156,157

Number of studies identified

Afer deduplication, the electronic database searches retrieved 2226 records and a further 24 from search updates. One additional study was included from reference lists of systematic reviews. From screening these citations, 106 primary studies, including conference abstracts, were identified as potentially relevant. Full-text articles were then retrieved for review. Five studies were non-English language (written in French, Hungarian, Japanese, Russian and Spanish). Seventeen citations were conference abstracts; each was checked for journal publication of the full reports. One foreign language paper was unable to be translated. 158 Study authors of five studies that reported combined data across stages were contacted for more information. 24,159–162

Following this process, two English-language studies met the inclusion criteria and were assessed. 163,164 These reported the diagnostic accuracy of FNB163 and the diagnostic accuracy of high-frequency ultrasonography. 164 We wrote to five study authors to seek further clarification and a breakdown by stage I disease to attempt to include further data.

In addition, other than the five studies awaiting classification and one needing translation, 98 full-text studies or conference abstracts that did not meet the inclusion criteria are recorded in Appendix 6, with reasons for exclusion. Studies were excluded based on the following: no analysis by disease stage or Breslow thickness (n = 29), stage not reported (n = 22), preoperative staging (n = 13), stages combined (n = 10), insufficient patients or data (n = 5), not diagnostic accuracy study (n = 4), no relevant outcomes (n = 3), advanced stage (n = 3), no diagnostic test (n = 1), letters (n = 3), review (n = 3), treatment (n = 1) and animal study (n = 1). Studies could meet one or more of these criteria. The PRISMA flow diagram in Figure 5 outlines the study selection process and the reasons for exclusion.

FIGURE 5.

The PRISMA flow diagram for the diagnostic accuracy review. DTA, diagnostic test accuracy.

Characteristics of included studies

One of the two included studies was conducted in Australia,163 and the other in Germany. 164 The two studies considered different diagnostic tools used at different points in the care pathway. The characteristics of each study are presented in Table 10.

Doubrovsky et al. 163 evaluated the diagnostic accuracy of FNB, which is analagous to FNAC, to detect metastatic melanoma. They used a retrospective cohort study design, with data collected between January 1992 and December 2002 from the Royal Alfred Hospital in Sydney, NSW, Australia. The sample comprised 1582 confirmed FNBs at melanoma stages I–IV. A total of 323 confirmed cases (20%, 323/1582) were included in an analysis of stage I disease. Males accounted for 63% of participants and ages ranged from 10 to ≥ 81 years. Diagnostic accuracy was evaluated by AJCC stage, the anatomical sites of lesions, use of immunostaining, year, sex, age, FNB attempts, needle size, presence of necrosis, pathologist case load, primary Breslow thickness, primary lesion ulceration status, primary lesion mitotic rate, histological subtype and predominant subtype.

Krüger et al. 164 evaluated the diagnostic accuracy of high-resolution ultrasonography of the lymph nodes to detect locoregional lymph node metastases. A prospective cohort design was used, with data collected prospectively between 2004 and 2008 from the Dermatology Department in Göttingen, Germany. Participants recruited had stage I–IV disease (AJCC 2002), and a median age of 58 years. Forty-eight per cent were male. The diagnostic accuracy of combined clinical and sonographic examinations was evaluated in 433 patients. The sample was composed of 1314 investigations in patients with melanoma stages I–IV. A total of 669 investigations (51%, 669/1314) were included in an analysis of stage I disease. An average of three paired investigations (clinical and sonographic) were performed for each participant on the same day. With respect to stage I disease, we estimated that 223 participants received 669 paired investigations. Diagnostic accuracy was evaluated by AJCC stage, melanoma subtype, lymph node dissection status and lymph node surgery before investigation.

Risk of bias

Risk-of-bias assessments were performed using the QUADAS-2 tool across four domains. 153 The results of this are summarised in Table 11.

Diagnostic accuracy

A meta-analysis was not conducted as there was only one study for each index test. The results of the studies by Doubrovsky et al. 163 and Krüger et al. 164 for diagnostic accuracy of FNB and ultrasonography, respectively, are shown in Table 12. Forest plots are also presented for the sensitivity and specificity of the index tests for the two studies (Figures 6 and 7).

FIGURE 6.

Forest plot of sensitivity and specificity of FNB for detection of stage I melanoma recurrence. 163 FN, false negative; FP, false positive; TN, true negative; TP, true positive.

FIGURE 7.

Forest plot of sensitivity and specificity of ultrasonography for detection of stage I melanoma recurrence. 164 FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Summary of findings

This systematic review sought to estimate the diagnostic accuracy of ultrasonography and FNAC procedures, routinely used to detect recurrence and metastases during follow-up of patients initially diagnosed with stage I cutaneous melanoma. Comprehensive literature searches were performed in a range of bibliographic databases from 1998 to July 2019, and were complemented by grey literature searches for guidelines and other potentially relevant literature. Searches were restricted to 1998, which coincided with the introduction of SLNB assessment following melanoma diagnosis in some centres internationally. Despite this extensive searching, only two studies met the inclusion criteria. As SLNB involves the assessment of an initial lymph node basin, and often the complete removal of all associated nodes when a positive SLNB is identified, this would probably have an impact on the routine clinician examination findings of patients, as well as the potential choice and application of further investigation modalities in such patients. By limiting the searches to dates after this change in routine management, a more appropriate and contemporaneous assessment of the evidence could be made.

The two identified studies considered different tests (lymph node ultrasonography and FNB). 163,164 The findings reported for diagnostic accuracy of FNB in patients who were diagnosed initially with stage I melanoma were comparable to those reported for patients at stages II–IV. 163 Doubrovsky et al. 163 reported that the sensitivity was positively correlated with the following factors: the use of immunostaining (currently widely accepted168); if the cytopathologists had performed > 500 FNBs; the case mix, especially in patients presenting with ulcerated primary melanomas; and lesions located in the skin and subcutis. False-negative findings were reported to be associated with masses located in the axillary lymph nodes and when sampling required more than one needle pass.

The diagnostic accuracy of ultrasonography for stages II–IV was reported to be comparable to that for stage I. Krüger et al. 164 found that the sensitivity and specificity varied between melanoma subtypes (e.g. superficial or nodular).

In line with Cochrane guidance82 on the use of narrative summary of findings tables, we did not include one for the systematic review of diagnostic accuracy of high-resolution ultrasonography, fine-needle aspiration or core biopsy to detect local recurrence, satellite lesions, new primary lesions, in-transit metastases, locoregional lymph node metastases or metastatic melanoma during surveillance. We did not calculate pooled results in this review; instead we presented the results of a single study. As a result, a narrative SoF table would not include any certainty of evidence judgements; thus, we felt that it was an unnecessary addition. Furthermore, heterogeneity is expected in diagnostic test accuracy reviews; therefore, with just one study included, by reporting on each test, we cannot make any inferences from the results presented.

Comparison of findings with other reviews

Four reviews retrieved by our searches had investigated the diagnostic accuracy of high-resolution ultrasonography or FNAC/core biopsy. None of the reviews conducted analyses by disease stage. Hall et al. 148 identified 10 studies published between 1980 and 2007 for inclusion. The review reported summary estimates for both palpation and ultrasonography-guided FNAC together as a sensitivity of 0.97 (95% CI 0.95 to 0.98) and a specificity of 0.98 (95% CI 0.98 to 1.00). The positive likelihood ratio was 58 (95% CI 23 to 139). These data are similar to those reported by Doubrovsky et al. 163 on FNB in stage I patients.

A second review summarised the diagnostic accuracy of different imaging techniques for primary staging and surveillance of lymph nodes and distant metastases. 147 This review included 74 studies, of which 21 considered ultrasonography. Ultrasonography was reported at primary staging and surveillance. Twenty-one studies that considered ultrasonography and which were published between 1990 and 2007 met inclusion criteria for the review. The mean QUADAS-2 score across the 21 included studies was 5.8, with a SD of ± 2.5, from a potential score of 14. Ultrasonography during surveillance imaging had a median sensitivity of 0.96 [95% credible interval (CrI) 0.85 to 0.99] and median specificity of 0.99 (95% CrI 0.95 to 1.00). Although no analysis was conducted by disease stage in this review, the results were similar to those reported by Krüger et al. 164

Another review compared ultrasonography and palpation to detect nodal invasion in patients with stage I–III melanoma. 169 Twelve studies published between 1997 and 2003 met the inclusion criteria. Positive likelihood ratios were 41.9 (95% CI 29 to 75) for ultrasonography and 4.55 (95% CI 2 to 18) for palpation. Negative likelihood ratios were 0.024 (95% CI 0.01 to 0.03) for ultrasonography and 0.22 (95% CI 0.06 to 0.31) for palpation.

A Cochrane review of ultrasonography and other imaging techniques for staging and re-staging of adults with cutaneous melanoma, at any stage of initial disease, included 11 studies. 170 It reported a summary sensitivity for ultrasonography alone of 0.35 (95% CI 0.17 to 0.59) and a specificity of 0.94 (95% CI 0.86 to 0 98) for detection of regional lymph node metastases before SLNB. A combination of pre-SLNB ultrasonography with FNAC reduced sensitivity to 0.18 (95% CI 0.036 to 0.57) and increased specificity to 1.00 (95% CI 0.99 to 1.00). These findings are at odds with those reported by Xing et al. 147 and those reported by Krüger et al. 164

Overall, it is likely that differences in sensitivity between the reviews may be due to heterogeneity between studies of mixed populations, for example patients at different stages of disease and variations in case mix of melanoma subtypes.

Strengths and limitations

This review sought to systematically identify all relevant studies looking at the diagnostic performance of tests used to detect recurrance after treatment for stage I melanoma. It followed a prespecified protocol and conducted rigorous searches. The methods for the review correspond to best-practice recommendations.

The review is limited in a number of respects. First, it considered only two diagnostic tests. The rationale for this was based on clincial opinion about which tests might be viable for the NHS to consider, but it also took into account judgements about which tests would not be relevant; for example, sophisicated imaging was excluded for the review. Although these tests are not recommended by many guidelines currently, whether or not those recommendations should change has not been assessed (although recent surveillance reviews by NICE suggest not, at least for stage I disease). 54

The dates of the searches were limited from 1998 to July 2019. This date was chosen because this is when SLNB became clinically utilised as a prognostic indicator. 100 However, it is possible that relevant studies could have been published before this date. The applicability of the findings of any such study to current practice is unclear, as staging criteria have changed several times since 1998.

The key limitation of this review is the lack of recent evidence on the diagnostic performance of ultrasonography and FNB in the target population. Planned meta-analyses were not possible as the two studies included investigated different index tests. Furthermore, Krüger et al. 164 identified only one participant initially treated for stage I disease as a true positive. Consequently, the CI for sensitivity was very wide (95% CI 0.03 to 1.00), indicating that the study is non-informative.

Development of a cost-effectiveness model

The development of the cost-effectiveness model started with a targeted review of the literature (see Appendix 7), which identified 15 economic evaluation studies of potential relevance. 24,171–184 None of these directly addressed the study question posed here. Therefore, a de novo model was developed to compare potential surveillance strategies provided by the NHS.

Using the information gathered from the systematic reviews reported in Chapters 3–5 and extensive dialogue with the clinical members of the study team, we developed a Markov microsimulation model to evaluate the cost-effectiveness of plausible surveillance options based on differing clinical specialty, interval between visits and duration of follow-up. We used the most relevant UK epidemiological statistics and best available data from the literature to populate the model, reflecting the clinical reality of care for these patients. This included the probability of self-detection, ‘false alarms’ that result in unscheduled emergency appointments and a pathway of care that included investigational diagnosis and any subsequent treatment.

Model overview

The cost-effectiveness of surveillance strategies for stage I melanoma patients was assessed using a Markov microsimulation model. A microsimulation model is a form of economic modelling whereby individuals are simulated through the model one by one, rather than as a cohort. Individual results are then stored and the experience of the cohort is obtained by aggregating the individual results. 185 Markov health states are used to simply describe a patient’s health status. In cancer models, disease progression will occur and the progression of the disease is described by a set of health states, typically of increasing severity. These states are commonly described as tunnel states. The minimum period of time (defined as the cycle length) that a person can be in a state before moving to another state was taken as 1 month. A 1-month cycle length was chosen as surveillance strategies were based on monthly surveillance interval differences.

The decision to use a microsimulation model as opposed to a simpler cohort-based Markov model186 was based on the properties of a microsimulation. In a microsimulation, the memorylessness restrictions of a cohort-based Markov model can be overcome. Building the Markov microsimulation model in TreeAge Pro 2019 R1.0 (TreeAge Software Inc., Williamstown, MA, USA) enabled us to keep track of the treatment history, time since last treatment and recurrence of simulated patients, which all have an impact on the cost and effect estimates. In a cohort-based Markov model, movement from a given state is not affected by how the person arrived in that state (the model has no memory of prior events or care).

The microsimulation model describes the care pathway of individuals from the point when they received treatment (i.e. wide local excision) for stage I melanoma. The model allows for alternative surveillance strategies to be compared. The model seeks to estimate their longer-term (i.e. lifetime) costs and consequences, including those that might arise from any subsequent melanoma diagnosis, be it for new or recurrent disease. Surveillance strategies were compared based on lifetime costs and health outcomes, expressed in terms of quality-adjusted life-years (QALYs). These data were then used to estimate cost-effectiveness. The costs were estimated in Great British pounds from an NHS and Personal Social Service perspective for the financial year 2017–18. As both costs and QALYs were estimated over a lifetime time horizon, they were discounted using a 3.5% annual discount rate, as per NICE’s reference case. 187

We used anonymised individual data from 160 early-stage patients obtained from a local hospital (University Hospital of North Durham, ethics approval: NREC 19/NE/004) to populate the basic demographic characteristics (mean age 56 years, range 17–98 years; ratio of males to females 36% : 64%). Stage I patients (ratio of stage IA to stage IB was 76% : 24%) had up to 8 years’ follow-up on recurrence (15%) and mortality (melanoma-specific mortality over that time was 6% and all-cause mortality was 17%). Any additional melanoma skin cancer statistics were taken from the Cancer Research UK website. 188

Model structure

Once discharged into follow-up care, a patient enters into the model as disease free. Surveillance is captured as an event undertaken at discrete time intervals. At any given point in time, an individual is in one of four Markov states: disease free, recurrence (diagnosed stage IA–IV), death from melanoma or death from other causes. There is a chance that, when a recurrence occurs and is not detected in the monthly cycle, disease progression will occur. This is depicted as a tunnel state in Figure 8. The detailed model structure is shown in Appendix 8.

FIGURE 8.

Description of a simplified version of the Markov microsimulation model.

In the microsimulation model, each individual has a chance (probability) of recurrence, metastasis and occurrence of new primary tumours in a given monthly cycle. In the model, any future melanoma diagnosis can be self-detected in a cycle by an individual/partner. This will result in an unscheduled emergency visit. There is also a chance that the emergency visit is a ‘false alarm’ and that the patient does not have melanoma. If the individual was scheduled for a surveillance visit that month, there is a chance that the attending clinician will detect the melanoma. There is also a chance that a clinician will fail to diagnose a melanoma if one is present.

For all suspicious lesions, the diagnostic process starts with a history and physical examination with dermoscopy by a clinician (note that, although it is recommended by NICE that all examinations should use dermoscopy, anecdotally there is varied uptake across different specialist settings). The use of subsequent investigations is determined by the findings of the prior investigations (i.e. if a local excision biopsy is positive, then SLNB will be performed; if SLNB is positive, then CT is performed).

If a cancer is present but is not detected during a 1-month cycle, then the cancer may progress. If this occurs, then, in the next cycle of the model, the individual moves into a tunnel health state. The tunnel state itself describes the AJCC staging of melanoma.

The whole process described here for the movement of an individual through the model is then repeated for each individual in the cohort, thereby generating individual life histories of the population modelled.

Model inputs

To provide estimates of relative cost-effectiveness, the model required estimated values for a range of different types of parameters. We aimed to use the best available values derived in a systematic and reproducible manner to avoid bias caused by the distorted and selected use of data. 190 We focused on identifying the most relevant data to the decision problem, which is the comparison of alternative surveillance regimens after treatment (i.e. local excision) for stage I melanoma.

We assembled the different types of data required for the economic model from analyses of existing data sets, a series of systematic reviews and focused searches for specific types of data (see Appendix 9). In brief, the broad types of data required to populate the economic model were as follows:

• patient behaviour with regards to surveillance and follow-up (e.g. self-detection/self-diagnosis and ‘false alarms’ resulting in emergency visits)

• clinical pathway once diagnosed with melanoma

• the performance of different regimens (e.g. clinical examinations) in terms of diagnostic accuracy (see Diagnostic accuracy)

• the prevalence, incidence and risk of progression of the disease, risk of recurrence [i.e. its epidemiology and natural history (see Natural history of melanoma)]

• resource use and unit costs required to estimate the costs of alternative surveillance regimens; the specific parameters and methods used to provide estimates that are relevant to the UK context (see Resource use and unit costs)

• health-state utilities (see Health utilities).

Diagnostic accuracy

Natural history of melanoma

Resource use and unit costs

The costs of surveillance were broken down into the following cost categories: scheduled surveillance (e.g. consultation time), further invasive tests (e.g. local biopsy, SLNB) and treatment (e.g. targeted therapy and immunotherapy).

We derived data on the costs incurred for the different surveillance regimens and their consequences from routine data sources, such as the NHS reference costs. 212 Table 14 shows the cost estimates used in the economic model. The costs of drug treatment, targeted therapy [trametinib in combination with dabrafenib, NICE Technology Appraisal (TA) 396213] for AJCC stage III and immunotherapy for AJCC stage IV disease (nivolumab in combination with ipilimumab, NICE TA400214) were obtained from the British National Formulary as per current clinical treatment regimen. 215 Gamma distributions were assigned to cost values for stages III and IV treatments (see Appendix 9, Table 38).

Health utilities

As melanoma and its treatment affects not only survival, but also quality of life, a focused search of the literature and other relevant sources [e.g. the Sheffield School of Health and Related Research Health Utilities Database (ScHARRHUD) and the Health Economics Research Centre (HERC), Oxford, database of mapping studies] was initiated (see Appendix 9, Table 37). However, in 2018, a systematic review and meta-analysis of melanoma utility weights based on 33 studies was published. 216 This review included studies from Australia, Europe and North America, and sought to define post-treatment utilities by stage with time of data collection component up to 3 months, 3–12 months and > 12 months. The synthesis of health-state utilities is controversial. Although it aims to generate a more accurate estimate of the mean health-state utility and the associated uncertainty, a formal synthesis may not be meaningful considering the variability in measures [e.g. EuroQol-5 Dimensions (EQ-5D) vs. Short Form questionnaire-6 Dimensions], valuation method, types of anchors used, the country where the valuation was done and who provided the preference weights. 217

As no relevant utility values derived from a UK population were available, the published values from the meta-analysis were used in the economic model (Table 15). It is further assumed that individuals in the ‘disease-free’ state have a health-state utility value of full health (utility score = 1), as per Wilson et al. 182 These are values without age adjustment. These values were chosen for the model as they are published in the literature and are consistent across all strategies. Beta distributions were assigned to utility values in the base-case analysis.

Main modelling assumptions

This section provides a brief summary of the key assumptions made when developing the economic model. It was assumed that the time interval between initial surgery and any subsequent event/treatment is 1 month. No utility decrements associated with undertaking any of the tests (local biopsy, SLNB, etc.) or treatments (immunotherapies have severe adverse effects) were incorporated in the model. It was assumed that the delay in recurrence/new primary diagnosis would eventually be captured by self-detection as advanced disease symptoms become evident. It was also assumed that individuals attend all of their scheduled follow-up appointments, that the initial surgical excision is curative and that individuals enter the model as disease free. Recurrence rates for stages IIIA, IIIB, IIIC and IV were assumed to be similar to the recurrence rates for stage IIC. Estimates of melanoma-specific risk of death for each stage were taken from Wilson et al. 182 It was assumed that patients received systemic treatment until treatment failure or death (Dr Janine Graham, personal communication). In the model, it was assumed that patients were BRAF positive and received dabrafenib and trametinib targeted therapy. The literature suggests that 50% of metastatic melanomas are BRAF positive. 218 If patients were BRAF negative, they would probably receive immunotherapy (e.g. ipilimumab). 219 The assumption that stages III and IV are now treated with new (expensive) therapies is captured in the model and the assumed costs are approximations. Finally, the ‘no-surveillance’ option was not considered in the model, as it was deemed to be unacceptable to patients by the advisory board.

Incremental cost-effectiveness analysis

Sensitivity analyses

Both deterministic and probabilistic sensitivity analyses were used to explore parameter uncertainty surrounding estimates of cost-effectiveness. Threshold analyses were used to explore further considerations around the main parameters of interest.

Value-of-information analysis

A critical role of the PSA was that it facilitated the estimation of the value of further research. Therefore, in addition to exploring the cost-effectiveness of different follow-up strategies for individuals with stages IA and IB melanoma, the main uncertainties were quantified to inform any future direction of research. This was explored through the expected value of partial perfect information (EVPPI) analysis. EVPPI analysis can be used to estimate the expected value of removing the uncertainty of specific parameters or groups of parameters in a given model. It can be used to help guide decisions on where future research should be directed to obtain more precise and reliable estimates of specific parameters. 225 EVPPI places an upper value on conducting further research on a specific area of information. It indicates the maximum gain that could be obtained by removing all uncertainty in that area. In reality, any piece of future research would remove only part of that uncertainty, but the approach can still help indicate where the most gain could be made from further research.

To calculate the population EVPPI, the individual EVPPI is first calculated by the model. The individual EVPPI is then multiplied by the size of the population that will be affected by the information over the anticipated lifetime of the technology. For the estimation of the size of the population, the annual prevalence (the number of patients affected by the decision each year) of stage I disease is needed. The assumption is that the annual incidence of stage I melanoma is, in essence, the number of patients affected by the decision each year. In 2017, 8555 patients were diagnosed with stage I disease. 226 The proportion (63% for IA vs. 37% for IB) was provided by Leiter et al. 209 in their work based in Germany.

Two-level simulations were conducted to estimate the EVPPI. The first level happened in the microsimulation by randomly selecting individuals of different age and sex from the individual-level data. Then, each selected individual was simulated 10,000 times (PSA) and values for the parameters were selected from the prespecified distributions. Results from the PSA were then used in the Sheffield Accelerated Value of Information (SAVI) tool to estimate the EVPPI. 227

Four groups of parameters were considered in the EVPPI analysis:

1. health utility values

2. diagnostic accuracy of health-care professional to detect a melanoma

3. probability of transitioning between stages

4. recurrence of melanoma.

Model validation

Model validation was achieved by clinical expert concurrence, by a review by modelling colleagues at Newcastle University and by being transparent in reporting. As laid out in a review on validating cost-effectiveness models, ‘internal validity’ related to input parameters, whereas ‘descriptive validity’ of a clinical pathway requires analysts to make trade-offs between accuracy, complexity and fulfilling the purpose of the model. 228 For this project, the objective was to compare surveillance strategies for stage I disease, rather than capturing the complex and evolving management of advanced-stage disease.

Base-case analysis results

Given the computational burden of running the model with 10,000 PSA simulations over 87 strategies, the model was first run deterministically to identify 20 strategies that were considered to have the most promise of being cost-effective for the stage IA model. This was repeated for the stage IB model. These strategies were then used in the base-case analysis.

Base-case analysis: stage IA

Monte Carlo simulation was performed to obtain probabilistic estimates of the cost-effectiveness of the different surveillance strategies, compared with NICE’s recommended follow-up schedule. The 20 strategies included in the analysis are presented in Table 16.

TABLE 16 - Strategies for surveillance of stage IA melanoma included in the base-case analysis

Strategy	Duration of follow-up	Intervals of follow-up each year	Health-care professional undertaking screening
1–NICE	1 year	Every 3 months for 1 year	Dermatologist
4	10 years	Every 3 months for 1 year and every 6 months thereafter	Dermatologist
7	10 years	Every 3 months the first year and every 12 months thereafter	Dermatologist
14	10 years	Every 4 months the first year and every 12 months thereafter	Dermatologist
15	1 year	Every 6 months for 1 year	Dermatologist
16	3 years	Every 6 months for 3 years	Dermatologist
19	3 years	Every 6 months for 1 year and every 12 months thereafter	Dermatologist
21	10 years	Every 6 months for 1 year and every 12 months thereafter	Dermatologist
22	1 year	Once for 1 year	Dermatologist
23	3 years	Every 12 months for 3 years	Dermatologist
29	10 years	Every 3 months for 1 year and every 6 months thereafter	Surgeon
32	10 years	Every 3 months for 1 year and every 12 months thereafter	Surgeon
37	3 years	Every 4 months for 1 year and every 12 months thereafter	Surgeon
39	10 years	Every 4 months for 1 year and every 12 months thereafter	Surgeon
40	1 year	Every 6 months for 1 year	Surgeon
41	3 years	Every 6 months for 3 years	Surgeon
44	3 years	Every 6 months for 1 year and every 12 months thereafter	Surgeon
46	10 years	Every 6 months for 1 year and every 12 months thereafter	Surgeon
47	1 year	Once for 1 year	Surgeon
48	3 years	Every 12 months for 3 years	Surgeon

Table 17 and Figure 9 (which excludes strategies that are, on average, dominated) report the results of the base-case analyses for the average patient treated for stage IA melanoma. Results are reported in terms of total and incremental costs and effectiveness, incremental cost per QALY (ICER) and NMB (were society’s WTP for a QALY (λ) = £20,000 per QALY), alongside the probability that each strategy would have the highest NMB for different threshold values used by NICE.

TABLE 17 - Results of the base-case analysis for patients treated for stage IA melanoma

See Table 16 for strategy description.

This strategy has a higher ICER (relative to a third option) and fewer benefits than the alternative. Strategy 22 (common baseline as it is least costly option) is the comparison or reference strategy.

This strategy is always more costly and less beneficial than comparator strategies.

Compared with strategy 22.

Compared with strategy 19.

Compared with strategy 7.

Strategya	Cost (£)	Incremental cost (£)	QALY	Incremental QALY	ICER (£) (Δcost/ΔQALY)	NMB (£)	Probability of being cost-effective for different threshold values for society’s WTP for a QALY (%)
							£20,000	£30,000	£50,000
22	8476		14.74			286,321	13.2	9.7	6.2
15	8824	348	14.75	0.01	33,367 (extendedly dominatedb)	286,181	7.1	5.6	4.1
23	8874	50	14.73	–0.02	Absolutely dominatedc	285,820	10.8	9.5	7.6
19	9215	391	14.77	0.02	25,894 d	286,153	9.8	8.4	6.7
1-NICE	9336	121	14.76	–0.00	Absolutely dominatedc	285,952	12.4	10.3	8.2
16	9613	399	14.77	0.00	199,677(extendedly dominatedb)	285,794	7.2	6.8	6.0
21	10,254	641	14.76	–0.01	Absolutely dominatedc	284,952	8.3	8.2	7.3
14	10,494	881	14.76	–0.01	Absolutely dominatedc	284,747	11.3	11.4	10.4
7	10,768	1154	14.78	0.01	103,890 e	284,899	7.8	7.9	7.1
4	12,222	1455	14.78	0.00	1,462,441 (extendedly dominatedb)	283,464	6.3	7.6	8.1
47	13,909	1687	14.74	–0.04	Absolutely dominatedc	280,893	1.7	2.7	3.6
40	14,427	2205	14.75	–0.04	Absolutely dominatedc	280,522	0.1	0.8	2.2
48	14,579	2356	14.74	–0.05	Absolutely dominatedc	280,173	0.7	1.4	2.2
44	15,072	2850	14.76	–0.03	Absolutely dominatedc	280,089	0.6	1.5	3.0
37	15,518	3296	14.76	–0.02	Absolutely dominatedc	279,749	0.7	1.5	2.6
41	15,754	3532	14.76	–0.02	Absolutely dominatedc	279,501	0.3	0.8	1.7
46	16,923	4701	14.77	–0.02	Absolutely dominatedc	278,449	0.5	1.2	3.2
39	17,367	5145	14.78	–0.00	Absolutely dominatedc	278,227	0.6	2.1	4.0
32	17,765	5543	14.78	0.00	10,081,791 (extendedly dominatedb)	277,932	0.3	1.9	3.6
29	20,319	2554	14.79	0.01	1,335,810 f	275,490	0.3	0.7	2.2

FIGURE 9.

The CEAC for stage IA melanoma (dominated strategies excluded). See Table 16 for strategy description.

For stage IA, over a lifetime time horizon, none of the strategies involving a specialist dermatological nurse is likely to be considered cost-effective. It is worth noting that, overall, all strategies for stage IA produced very similar QALYs. This might be expected, given the relatively low chance of recurrence or metastasis and the high rate of self-diagnosis (it has been assumed that 60% of all melanomas are self-diagnosed within 12 months). Therefore, the lifetime costs are the main driver for evaluating surveillance strategies.

Strategy 22, which is follow-up once at 12 months by a dermatologist, is, on average, the least costly surveillance strategy (£8476 per person) and generates 14.74 QALYs. This strategy also has the highest NMB (£286,321) at a WTP threshold of £20,000 per QALY. Should the NHS wish to make a decision on cost alone, then this strategy has approximately a 60% chance of being considered the most cost-effective. As society’s willingness for a QALY increases, other, more effective, strategies become increasingly cost-effective. At a £20,000 threshold for society’s WTP for a QALY, strategy 22 has almost a 13% probability of being considered cost-effective. At the same threshold, three other strategies have a probability of > 10% of being cost-effective. These are strategy 23 (follow-up every 12 months for 3 years), strategy 1 (the NICE-recommended strategy: every 3 months for 1 year) and strategy 14 (surveillance every 4 months for the first year and then every 12 months for the next 9 years). Two strategies have a probability of 8–10% of being cost-effective: strategy 19 (every 6 months for 1 year and every 12 months for the next 2 years) and strategy 21 (every 6 months for 1 year and every 12 months for the next 9 years).

Given that 20 strategies are being compared, should each strategy be equally likely to be considered cost-effective, it would be expected that each strategy would have only a 5% chance of being considered cost-effective. Several of the dermatology-based strategies (including the NICE-recommended strategy) have probabilities greater than this, and four strategies have probabilities of > 10%. These four include strategies that are both less intensive and more intensive than the strategy recommended by NICE.

Base-case analyses: stage IB

The 20 strategies included in the analysis for stage IB disease are presented in Table 18. As with strategies for stage IA, strategies involving the specialist dermatological nurse were judged very unlikely to be considered cost-effective and were not included in the base-case analysis.

TABLE 18 - Strategies for surveillance of stage IB melanoma included in the base-case analysis

Strategy	Duration of follow-up	Intervals of follow-up each year	Health-care professional undertaking screening
1–NICE	5 years	Every 3 months for the first 3 years and every 6 months thereafter	Dermatologist
2	3 years	Every 3 months for 3 years	Dermatologist
4	20 years	Every 3 months for the first 3 years and every 6 months thereafter	Dermatologist
5	5 years	Every 3 months for the first 3 years and every 12 months thereafter	Dermatologist
8	3 years	Every 4 months for 3 years	Dermatologist
9	5 years	Every 4 months for the first 3 years and every 6 months thereafter	Dermatologist
11	20 years	Every 4 months for the first 3 years and every 6 months thereafter	Dermatologist
15	3 years	Every 6 months for 3 years	Dermatologist
18	20 years	Every 6 months for 20 years	Dermatologist
23	5 years	Every 12 months for 5 years	Dermatologist
25	20 years	Every 12 months for 20 years	Dermatologist
29	20 years	Every 3 months for the first 3 years and every 6 months thereafter	Surgeon
77	1 year	Once for 1 year	Dermatologist
78	1 year	Once for 1 year	Surgeon
80	2 years	Every 12 months for 2 years	Dermatologist
81	2 years	Every 12 months for 2 years	Surgeon
82	2 years	Every 6 months for 2 years	Surgeon
83	2 years	Every 6 months for 2 years	Dermatologist
86	1 year	Every 6 months for 1 year	Dermatologist
87	1 year	Every 6 months for 1 year	Surgeon

Table 19 and Figure 10 report the results of the base-case analyses for stage IB melanoma patients. None of the strategies involving a surgeon conducting the surveillance had a probability of > 1% of being cost-effective across any of the values for society’s WTP for a QALY. The strategy recommended by NICE (strategy 1, surveillance by a dermatologist every 3 months for the first 3 years and then every 6 months thereafter for the next 2 years) also never had a probability of > 6% of being cost-effective over any of the values for society’s WTP for a QALY. As with the analysis for stage IA disease, given that 20 strategies are being compared, should each strategy be equally likely to be considered cost-effective, it would be expected that each strategy would have only a 5% chance of being considered cost-effective.

TABLE 19 - Results of the base-case analysis for melanoma treated for stage IB melanoma

See Table 18 for strategy description.

This strategy has a higher ICER (relative to a third option) and fewer benefits than the alternative. Strategy 77 (common baseline as it is least costly option) is the comparison or reference strategy.

This strategy is always more costly and less beneficial than comparator strategies.

Strategya	Cost (£)	Incremental cost (£)	QALY	Incremental QALY	ICER (£) (Δcost/ΔQALY)	NMB (£)	Probability of being cost-effective for different threshold values for society’s WTP for a QALY (%)
							£20,000	£30,000	£50,000
77	9536		14.57	0.00		281,945	12.9	9.7	6.8
80	9833	297	14.61	0.04	7818	282,408	9.8	7.9	6.2
86	9971	141	14.62	0.00	47,383 (extendedly dominatedb)	282,326	10.4	8.4	6.0
23	10,320	346	14.60	–0.01	Absolutely dominatedc	281,727	8.6	7.7	6.1
83	10,475	501	14.63	0.02	31,172	282,146	6.2	5.2	4.0
15	10,696	221	14.62	–0.01	Absolutely dominatedc	281,642	4.5	3.6	3.2
8	11,420	945	14.61	–0.02	Absolutely dominatedc	280,811	6.5	5.7	5.0
25	11,910	1435	14.63	–0.01	Absolutely dominatedc	280,600	7.9	7.7	7.1
9	12,158	1683	14.62	–0.01	Absolutely dominatedc	280,332	6.5	6.8	7.1
2	12,187	1712	14.64	0.01	175,956 (extendedly dominatedb)	280,629	3.6	3.6	4.0
5	12,485	297	14.64	–0.01	Absolutely dominatedc	280,317	3.6	4.1	4.4
1–NICE	12,748	561	14.64	0.00	3,079,045 (extendedly dominatedb)	280,072	4.1	4.9	5.5
18	14,607	1858	14.65	0.01	139,038 (extendedly dominatedb)	278,480	5.5	7.1	8.1
78	14,970	363	14.59	–0.06	Absolutely dominatedc	276,838	0.2	1.6	2.8
11	15,352	745	14.65	–0.00	Absolutely dominatedc	277,657	4.7	6.1	7.5
81	15,391	784	14.62	–0.04	Absolutely dominatedc	276,926	0.7	1.2	2.0
87	15,552	946	14.62	–0.04	Absolutely dominatedc	276,748	0.8	1.5	2.3
4	16,045	1438	14.69	0.03	45,598	277,673	2.8	4.6	6.2
82	16,353	309	14.64	–0.05	Absolutely dominatedc	276,400	0.7	1.8	2.6
29	25,871	9826	14.69	–0.00	Absolutely dominatedc	267,840	0.0	0.8	3.1

FIGURE 10.

The CEAC for stage IB strategies (dominated strategies excluded). See Table 18 for strategy description.

For stage IB, over a lifetime time horizon, strategy 77 (follow-up once for 1 year by a dermatologist) is, on average, the least costly strategy (£9536) and generates 14.57 QALYs. Should society not be willing to pay for additional QALYs, then this strategy has a 55% chance of being cost-effective. As society’s WTP for a QALY increases, the probability that strategy 77 would be considered cost-effective falls, but strategy 77 has the highest probability of being cost-effective (12.9%) when society is willing to pay £20,000 per QALY. At the same threshold value, only one other strategy considered had a probability of being cost-effective of > 10%: strategy 86 (surveillance by a dermatologist every 6 months for 1 year). It is worth noting that, at a WTP threshold of £20,000 per QALY, strategy 80 (surveillance by a dermatologist every 12 months for 2 years) had the highest NMB and an ICER of £7818 per QALY gained, compared with strategy 77. Three further strategies had a probability of being cost-effective of between 7% and 10%: strategy 80, strategy 23 (surveillance by a dermatologist every 12 months for 5 years) and strategy 25 (surveillance by a dermatologist every 12 months for 20 years). With the exception of strategy 25, all of the aforementioned strategies are less intensive than the NICE-recommended strategy. Strategy 25 has a longer follow-up duration (20 years), with patients followed up annually.

Sensitivity analyses results

The results of the PSA base case best capture the uncertainty in the decision problem. The one-way sensitivity can help indicate the effect of parameter values on the expected NMB of each comparator strategy (see Appendix 10).

Keeping all other parameters at their base-case values, the impact of the probability of self-diagnosis on the NMB was examined. The model uses a yearly probability of 60%, which was transformed to a monthly probability (0.074 or 7%). A number of sensitivity analyses were run for lower and higher values of self-diagnosis. Strategies with a probability of > 1% of being cost-effective at a WTP of £20,000 were chosen for stages IA and IB. Results are presented in Figures 11 and 12 in terms of NMB, and in Appendix 10, Tables 39 and 40, in terms of cost, effectiveness, ICER and NMB of each strategy. Figure 11 and Appendix 10, Table 39, report the results for stage IA; Figure 12 and Appendix 10, Table 40, report the results for stage IB. The results indicate that, without self-diagnosis, more intensive and longer surveillance strategies for stage IA (strategies 21, 14, 7 and 4) and stage IB (strategies 25, 18, 11 and 4) produce a fraction more QALYs (< 0.2 over the estimated patient’s average life expectancy) and are more expensive (stage IA: £2000–5000; stage IB: £4000–10,000). At higher monthly probabilities of self-diagnosis (≥ 0.35, or 35%), the benefits of surveillance are questionable as there is such little gain in QALYs. Thus, more costs are incurred from the continuous surveillance without any substantial gain in QALYs.

FIGURE 11.

One-way sensitivity analysis of monthly probability of self-diagnosis for stage IA patients.

FIGURE 12.

One-way sensitivity analysis of monthly probability of self-diagnosis for stage IB patients.

Similar patterns are observed when the probabilities of recurrence are varied and all other parameters are kept at their base-case values. Results are presented in Figures 13 and 14 and in Tables 19 and 20 for stages IA and IB, respectively (further results are reported in Appendix 10, Tables 41 and 42). At no or very low monthly probability of recurrence, more intensive and longer strategies for stage IA (strategies 21, 14, 7 and 4) and stage IB (strategies 25, 1-NICE, 18, 11 and 4) are more costly without producing any additional QALYs. As monthly recurrence probabilities increase, QALYs tend to drop (as would be expected because there are reductions in quality of life and survival). When monthly recurrence probability reaches ≥ 0.29, effectiveness between high and low resource use surveillance strategies, in terms of QALYs gained, is low (e.g. for stage IB, for strategy 11, there are 19.49 QALYs, and for strategy 77, there are 19.45 QALYs). Although the benefits of surveillance are faster detection of the recurrence, the actual gain in QALYs will be minimal.

FIGURE 13.

One-way sensitivity analysis of monthly probability of recurrence for stage IA patients.

FIGURE 14.

One-way sensitivity analysis of monthly probability of recurrence for stage IB patients.

FIGURE 15.

One-way sensitivity analysis for utility values for stage IA patients.

FIGURE 16.

One-way sensitivity analysis for utility values for stage IB patients.

In the case of utility, Figures 15 and 16 show that, as long as the utility value of the recurrence is > 0.1 for stage IA and IB patients, then the NMBs of all the surveillance strategies are positive. The most important utility values are related to stage I, as approximately 70% of all disease is diagnosed when the cancer is stage I. 226 Without data to the contrary, it is likely that most second primary melanoma diagnosis or first primary recurrences are also diagnosed at stage I.

Role of specialist dermatological nurse in surveillance

One of the research questions at the start of this analysis was whether or not specialist dermatological nurses/CNSs have a role to play in surveillance. From the initial analyses presented in the base-case analyses, none of the nurse strategies was cost-effective; therefore, these strategies were not included in the PSA. To explore the reason why they were not cost-effective, additional analyses, based on the diagnostic accuracy of the strategies, were performed (i.e. the ability of dermatological nurses to correctly diagnose whether or not a recurrence has occurred). Given that the specificity and sensitivity of nurses were obtained from a single study, which assessed the performance of eight ‘physician assistants’ in rating 173 dermoscopic images of skin lesions with known histological diagnosis,207 sensitivity analysis explored the impact on the cost-effectiveness of changes in the sensitivity and specificity of diagnosing recurrences in the nurse-based strategies. One-way sensitivity analysis showed that, at low values of specificity, nurse strategies are not expected to be cost-effective. This is because individuals are referred for further tests instead of being discharged. However, as specificity increases, nurses become more cost-effective, compared with alternative strategies. When the value of specificity reaches 0.876 for stage IA and 0.878 for stage IB, nurses become as cost-effective as dermatologists (Figures 17 and 18).

FIGURE 17.

One-way sensitivity analysis of specificity of specialist dermatological nurses for stage IA patients.

FIGURE 18.

One-way sensitivity analysis of specificity of specialist dermatological nurses for stage IB patients.

Potential role of prognostic test

A hypothetical scenario in which a validated prognostic test, be it a risk factor-based model or a biomarker or a combination of both, is available to the NHS (cost: £250) was added to the model to compare surveillance strategies. It was assumed that the test has a sensitivity of 0.8 and a specificity of 0.8 in identifying high- and low-risk patients. It was also assumed that the ‘true’ prevalence of recurrence over a lifetime is 20%, based on an Australian study. 210 Those identified as being at high risk received the recommended NICE strategy and those identified as being at low risk received the strategy that was identified in the base-case analysis as being cost-effective (stage IA: strategy 22; stage IB: strategy 77). This hypothetical prognostic test strategy was compared with a few strategies, including the current NICE strategy for stage IA and IB patients.

The results of the costs, effectiveness, ICER and NMB are presented in Appendix 10, Table 43. In this simplified analysis, the least costly option had the highest NMB for stage IB patients. However, the prognostic test strategy had the highest NMB in stage IA patients and performed slightly better than the NICE strategy in both subgroups. When the associated CEACs are taken into account, the NICE strategy is likely to be the most cost-effective strategy at WTP thresholds of > £12,000 for stage IA and £26,000 for stage IB (Figures 19 and 20).

FIGURE 19.

The CEAC for prognostic test strategy for stage IA patients (only three strategies included).

FIGURE 20.

The CEAC for prognostic test strategy for stage IB patients (only four strategies included).

In these illustrated examples (see Figures 19 and 20), only a few of the possible strategies were compared in this analysis (compared with the 20 strategies in the base-case analysis). Furthermore, the way in which the prognostic test is incorporated into surveillance strategy in this sensitivity analysis may not be optimal. Both these points will have an effect on what is deemed cost-effective. What these results illustrate is that, other things being equal, increases to the sensitivity and specificity of the prognostic test (or reductions in the cost of the prognostic test) could markedly change which strategy is considered cost-effective (and may lead to additional strategies being considered cost-effective). If a prognostic test that can acceptably differentiate stage IA and IB into high and low risk of recurrence were to become available in the future at a reasonable price, then the recommended surveillance strategy may change.

Value-of-information analysis

The usefulness of a value-of-information/expected value of perfect information (EVPI) analysis is in calculating a maximum reasonable price for information. Additional evidence is valuable because it can improve patient outcomes by resolving existing uncertainty about the cost-effectiveness of the interventions available, thereby informing treatment choices for subsequent patients. 229 It also allows for the comparison of the cost of uncertainty with the cost of obtaining additional evidence.

The overall EVPI for IA is £5545 per person affected by the decision, and the population EVPI is £30,824,655 per year in England. The overall EVPI for stage IB is £6339 per person affected by the decision, and the population EVPI is £20,256,750 per year in England. We assume that the population EVPI exceeds the expected cost of research and consider the EVPPI for groups of parameters for which it is likely that a new study (or studies) would be informative for the whole group, rather than for individual parameters.

The parameters that were considered to contribute most significantly to EVPPI estimates for both stage IA and IB by the SAVI tool were probabilities of transitioning between stages (natural history), diagnostic accuracy, health utility values and recurrence of melanoma on the overall decision uncertainty in the economic evaluation model. These parameters are potentially relevant for the design of future research, as well as for broader policy questions. Table 20 reports the EVPPI for the four groups of parameters, with the overall decision EVPI at a threshold of £20,000 per QALY. The EVPPI associated with probabilities of transitioning between stages is relatively high for both stage IA (see Table 20) and stage IB (Table 21). This research is more likely to take the form of an epidemiological study. Other parameters, such as diagnostic accuracy and utility values, require other forms of study design.

For an NHS decision-maker, the EVPPI for England per year is based on the number of patients affected by the decision each year. This was assumed to be equal to the annual incidence of stage I melanoma. In 2017, 8555 patients were diagnosed with stage I melanoma. 226 The split in IA and IB patients (63% : 37%) was taken from incident data from a German Registry data set. 209 EVPI is expressed for the total population of patients who stand to benefit from additional information over the expected lifetime of the technology. The EVPPI over an arbitrary 20-year time horizon is presented.

Summary of the cost-effectiveness analysis

The base-case PSAs economic model presented in this report compared 20 different follow-up surveillance strategies for stage IA melanoma and 20 different strategies for stage IB melanoma. Cumulative costs and QALYs were compared over a lifetime time horizon, with the results suggesting that a less intensive follow-up strategy for both stages IA and IB has a higher NMB than the surveillance strategy recommended in the NICE guidelines. The results indicate that, for the strategies examined, the main difference is in the costs of each strategy, and there are only minimal differences observed in QALYs between strategies. Strategy 22 for stage IA and strategy 77 for stage IB are the least costly options. However, in the base-case analyses, only restricted strategies were analysed and, by omitting relevant comparators, an underestimation of ICERs may have occurred. 230 However, the NMB statistic highlights the similarity of all evaluated strategies.

The reported one-way sensitivity analyses carried out on the base-case model results suggest that altering the probability of self-diagnosis did not make a major impact on the NMB. Recurrence rates are estimated to be 0.22% for stage IA and 0.46% for stage IB per month and, even if all patients had a recurrence, the analysed surveillance strategies would still have a positive NMB. This suggests that surveillance is worthwhile.

There is uncertainty in the appropriate utility values to use in the base-case analysis. In the one-way sensitivity analysis, varying utility values associated with recurrence between zero and one was explored. As utility values associated with recurrence increase, so does the NMB. It is likely that most recurrences are going to be stage I and utility values are likely to be > 0.7. However, the analysis is not able to distinguish between strategies and, therefore, is not as informative to decision-makers.

Strategies with dermatological nurses/CNSs as the main health-care professional performing the screening were the least cost-effective, despite their lower resource use cost. This was because their diagnostic accuracy and, most importantly, their specificity was lower than those of dermatologists/surgeons. The consequence of this is that patients who are initially seen by the dermatological nurse/CNS accrue further costs because of the additional diagnostic tests performed.

A hypothetical prognostic test is likely to result in some savings, relative to the current NICE strategy, and may have a role in surveillance. Further work is warranted to identify a relevant and viable prognostic test and how surveillance strategies may be altered to accommodate such a test in a cost-effective manner. The analysis reported in this chapter has been partial with respect to how surveillance strategies may be altered to accommodate a prognostic test; further work would be warranted if and when a viable prognostic test becomes available.

Value-of-information analyses, and specifically EVPPI, were also carried out on the base-case analysis. This analysis sought to estimate the value of removing uncertainty around particular parameters or groups of parameters in the model. The results indicate that further research is valuable and warranted in removing uncertainty around the diagnostic accuracy of health-care professionals, the probability of transitioning between stages, health-state utility values and the recurrence of melanoma to make a confident decision to change current NICE guidelines.

Strengths and limitations

The main strength of the economic evaluation is that a de novo model has been created to answer the research question from the perspective of the NHS. The research team has attempted to use rigorous and systematic methods to obtain parameter inputs into the economic evaluation. These were then assembled in the economic model, whose structure was informed by detailed discussions with the clinical members of the research team. One of the most important challenges faced when conducting this economic evaluation was incorporating patient behaviour. A unique feature of this model is that it considers self-diagnosis by patients and the heightened anxiety melanoma survivors experience that result in ‘false-alarm’ appointments.

A number of limitations in the economic evaluation need to be acknowledged. With respect to patient characteristics, the subtype of melanoma was not accounted for in the model. The location of the primary site of the melanoma was available in the Durham cohort, but was not utilised in the model. Accordingly, it was assumed that recurrence is independent of primary location of melanoma. Recent Australian data suggested that head and neck location of the primary tumour had the highest rate of 2-year recurrence, compared with primary tumours occurring on the trunk (HR 1.67, 95% CI 1.01 to 2.67), with upper and lower limbs having lower HRs than the trunk area. 231

The model was based on current NICE guidelines; given the complex nature of the model, the assessment of structural uncertainty (i.e. whether or not relevant processes are reflected in the model and whether or not the model reflects reality) was considered with respect to the number of follow-up regimens considered, including the frequency and duration of follow-up, and who completes that follow-up. Other areas of structural uncertainty were not addressed. Few data were available for many of the model parameters. What data were available were not ideally suited to the research question being addressed. For example, health-state ‘utilities’ values are based on patients with first primary melanoma diagnosis, rather than recurrence. Utility values are required to estimate QALYs. In this analysis, data were taken from a meta-analysis of utility values, which could be questioned. 216 As highlighted in an editorial,232 instruments or questionnaires used should be sensitive to the domains of quality of life that are likely to change as a result of the disease, the routine treatment or a targeted intervention. It is unclear whether or not the EQ-5D is sufficiently sensitive to changes in some of the domains of quality of life that might occur. For example, a European study233 reported that over half the patients with melanoma in their study reported anxiety. However, the EuroQol-5 Dimensions, five-level version, instrument was somewhat insensitive to detecting this. 233

Given the limitations of the data, the base-case and sensitivity analyses were not able to provide a strong basis on which the NHS could draw robust conclusions about what surveillance strategy would be best for the UK to adopt. Instead, ‘best bets’ that would be worthy of further consideration were identified.

Impact and implications

The impact of the work presented in this chapter is important for patients, practitioners, the NHS and for researchers. One key feature of melanoma surveillance is the important role of patients (and their partners) in detecting changes in their moles, and thus detecting recurrences or new primary melanomas. This suggests that patient education is important. The evidence suggests that dermatologists have the highest accuracy in detecting melanomas. For nurses to be considered the main health-care provider in surveillance, further work would be needed to determine and, if necessary, support the development of their diagnostic performance. Should their diagnostic performance be less than dermatologists, then training may be worthwhile. This is because, other things being equal, nurses providing surveillance would be less costly than dermatologists and it may be relatively easier to increase the cadre of nurses able to provide surveillance, rather than increasing the numbers of dermatologists.

Although no viable stratification approach above AJCC staging has thus far been identified, should one be developed, then the initial results suggest that there may be merit in further research to identify how it could be used to refine surveillance in the optimal way.