Multimodal imaging interpreted by graders to detect re-activation of diabetic eye disease in previously treated patients: the EMERALD diagnostic accuracy study

Setting

EMERALD was conducted in 13 specialised hospital eye services in the UK. Participating sites were in England (n = 11), Scotland (n = 1) and Northern Ireland (n = 1). All participating sites had extensive experience in the management of people with DR, DMO and PDR.

Population

Patients with DR and DMO and/or PDR were eligible for EMERALD if they met the following inclusion criteria:

• Adults (aged ≥ 18 years) with type 1 or type 2 diabetes and with previously successfully treated DMO and/or PDR in one or both eyes, and in whom, at the time of enrolment in the study, DMO and/or PDR was active or inactive. Patients could be recruited into the study only once.

Active DMO was defined as a CRT of > 300 µm on SD-OCT and/or by the presence of intraretinal/subretinal fluid on SD-OCT owing to DMO. Inactive DMO was defined as the lack of intraretinal/subretinal fluid at the macula on SD-OCT. Active PDR was defined as the presence of subhyaloid/vitreous haemorrhage and/or active new vessels (new vessels with a lack of fibrosis on them), whether in the disc (NVD) or elsewhere in the retina (NVE). Inactive PDR was defined as the lack of preretinal/vitreous haemorrhage and the lack of active NVD or NVE.

The exclusion criteria used were as follows:

• patients unable to provide informed consent

• patients unable to read, speak or understand English.

Pathways evaluated

Outcome measures

Selection of ophthalmic graders and training

Currently, ophthalmic photographers/imaging technicians obtain images and interpret them routinely, but make no decisions on the care of patients. In ophthalmic services, there are also ophthalmic graders that have been trained to interpret findings on fundus images for the purpose of undertaking DR screening.

For EMERALD, the ophthalmic graders at each participating site were selected as follows. To start, local principal investigators (PIs) provided names of individuals that they believed had experience of obtaining and/or grading images of patients with DMO and PDR; these individuals were approached to confirm their interest and willingness to participate in EMERALD. Some ophthalmic graders who were selected for EMERALD were already involved in the grading of images for DR screening.

Graders identified by the PIs were asked to fill in questionnaires detailing their experience of imaging/grading DMO/PDR, their experience of recognising features of DMO/PDR and whether or not they felt confident identifying DMO on SD-OCT and new vessels on fundus images. Graders who stated that they did not have experience of imaging/grading DMO/PDR and those stating that they could not recognise features of DMO/PDR were not invited to take part in EMERALD as graders.

Formal training was then provided to potential EMERALD graders; during training sessions, features of active/inactive DMO/PDR were reviewed and discussed and extensive clinical examples were presented. This training was provided to all EMERALD ophthalmic graders prior to the initiation of the study. Thus, prior to the initiation of the study, a 2-day face-to-face course was provided to potential graders. This was followed by two additional half-day training sessions.

A web-based teaching module with examples of DMO/PDR was prepared so that graders could consolidate their knowledge. Clear guidelines on when patients would need to be referred for an assessment by an ophthalmologist were also given.

To ensure that the graders who were selected had the required level of experience to undertake the task of grading images, all potential graders were required to take a test involving the reading of optical coherence tomographies (OCTs) and fundus images. Only graders who reached a minimum of 80% of correct answers in detecting the presence of DMO (active DMO) and active PDR, when present, were invited to act as graders. Graders were allowed to undergo further training and take the test a second time, but if the minimum number of correct answers was not reached at this second test they were not selected to be graders for EMERALD and did not carry out this role for the study.

Patient flow in EMERALD

Patients with previously successfully treated DMO/PDR were identified from clinical records, electronic databases or in clinic. Verbal and written information about the study was given to all potential participants and their questions, if any, were answered. At their review appointment, an ophthalmologist confirmed patient eligibility, obtained informed consent and determined whether or not DMO was present and whether there was active or inactive PDR, setting the reference standard. All participants underwent visual acuity testing, SD-OCT scans and fundus examination, as undertaken per routine standard practice. In some participating sites, 159 (40%) patients were evaluated in ‘research’ clinics, and in others 238 (60%) patients were evaluated fully within usual NHS clinics.

Once the reference standard was set, non-stereoscopic seven-field ETDRS and ultra-wide angle fundus images were obtained. These images were then coded (identifiers removed), uploaded to a central facility [the Central Administrative Research Facility (CARF), Queen’s University, Belfast] and allocated randomly to the EMERALD ophthalmic graders. Graders did not grade images from their own centre (to ensure masking to the reference standard) and did not grade seven-field ETDRS/ultra-wide fundus images from the same patient to prevent the grading of one technology influencing the grading of the other. For each patient/imaging modality, graders judged whether there was active/inactive DMO and/or active/inactive PDR, or if they were uncertain about this. Graders also determined, based on their findings, whether patients needed referral to an ophthalmologist for review/treatment or they could continue their care in the ophthalmic grader pathway.

Schedule of assessments for EMERALD

Case report forms (CRFs) were specifically designed to collect all of the information required for the purpose of the EMERALD study. The following information was obtained during the standard care pathway and recorded:

• medical and ophthalmic history, including previous treatments for DMO/PDR

• best-corrected visual acuity

• ophthalmologists’ findings on slit-lamp biomicroscopy, including:

  • presence of anterior segment neovascularisation and possible determinants of poor-quality images, including media opacity and small pupillary size

  • presence/absence of DMO and/or active/inactive PDR

  • presence/absence and location of active/inactive NVD and NVE in the retina and/or pre-retinal haemorrhage/vitreous haemorrhage

  • presence of any other co-existent eye disease (e.g. glaucoma)

  • proposed plan for the patient (observation or treatment).

While in clinic, patients completed the EuroQol-5 Dimensions, five-level version (EQ-5D-5L), National Eye Institute Visual Function-25 (NEI VFQ-25) and Vision and Quality of Life (VisQoL) questionnaires. Those willing to participate in the focus group (FG) discussions were asked for their consent and informed that they might be contacted at a later date for this purpose.

Patients underwent seven-field ETDRS and ultra-wide field imaging. Images were obtained following the guidelines provided in the standard operational procedures, as set in the EMERALD study manual. Once obtained, fundus photographs and SD-OCT images were anonymised and transferred to the CARF of Queens’ University Belfast, where they were uploaded to an electronic website developed for the study and randomly allocated and made accessible to the EMERALD ophthalmic graders and ophthalmologists determining the ERS for PDR. Images (SD-OCTs for graders and seven-field ETDRS and ultra-wide field fundus images for graders and ophthalmologists) were viewed in an ophthalmic viewer platform (Ophthalsuite, BlueWorks, Coimbra, Portugal) and graded by ophthalmic graders and ophthalmologists.

Ophthalmic graders determined and recorded in the appropriate CRF:

• whether there was active/inactive DMO/PDR or if they were unsure or unable to grade presence of the disease (ungradable)

• the presence/absence, location and activity of new vessels or whether they were unsure or unable to grade presence of the new vessels (ungradable)

• the presence/absence and degree of completeness (partial or complete) of previous laser PRP

• the presence/absence of pre-retinal haemorrhage and vitreous haemorrhage

• the presence and type of other abnormalities, if observed

• whether the patient could continue surveillance in the ophthalmic grader pathway or if an assessment by an ophthalmologist was required and the reasons why.

Similarly, ophthalmologists who were determining the ERS determined and recorded the following information in the appropriate CRF:

• whether there was active/inactive PDR or if they were unsure or unable to grade presence of the disease (ungradable)

• presence/absence, location and activity of new vessels or whether they were unsure or unable to grade presence of the new vessels (ungradable)

• presence/absence and degree of completeness (partial or complete) of previous laser PRP

• presence/absence of pre-retinal haemorrhage and vitreous haemorrhage

• presence and type of other abnormalities, if observed

• whether or not the patient could continue surveillance in the ophthalmic grader pathway or if an assessment by an ophthalmologist was required and the reasons why.

Masking

Ophthalmic graders and ophthalmologists evaluating images for the ERS were masked to the results of the reference standard. To ensure this, graders and ophthalmologists did not interpret images from patients recruited in their own centres; they did not have access to the results of the reference standard. Ophthalmologists undertaking the standard-of-care evaluation were also masked to the findings/decisions made by the ophthalmic graders (who reviewed the images at a later date). Patients were also masked to the findings/decisions made by the ophthalmic graders (these were not available at the time of the clinical visit for the study).

Data collection and quality checks

As stated above, CRFs were specifically prepared for the purpose of EMERALD and used to collect study data. Monitoring was undertaken during the study to check the accuracy of entries in CRF’s against source documents, adherence to protocol and procedures and adherence to the International Conference of Harmonisation Good Clinical Practice guidelines26 and regulatory requirements. Monitoring visits were carried out by a monitor from the Northern Ireland Clinical Trials Unit (NICTU). Training on the protocol and study procedures was provided by the chief investigator and the NICTU to research staff at all participating sites prior to the initiation of the study.

Study data were transferred from CRFs to a web-based clinical trial database, which was elaborated for the purpose of EMERALD, by NICTU personnel and processed electronically. Data quality-control checks were carried out by a data manager to ensure accuracy. Data errors were documented in quality control reports and corrective actions implemented. Data validation and discrepancy reports were generated following data entry to identify discrepancies, such as data out of range, inconsistencies or protocol deviations, based on the data validation checks programmed in the clinical trial database.

Sample size

The sample size was determined by setting a target number of people with reactivated (active) DMO and PDR required to enable sensitivity to be tested against a prespecified target level of 80%. The required sample size was calculated using formula T1 from Obuschowski27 in Microsoft Excel^® (Microsoft Corporation, Redmond, WA, USA); this was a Wald test-based calculation. This target level was considered to be the minimum acceptable for the new pathway (ophthalmic grader pathway) to be clinically viable. A lower specificity was considered acceptable; a target level of 65% was used to confirm sufficiency of the sample size to assess specificity. Eighty-nine participants with DMO/PDR that had reactivated (active DMO/PDR) was sufficient to detect if the sensitivity of the new pathway is 10% and 12% higher than the 80% minimal target set with 80% and 90% power, respectively, at the two-sided 5% significance level. 28 Ninety-three participants who have not reactivated would enable detection with a specificity 15% higher than the 65% target with 90% power. A 95% confidence interval for the sensitivity and specificity of the ophthalmic grader pathway would have a confidence interval (Wilson method) with a width of 10–20% depending on the observed level. 29 Allowing for 10% of missing/indeterminate results, 104 individuals who have reactivated and 104 who have not reactivated were required (208 individuals for each DMO and PDR), which led to a need for a maximum of 416 participants in the study. Because some participants had both DMO and PDR, and they contributed to both DMO and PDR targets, the number of patients required was smaller than 416.

Statistical analyses

Eligible participants for the EMERALD ophthalmic grader’s pathway

Eligible participants contributed to the principal analyses under the new pathway for DMO (see Table 2) if they had at least one eye with DMO that was eligible (i.e. previously successfully treated). Similarly, eligible participants contributed to the principal analyses under the new pathway for PDR (see Tables 2 and 3) if they had at least one eye with PDR that was eligible (i.e. previously successfully treated).

Participants in EMERALD, however, could have one or both eyes with ‘active DMO’ or ‘active PDR’ (‘recurrent’, ‘de novo’ or ‘persistent’; see below for definitions), or ‘inactive DMO’ or ‘inactive PDR’, or ‘no DMO’ or ‘no PDR’ according to the diagnosis established at the standard care pathway.

Eye-level classification

The eye-level classification was based on the following six scenarios:

1. ‘Scenario 1 – Previous Active, successfully treated, Today Active’ (corresponding to ‘recurrence’) – also defined as ‘Eligible eye’ for the analysis of new pathway.

2. ‘Scenario 2 – Previous Active, successfully treated, Today Inactive’ (corresponding to ‘inactive’) – also defined as ‘Eligible eye’ for the analysis of new pathway.

3. ‘Scenario 3 – Previous Active, not successfully treated, Today Active’ (corresponding to ‘persistence’) – also defined as ‘Ineligible eye’ for the analysis of new pathway.

4. ‘Scenario 4 – Never present, Today Active’ (corresponding to ‘de novo’, i.e. newly diagnosed) – also defined as ‘Ineligible eye’ for the analysis of new pathway.

5. ‘Scenario 5 – Previous Active, not successfully treated, Today Inactive’ – (corresponding to ‘inactive’) also defined as ‘Ineligible eye’ for the analysis of new pathway.

6. ‘Scenario 6 – DMO/PDR never present, Today No DMO/PDR’ (i.e. no disease) – also defined as ‘Ineligible eye’ for the analysis of new pathway.

Participant-level classification

All participants who were included in the EMERALD study were at least eligible for the new DMO or the new PDR pathway, or eligible for both the DMO and the PDR pathways. Each eye for each EMERALD participant was assessed with regard to the status of PDR and DMO, irrespective of whether or not they were ‘eligible’ for both cohorts. What data were included varies according to the analysis (see Tables 2 and 3 for a full list). The statement ‘eligible participants’ for the new pathway refers to analyses from data obtained from participants contributing with one or both eligible eyes (one or both eyes had to have previous successfully-treated DMO or PDR, respectively) (Scenario 1 or 2 in one or both eyes).

To be eligible for the new DMO pathway (see Table 2), participants needed to have at least one eye with previously successfully treated DMO (i.e. ‘eligible’). At the time of the EMERALD examination, if DMO was present in one or both eyes they were classed as having active DMO (‘recurrent DMO’), and if neither eye had DMO then participants were classed as having inactive DMO. Inactive DMO includes patients who have one previously successfully treated eye in which DMO is inactive at the time of examination and who have no DMO in the other eye. It also includes those for whom one or both eyes had been previously successfully treated for DMO, but both eyes were inactive at the time of examination. Participants who did not have an eye that had been previously successfully treated were ineligible for the new DMO pathway.

To be eligible for the new PDR pathway (see Tables 2 and 3), participants needed to have at least one eye with previously successfully treated PDR (i.e. ‘eligible’). At the time of the EMERALD examination, if PDR was active in one or both eyes they were classed as having active PDR (‘recurrent PDR’), and if neither eye had active PDR then participants were classed as having inactive PDR. Inactive PDR includes patients who had one previously successfully treated eye in which PDR is inactive at the time of examination and who had no PDR in the other eye. It also includes those in whom one or both eyes had been previously successfully treated for PDR but both eyes were inactive at the time of examination. Participants who did not have an eye that had been previously successfully treated were ineligible for the new PDR pathway.

Definition of grader’s assessments of optical coherence tomography, OPTOS and ETDRS

A grader’s decision on whether or not to refer the patient, an inherently person-level decision, was used, rather than an assessment of the single eye. The basis of the referral decision was the grader’s assessment of the corresponding disease (DMO/PDR). Furthermore, to more closely reflect how the new pathway would function in practice, a patient whom graders classified as having DMO or PDR within the eye under the category ‘unsure’ or ‘ungradable’ were considered alongside those classified as ‘active’, as both would be anticipated to require further examination by an ophthalmologist under the main analysis (referral = ‘active’ + ‘unsure’ + ‘ungradable’). How the ophthalmic graders decision was tested varied according to the analysis (see Tables 2 and 3 for a full list). Table 4 shows the graders decision of person-level active status based on the combination of eye-level results. Only those samples in which there was an overall classification available were included in the sensitivity analyses.

Diagnostic performance of the EMERALD ophthalmic graders

The diagnostic performance of the EMERALD ophthalmic graders was assessed against the standard care reference standard to determine:

1. sensitivity (the proportion of patients determined by the ophthalmologist to suffer from active DMO/PDR who had been correctly referred/identified by EMERALD ophthalmic graders)

2. specificity (the proportion of patients determined by the ophthalmologist to suffer from inactive DMO/PDR who had been correctly referred/identified by EMERALD ophthalmic graders)

3. overall agreement (a measure of how well the ophthalmic graders’ assessment agrees with the ophthalmologist assessment)

4. positive likelihood ratio (the probability that a patient with active DMO/PDR is correctly assessed by the ophthalmic graders, divided by the probability that a patient with inactive DMO/PDR is incorrectly assessed by the ophthalmic graders as being active)

5. negative likelihood ratio (the probability that a patient with active DMO/PDR is incorrectly assessed by the ophthalmic graders as being active, divided by the probability that a patient with inactive DMO/PDR is correctly assessed).

Agreement between assessment methods was also quantified, where appropriate.

Diagnostic performance analysis methods

For all diagnostic accuracy analyses the sensitivity, specificity, and positive and negative likelihood ratios were calculated (with appropriate 95% CIs). The difference in sensitivity and specificity between OPTOS and ETDRS fundus images, as assessed by the ophthalmic graders, was compared with corresponding 95% CIs produced using Newcombe’s method for paired data30 and McNemar’s test. 31

Sensitivity analyses of diagnostic performance

In addition to the main analysis for DMO and PDR, a number of sensitivity analyses were conducted (listed in Table 2).

Sensitivity analysis (SENA)1 assessed the impact of the ‘unsure’ and ‘ungradable’ test classifications and of the ophthalmic grader’s grade on the diagnostic performance by defining the index test positive as definite assessment of the active disease (in contrast to allowing referral for ‘unsure’ or ‘ungradable’). The reference standards of the O-FTF + OCT assessment for DMO and the O-FTF for PDR of those present with active disease were used.

SENA2 assessed the ophthalmic graders’ referral assessments against the reference standard of those requiring treatment for both DMO and PDR. For DMO only, SENA3 focused on diagnostic performance for assessment among patients considered to possess a form of DMO that would be more important to recognise earlier, central-involving DMO (in contrast to non-central-involving DMO against the reference standard of O-FTF + OCT assessment). For PDR only, SENA4 assessed the diagnostic performance of the ophthalmic grader against the reference standard of O-FTF assessment to detect active PDR with pre-retinal or vitreous haemorrhage (i.e. high risk PDR). For PDR only, SENA5 assessed the diagnostic performance of the ophthalmic grader against the ERS (O-FTF assessment supplemented by ophthalmologist evaluation using OPTOS and ETDRS images) to detect active PDR. A further sensitivity analysis (SENA6) assessed diagnostic accuracy for a subset of participants only, who were assessed in a ‘typical’ NHS clinic setting, in contrast to a research clinic, but otherwise under the same conditions as the main analysis.

For PDR only, three additional post hoc analyses (see Table 3) that were not listed in the statistical analysis plan were also conducted in the light of the main PDR results, to further understand the content and potential value of the two imaging modalities evaluated (ultra-wide field and seven-field fundus images) along with the reliability of the O-FTF reference standard for PDR. ‘Additional 1’ compared the ophthalmologists’ identification of active PDR by OPTOS or by ETDRS in either eye with the reference standard of O-FTF assessment for active PDR. ‘Additional 2’ compared the ophthalmologists’ identification of active PDR by OPTOS or by ETDRS in either eye with the reference standard of O-FTF assessment for patients with more severe disease (i.e. PDR with pre-retinal or vitreous haemorrhage). ‘Additional 3’ compared the ophthalmic graders’ referral for either eye by OPTOS or by ETDRS with the reference standard of O-FTF assessment + ophthalmologist OPTOS assessment of active PDR in either eye or O-FTF assessment + ophthalmologist ETDRS assessment of active PDR in either eye.

Secondary analyses of diagnostic accuracy

In addition to the principal analyses that were conducted separately at the person level for DMO and PDR, two distinct secondary analyses of diagnostic accuracy were planned (Table 5). Both of these secondary analyses focused on the entire EMERALD participant population (‘All participants’), whether or not they were eligible for a disease (DMO/PDR) pathway. First, a limited set of eye-level analyses were carried out using the positive identification of active disease at the eye level [secondary analysis (SECA) 1A–C]. A random eye was selected when two eyes were eligible, with ophthalmic grader eye-specific assessment against the reference standard. Second, person-level analyses were carried out for the overall referral status of a patient, irrespective of whether it was active DMO or PDR that required referral (SECA2A–C). SECA2A utilised an ophthalmic grader referral assessment for either disease; for DMO, OCT assessment was used, and for PDR the OPTOS and ETDRS assessment were used in turn (i.e. two sets of results). The reference standards of O-FTF + OCT assessment for DMO and O-FTF assessment for PDR for those present with active disease were used (either disease constituting active disease at the person level). SECA2B was the same as SECA2A, except that a visual acuity of less than 6 out of 12 would also be considered a valid reason for referral. SECA2C was the same as SECA2A, except the reference standard was an ophthalmologist’s assessment of the presence or absence of active disease that required treatment. Diagnostic performance for both secondary analyses was assessed using the same outcomes and methods as for the principal analyses.

One additional post hoc analysis (‘Additional 4’) that was not listed in the statistical analysis plan was also conducted to further understand the content and the potential implementation method of the new pathway. The test results for Additional 4 were graders’ assessment of active disease(s) in either eye rather than graders’ referral [which would include active disease(s), unsure or ungradable] in either eye (SECA2A). Both SECA2A and Additional 4 would be of interest to how the new pathway would function in clinical practice.

Participant characteristics

A flow diagram is provided (Figure 9) that shows the flow of participants and the O-FTF assessment reference standard for each disease. In total, 490 patients were screened, of whom 401 (82%) consented and were recruited. A total of 397 patients were eligible to participate in the study and four patients were ineligible. Five eyes could not be assessed and were excluded at eye-level assessment, but were included at patient-level assessment (assessed the other eye only): one right eye could not be assessed owing to blindness; four left eyes could not be assessed owing to blindness (n = 2), total retinal detachment (n = 1) and artificial eye (n = 1).

FIGURE 9.

Study flow diagram.

Sample description

Diagnostic performance for diabetic macular oedema patients eligible for the new ophthalmic grader pathway

Summary of ophthalmic graders’ grading results for diabetic macular oedema

Table 12 displays the OCT-based ophthalmic graders’ grading results at the person level against the reference standard. The graders’ decision on whether or not to refer the patient, an inherently person-level decision, was used rather than an assessment of the single eye. Furthermore, to address how the new pathway would function in practice, graders’ assessment results of DMO of ‘unsure’ and ‘ungradable’ were considered alongside those classified as ‘active’ to be referred to an ophthalmic clinic for ophthalmological assessment. Of the 272 DMO patients who were eligible for the new pathway, 152 (56%) were identified by the ophthalmologists’ slit-lamp biomicroscopic examination with access to SD-OCT images (O-FTF + OCT is the reference standard) as having DMO that was active at the time of recruitment, whereas 120 (44%) were identified by the O-FTF + OCT reference standard as having DMO that was inactive at the time of recruitment. Compared with this reference standard, 220 (81%) out of the 272 patients under graders’ assessment (G-OCT) were referred for further ophthalmological examination; of these, 209 (209/220; 95%) were referred because graders (G-OCT) identified active DMO (i.e. active DMO present in their opinion) and 11 (11/220; 5%) because graders were unsure or considered OCT images ungradable.

TABLE 12 - Diabetic macular oedema classifications: eligible patients for new pathway (n = 272)

G-OCT, grader assessment based on SD-OCT images; O-FTF + OCT, reference standard for DMO.

Ophthalmologist slit-lamp biomicroscopic examination with access to OCT images identifying active DMO in either eye.

Ophthalmologist slit-lamp biomicroscopic examination with access to OCT images identifying active DMO in either eye requiring treatment.

Ophthalmologist slit-lamp biomicroscopic examination with access to OCT images identifying central-involving active DMO in either eye.

Grader referral for DMO: ‘active’ + ‘unsure’ + ‘ungradable’.

Graders’ referral of DMO based on OCT images, which includes ‘DMO present’, ‘Unsure if DMO present’ and ‘Ungradable’. In practice, patients under these three categories would be referred to an ophthalmic clinic.

Grader identification of active DMO (active DMO only, ‘unsure’ and ‘ungradable’ excluded) in either eye based on OCT images.

Grader identification of central-involving active DMO in either eye based on OCT images.

Shaded cells refer to the reference standard.

Classification	Reference standard (O-FTF + OCT)			G-OCT
	Active DMOa	Active DMO requiring treatmentb	Central-involving active DMOc	Referrald for DMOe	Active DMOf	Central-involving active DMOg
Positive	152	85	132	220	209	177
Negative	120	187	139	40	50	80
Missing/cannot classify	0	0	1	12	13	15

The flow of study participants who followed the new pathway of graders’ assessment for DMO based on OCT is shown in Figure 10. Of the 397 EMERALD participants, four (1%) had missing OCT images for both eyes and were excluded from the analyses. Eleven (3%) participants were excluded from the main analysis and 12 (3%) were excluded from SENA1 because of missing OCT images for one eye, which led to no overall graders assessment result at the patient level. Of the 382 participants who had valid grading results from the graders to be included in the main analysis, 281 (74%) were identified by the graders as referring for DMO (i.e. having active DMO or unsure or ungradable), of whom 220 (78%) were eligible for the new DMO pathway. In total, 101 (26%) of the 382 participants were identified by the graders as not having active DMO (including no DMO and inactive DMO), of whom 40 (40%) were eligible for the new DMO pathway. Of the 381 participants who had valid grading results from the graders to be included in SENA1, 262 (69%) were identified by the graders as having active DMO (i.e. excluding unsure and ungradable), of whom 209 (80%) were eligible for the new DMO pathway. In total, 119 (31%) out of the 381 participants were identified by the graders as not having active DMO, of whom 50 (42%) were eligible for the new DMO pathway. The diagnostic performance for the main analysis and sensitivity analyses for DMO patients are given in Table 13. Results from the main analysis and SENA1 are also presented in the flow diagram (see Figure 10). Owing to missing OCT images that led to no overall graders assessment result, the grader’s diagnostic performance under main analysis was tested with a slightly smaller number of referral for DMO [disease present, n = 147; true positive (TP), n = 142; false negative (FN), n = 5] than the O-FTF-OCT (ophthalmologist face-to-face examination with access to optical coherence tomography scans) examination of active DMO (n = 152), and a slightly smaller number of not active DMO [disease absent, n = 113; false positive (FP), n = 78; true negative (TN), n = 35] than the O-FTF-OCT examination of not active DMO (n = 120). The grader’s diagnostic performance under SENA1 was also tested with a slightly smaller number of active DMO (disease present, n = 146; TP, n = 139; FN, n = 7) than the O-FTF-OCT examination of active DMO (n = 152), and a slightly smaller number of not active DMO (disease absent, n = 113; FP, n = 70; TN, n = 43) than the O-FTF-OCT examination of not active DMO (n = 120).

FIGURE 10.

Flow diagram: diagnosis as determined by the ophthalmic graders for DMO patients – main and SENA1. Note that this flow diagram presents only the grading results from the graders’ assessment based on SD-OCT images. FN, false negative; FP, false positive; TN, true negative; TP, true positive.

TABLE 13 - Diagnostic performance for DMO patients

G-OCT, grader assessment based on SD-OCT images; O-FTF + OCT, reference standard for DMO.

Grader referral for DMO = ‘active’ + ‘unsure’ + ‘ungradable’.

	Test positive	Reference standard	Diagnostic parameter	n/N	Estimate (95% CI)
Main	G-OCT referrala for DMO	O-FTF + OCT examination of active DMO in either eye	Sensitivity (%)	142/147	97% (92% to 99%)
			Specificity (%)	35/113	31% (23% to 40%)
			Positive likelihood ratio	–	1.40 (1.23 to 1.59)
			Negative likelihood ratio	–	0.11 (0.04 to 0.27)
SENA1	G-OCT identified active DMO	O-FTF + OCT examination of active DMO in either eye	Sensitivity (%)	139/146	95% (90% to 98%)
			Specificity (%)	43/113	38% (30% to 47%)
			Positive likelihood ratio	–	1.54 (1.32 to 1.78)
			Negative likelihood ratio	–	0.13 (0.06 to 0.27)
SENA2	G-OCT referral for DMO	O-FTF + OCT examination of active DMO in either eye requiring treatment	Sensitivity (%)	81/85	95% (89% to 98%)
			Specificity (%)	36/175	21% (15% to 27%)
			Positive likelihood ratio	–	1.20 (1.10 to 1.31)
			Negative likelihood ratio	–	0.23 (0.08 to 0.62)
SENA3	G-OCT identified central-involving active DMO	O-FTF + OCT examination of central-involving active DMO in either eye	Sensitivity (%)	121/129	94% (88% to 97%)
			Specificity (%)	72/128	56% (48% to 65%)
			Positive likelihood ratio	–	2.14 (1.75 to 2.62)
			Negative likelihood ratio	–	0.11 (0.06 to 0.22)
SENA6	G-OCT referral for DMO in routine clinic	O-FTF + OCT examination of active DMO in either eye of eligible patients in routine clinic	Sensitivity (%)	81/85	95% (89% to 98%)
			Specificity (%)	26/65	40% (29% to 52%)
			Positive likelihood ratio	–	1.59 (1.30 to 1.95)
			Negative likelihood ratio	–	0.12 (0.04 to 0.32)

Sensitivity analysis 1

Sensitivity analysis 1 assessed the impact of the ‘unsure’ and ‘ungradable’ ophthalmic graders’ grading results on the diagnostic performance, by defining the index test positive as definite assessment of active DMO (in contrast to allowing referral for unsure or ungradable DMO). Under this analysis, graders were found to have a sensitivity of 95% (95% CI 90% to 98%) and a specificity of 38% (95% CI 30% to 47%).

Sensitivity analysis 2, 3 and 6

Similar results were obtained for the analysis evaluating people with DMO requiring further treatment, as determined by the ophthalmologist in the standard of care pathway (sensitivity 95%, 95% CI 89% to 98%; specificity 21%, 95% CI 15% to 27%), and those with central-involving DMO (sensitivity 94%, 95% CI 88% to 97%; specificity 56%, 95% CI 48% to 65%) when compared with the reference standard. Sensitivity and specificity values for the grader’s pathway were found to be similar to those presented for the main analysis when referral for active DMO was considered only (excluding ‘unsure’ and ‘ungradable’) and when patients were assessed specifically in NHS clinics (vs. in ‘research’ clinics).

Diagnostic performance for proliferative diabetic retinopathy patients eligible for the new ophthalmic grader pathway

Summary of the ophthalmic graders’ grading results

Table 14 shows the UWF OPTOS and seven-field ETDRS-based ophthalmic graders’ and ophthalmologists’ grading results for PDR patients at the eye level against the reference standard. To mimic the way that the new pathway would function in clinical practice, graders’ results of PDR of ‘active’, ‘unsure’ and ‘ungradable’ were considered to require a referral to the clinic to be assessed by an ophthalmologist. Of the 281 PDR patients who were eligible for the new ophthalmic grader pathway, 111 (40%) were identified by the ophthalmologist slit-lamp biomicroscopic examination (O-FTF is the reference standard for PDR) as having active PDR at the time of recruitment in the EMERALD study. A total of 170 (60%) patients were identified by the O-FTF reference standard as having inactive PDR at the time of recruitment. In total, 161 (57%) out of the 281 patients under graders’ assessment based on OPTOS (G-OPTOS) were referred for further examination. Of these patients, 109 (68%; 109/161) were assessed by the graders based on OPTOS (G-OPTOS) as having PDR that was ‘active’, whereas 52 (32%; 52/161) were assessed by the graders (G-OPTOS) as ‘unsure’ or ‘ungradable’. In total, 170 (60%) of the 281 patients under graders’ assessment based on ETDRS (G-ETDRS) were referred for further examination. Of these, 118 (69%; 118/170) patients were assessed by the graders based on ETDRS (G-ETDRS) as having ‘active’ PDR, whereas 52 (31%; 52/170) were assessed by the graders (G-OPTOS) as ‘unsure’ or ‘ungradable’. Compared with the reference standard, 146 (52%) of the 281 patients were identified by the enhanced standard as having ‘active’ PDR.

TABLE 14 - Proliferative diabetic retinopathy classifications: eligible patients for the new pathway (n = 281)

G-ETDRS, grader assessment based on seven-field ETDRS fundus images; G-OPTOS, grader assessment based on ultra-wide field fundus images using OPTOS; O-ETDRS, ophthalmologist assessment based on seven-field ETDRS fundus images; O-FTF, reference standard; O-OPTOS, ophthalmologist assessment based on ultra-wide field fundus images using OPTOS.

Ophthalmologist slit-lamp biomicroscopic examination of active PDR.

Ophthalmologist’s identification of active PDR based on UWF OPTOS fundus images.

Ophthalmologist’s identification of active PDR based on seven-field ETDRS fundus images.

ERS combined the ophthalmologist slit-lamp biomicroscopic examination, ophthalmologist grading of UWF OPTOS and ophthalmologist grading of seven-field ETDRS imaging. Active PDR was defined as ‘Active’ detected by any of these three ophthalmologist assessments.

Ophthalmologist slit-lamp biomicroscopic examination of active PDR in either eye requiring treatment.

Ophthalmologist slit-lamp biomicroscopic examination disclosed pre-retinal or vitreous haemorrhages in either eye indicative of high-risk PDR.

Grader referral for PDR = ‘active’ + ‘unsure’ + ‘ungradable’.

Grader referral for PDR based on UWF OPTOS fundus images, which includes ‘active PDR’, ‘unsure if PDR active’, ‘unsure if PDR present’ and ‘ungradable’. In practice, patients under these four categories would be referred to an ophthalmic clinic for ophthalmological assessment.

Grader identification of ‘active’ PDR in either eye by UWF OPTOS fundus images.

Grader referral for PDR based on seven-field ETDRS fundus images, which includes ‘active PDR’, ‘unsure if PDR active’, ‘unsure if PDR present’ and ‘ungradable’. In practice, patients under these four categories would be referred to an ophthalmic clinic.

Grader identification of ‘active’ PDR in either eye based on seven-field ETDRS fundus images.

Classification	Ophthalmologist assessment						Grader assessment
	O-FTF: active PDRa	O-OPTOS: active PDRb	O-ETDRS: active PDRc	ERSd	O-FTF: active PDR requiring treatmente	O-FTF: active PDR with pre-retinal or vitreous haemorrhagef	G-OPTOS: referralg for PDRh	G-OPTOS: active PDRi	G-ETDRS: referral for PDRj	G-ETDRS: active PDRk
Positive	111	96	85	146	95	75	161	109	170	118
Negative	170	166	167	135	186	206	104	155	92	139
Missing/cannot classify	0	19	29	0	0	0	16	17	19	24

The flow of study participants who followed the new pathway of graders’ assessment of PDR based on OPTOS and ETDRS images is shown in Figure 11. Of the 397 EMERALD participants, nine (2%) participants who had missing OPTOS images for both eyes were excluded from the analyses that used OPTOS images. Twelve (3%) participants who had missing ETDRS images for both eyes were excluded from the analyses that used ETDRS images. Of the 388 participants assessed using OPTOS images, 13 (3%) were excluded from the main analysis and 14 (4%) were excluded from SENA1 owing to missing OPTOS images for one eye that led to no overall graders’ assessment result at the patient level. In comparison, 15 (4%) and 20 (5%) of the 385 participants assessed using ETDRS images were excluded from the main analysis and SENA1, respectively; this was because of missing ETDRS images for one eye that led to no overall graders’ assessment result at the patient level.

FIGURE 11.

Flow diagram: diagnosis as determined by the ophthalmic graders for PDR patients comparing OPTOS and ETDRS – main and SENA1. This flow diagram only presents the grading results from the graders’ assessment based on UWF OPTOS images and seven-field ETDRS images.

In total, 375 participants who had valid OPTOS grading results were available to be included in the main analysis, of whom 207 (55%) were identified by the graders as referring for PDR (i.e. having active PDR or unsure or ungradable). A total of 161 (78%) out of the 207 referrals for PDR were eligible for the new PDR pathway. In total, 168 (45%) out of the 375 participants were identified by the graders as having not active PDR (including no PDR and inactive PDR), of whom 104 (62%) were eligible for the new PDR pathway. A total of 374 participants had valid OPTOS grading results and were available to be included in SENA1. In total, 131 (35%) participants were identified by the graders as having active PDR (i.e. excluding unsure and ungradable), of whom 109 (83%) were eligible for the new PDR pathway. A total of 243 (65%) out of the 374 participants were identified as having not active PDR, of whom 155 (64%) were eligible for the new PDR pathway. Results from the main analysis and SENA1 for OPTOS are also presented in the flow diagram (see Figure 11). Owing to missing OPTOS images that led to no overall graders’ assessment result, grader’s diagnostic performance under main analysis was tested with a slightly smaller number of referrals for PDR (disease present, n = 105; TP n = 87; FN, n = 18) than that for the O-FTF examination of active PDR (n = 111), and a slightly smaller number of not active PDR referrals (disease absent, n = 160; FP, n = 74; TN, n = 86) than that for the O-FTF examination of not active PDR (n = 170). Graders’ diagnostic performance under SENA1 was also tested with a slightly smaller number of active PDR referrals (disease present, n = 105; TP, n = 66; FN, n = 39) than that for the O-FTF examination of active PDR (n = 111), and a slightly smaller number of not active PDR referrals (disease absent, n = 159; FP, n = 43; TN, n = 116) than that for the O-FTF examination of not active PDR (n = 170).

In total, 370 participants who had valid ETDRS grading results were available to be included in the main analysis, of whom 214 (58%) were identified by the graders as referring for PDR (i.e. having active PDR or unsure or ungradable). A total of 170 (79%) out of the 214 referrals for PDR were eligible for the new PDR pathway. A total of 156 (42%) out of the 370 participants were identified by the graders as having not active PDR (including no PDR and inactive PDR), of whom 92 (59%) were eligible for the new PDR pathway. In total, 365 participants had valid ETDRS grading results and were available to be included in SENA1. A total of 138 (38%) participants were identified by the graders as having active PDR (i.e. excluding unsure and ungradable), of whom 118 (86%) were eligible for the new PDR pathway. A total of 227 (62%) out of the 365 participants were identified as having not active PDR, of whom 139 (61%) were eligible for the new PDR pathway. Results from the main analysis and SENA1 for ETDRS are also presented in the flow diagram (see Figure 11). Owing to missing ETDRS images that led to no overall graders assessment result, grader’s diagnostic performance under the main analysis was tested with a slightly smaller number of referrals for PDR (disease present, n = 102; TP, n = 87; FN, n = 15) than that for the O-FTF examination of active PDR (n = 111), and a slightly smaller number of not active PDR referrals (disease absent, n = 160; FP, n = 83; TN, n = 77) than that for the O-FTF examination of not active PDR (n = 170). Graders’ diagnostic performance under SENA1 was also tested with a slightly smaller number of active PDR referrals (disease present, n = 99; TP, n = 70; FN, n = 29) than that for the O-FTF examination of active PDR (n = 111), and a slightly smaller number of not active PDR (disease absent, n = 158; FP, n = 48; TN, n = 110) than that for the O-FTF examination of not active PDR (n = 170).

The diagnostic performance for the main analysis, sensitivity analyses and additional analyses for PDR patients are given in Table 15.

TABLE 15 - Diagnostic performance for PDR patients

G-ETDRS, grader assessment based on seven-field ETDRS fundus images; G-OPTOS, grader assessment based on ultra-wide field fundus images using OPTOS; O-ETDRS, ophthalmologist assessment based on seven-field ETDRS fundus images; O-FTF, reference standard for PDR; O-OPTOS, ophthalmologist assessment based on ultra-wide field fundus images using OPTOS.

Grader referral for PDR = ‘active’ + ‘unsure’ + ‘ungradable’.

Analysis	Results	Reference standard	Diagnostic parameter	n/N	Estimate (95% CI)
Main	G-OPTOS referrala for PDR	O-FTF examination of active PDR in either eye	Sensitivity (%)	87/105	83% (75% to 89%)
			Specificity (%)	86/160	54% (46% to 61%)
			Positive likelihood ratio	–	1.79 (1.48 to 2.16)
			Negative likelihood ratio	–	0.32 (0.20 to 0.50)
	G-ETDRS referral for PDR	O-FTF examination of active PDR in either eye	Sensitivity (%)	87/102	85% (77% to 91%)
			Specificity (%)	77/160	48% (41% to 56%)
			Positive likelihood ratio	–	1.64 (1.39 to 1.95)
			Negative likelihood ratio	–	0.31 (0.19 to 0.50)
SENA1	G-OPTOS identified active PDR	O-FTF examination of active PDR in either eye	Sensitivity (%)	66/105	63% (53% to 71%)
			Specificity (%)	116/159	73% (66% to 79%)
			Positive likelihood ratio	–	2.32 (1.73 to 3.12)
			Negative likelihood ratio	–	0.51 (0.39 to 0.66)
	G-ETDRS identified active PDR	O-FTF examination of active PDR in either eye	Sensitivity (%)	70/99	71% (61% to 79%)
			Specificity (%)	110/158	70% (62% to 76%)
			Positive likelihood ratio	–	2.33 (1.78 to 3.04)
			Negative likelihood ratio	–	0.42 (0.30 to 0.58)
Additional 1	O-OPTOS identified active PDR	O-FTF examination of active PDR in either eye	Sensitivity (%)	74/103	72% (62% to 80%)
			Specificity (%)	137/159	86% (80% to 91%)
			Positive likelihood ratio	–	5.19 (3.46 to 7.80)
			Negative likelihood ratio	–	0.33 (0.24 to 0.45)
	O-ETDRS identified active PDR	O-FTF examination of active PDR in either eye	Sensitivity (%)	65/98	66% (57% to 75%)
			Specificity (%)	134/154	87% (81% to 91%)
			Positive likelihood ratio	–	5.11 (3.31 to 7.87)
			Negative likelihood ratio	–	0.39 (0.29 to 0.51)
SENA2	G-OPTOS referral for PDR	O-FTF examination of active PDR in either eye requiring treatment	Sensitivity (%)	77/90	86% (77% to 91%)
			Specificity (%)	91/175	52% (45% to 59%)
			Positive likelihood ratio	–	1.78 (1.49 to 2.13)
			Negative likelihood ratio	–	0.28 (0.16 to 0.47)
	G-ETDRS referral for PDR	O-FTF examination of active PDR in either eye requiring treatment	Sensitivity (%)	74/84	88% (79% to 93%)
			Specificity (%)	82/178	46% (39% to 53%)
			Positive likelihood ratio	–	1.63 (1.40 to 1.91)
			Negative likelihood ratio	–	0.26 (0.14 to 0.47)
SENA4	G-OPTOS referral for PDR	O-FTF examination of pre-retinal or vitreous haemorrhage in either eye	Sensitivity (%)	62/71	87% (78% to 93%)
			Specificity (%)	95/193	49% (42% to 56%)
			Positive likelihood ratio	–	1.71 (1.45 to 2.02)
			Negative likelihood ratio	–	0.26 (0.14 to 0.48)
	G-ETDRS referral for PDR	O-FTF examination of pre-retinal or vitreous haemorrhage in either eye	Sensitivity (%)	53/66	80% (69% to 88%)
			Specificity (%)	79/196	40% (34% to 47%)
			Positive likelihood ratio	–	1.35 (1.14 to 1.59)
			Negative likelihood ratio	–	0.49 (0.29 to 0.82)
Additional 2	O-OPTOS identified active PDR	O-FTF examination of pre-retinal or vitreous haemorrhage in either eye	Sensitivity (%)	57/70	81% (71% to 89%)
			Specificity (%)	153/192	80% (73% to 85%)
			Positive likelihood ratio	–	4.01 (2.96 to 5.42)
			Negative likelihood ratio	–	0.23 (0.14 to 0.38)
	O-ETDRS identified active PDR	O-FTF examination of pre-retinal or vitreous Haemorrhage in either eye	Sensitivity (%)	42/64	66% (53% to 76%)
			Specificity (%)	145/188	77% (71% to 83%)
			Positive likelihood ratio	–	2.87 (2.09 to 3.94)
			Negative likelihood ratio	–	0.45 (0.31 to 0.63)
SENA5	G-OPTOS referral for PDR	ERS	Sensitivity (%)	110/138	80% (72% to 86%)
			Specificity (%)	76/127	60% (51% to 68%)
			Positive likelihood ratio	–	1.98 (1.58 to 2.49)
			Negative likelihood ratio	–	0.34 (0.24 to 0.49)
	G-ETDRS referral for PDR	ERS	Sensitivity (%)	111/135	82% (75% to 88%)
			Specificity (%)	68/127	54% (45% to 62%)
			Positive likelihood ratio	–	1.77 (1.45 to 2.17)
			Negative likelihood ratio	–	0.33 (0.22 to 0.49)
Additional 3	G-OPTOS referral for PDR	O-FTF + O-OPTOS identified active PDR in either eye	Sensitivity (%)	101/125	81% (73% to 87%)
			Specificity (%)	80/140	57% (49% to 65%)
			Positive likelihood ratio	–	1.89 (1.53 to 2.32)
			Negative likelihood ratio	–	0.34 (0.23 to 0.49)
	G-ETDRS referral for PDR	O-FTF + O-ETDRS identified active PDR in either eye	Sensitivity (%)	103/122	84% (77% to 90%)
			Specificity (%)	73/140	52% (44% to 60%)
			Positive likelihood ratio	–	1.76 (1.46 to 2.13)
			Negative likelihood ratio	–	0.30 (0.19 to 0.46)
SENA6	G-OPTOS referral for PDR in routine clinic	O-FTF examination of active PDR in either eye of eligible patients in routine clinic	Sensitivity (%)	63/77	82% (72% to 89%)
			Specificity (%)	47/92	51% (41% to 61%)
			Positive likelihood ratio	–	1.67 (1.32 to 2.11)
			Negative likelihood ratio	–	0.36 (0.21 to 0.60)
	G-ETDRS referral for PDR in routine clinic	O-FTF examination of active PDR in either eye of eligible patients in routine clinic	Sensitivity (%)	60/74	81% (71% to 88%)
			Specificity (%)	41/91	45% (35% to 55%)
			Positive likelihood ratio	–	1.48 (1.19 to 1.83)
			Negative likelihood ratio	–	0.42 (0.25 to 0.71)

Sensitivity analysis 1

Sensitivity analysis 1 (SENA1) assessed the impact of the ‘unsure’ and ‘ungradable’ ophthalmic graders’ grading results on the diagnostic performance by defining the index test positive as definite assessment of active PDR. Under this analysis, graders were found to have a sensitivity of 63% based on OPTOS (G-OPTOS: sensitivity 63%, 95% CI 53% to 71%; specificity 73%, 95% CI 66% to 79%) and a sensitivity of 71% based on ETDRS (G-ETDRS: sensitivity 71%, 95% CI 61% to 79%; specificity 70%, 95% CI 62% to 76%).

Sensitivity analyses 2, 4, 5 and 6

Values for sensitivity and specificity were very similar to those from the main analysis when graders evaluated patients who required further treatment, as determined by the ophthalmologist in the standard care pathway (G-OPTOS: sensitivity 86%, 95% CI 77% to 91%; specificity 52%, 95% CI 45% to 59%) (G-ETDRS: sensitivity 88%, 95% CI 79% to 93%; specificity 46%, 95% CI 39% to 53%). When determining the sensitivity of the ophthalmic graders’ pathway to detect more severe disease (PDR with pre-retinal or vitreous haemorrhage), the sensitivity and specificity of the graders pathway when using ultra-wide-field fundus imaging was slightly higher (G-OPTOS: sensitivity 87%, 95% CI 78% to 93%; specificity 49%, 95% CI 42% to 56%) than that when using seven-field ETDRS images (G-ETDRS: sensitivity 80%, 95% CI 69% to 88%; specificity 40%, 95% CI 34% to 47%) when compared with the reference standard. When determining the sensitivity and specificity of the graders pathway against the ERS, results were similar to the comparison with the reference standard (G-OPTOS sensitivity: 80%, 95% CI 72% to 86%; specificity 60%, 95% CI 51% to 68%) (G-ETDRS: sensitivity 82%, 95% CI 75% to 88%; specificity 54%, 95% CI 45% to 62%). The sensitivity and specificity of graders were found to be similar to those presented in the main analysis when patients were assessed specifically in NHS clinics (vs. in ‘research’ clinics).

Additional analyses 1–3

Three sets of additional ad hoc analyses for PDR patients who were not listed in the statistical analysis plan were also conducted in the light of the results of the pre-planned analysis for PDR to understand the content and potential value of the two imaging modalities better along with the reliability of the O-FTF reference standard for PDR. Additional 1 compared the ophthalmologist identifying active PDR by OPTOS or by ETDRS in either eye with the reference standard of O-FTF assessment of active PDR. The sensitivities of ophthalmologist performance (O-OPTOS: sensitivity 72%, 95% CI 62% to 80%; O-ETDRS: sensitivity 66%, 95% CI 57% to 75%) did not differ greatly from those of graders’ performance (SENA1). Ophthalmologist performance was only better than that of the graders when using UWF OPTOS fundus images (difference in sensitivity when comparing the diagnostic performance of ophthalmologists with that of graders based on OPTOS: 9%); whereas graders’ performance was better than that of the ophthalmologists when using seven-field ETDRS fundus images (difference in sensitivity when comparing the diagnostic performance of ophthalmologists with that of graders’ based on ETDRS: –5%). However, compared with graders, the ophthalmologists were much better at ruling out not active PDR. The specificity of the ophthalmologists (O-OPTOS specificity 86%, 95% CI 80% to 91%; O-ETDRS specificity 87%, 95% CI 81% to 91%) was higher than that of the graders (G-OPTOS specificity 73%, 95% CI 66% to 79%; G-ETDRS specificity 70%, 95% CI 62% to 76%). Additional 2 compared the ophthalmologist identification of active PDR by OPTOS or by ETDRS in either eye against the reference standard in patients where the O-FTF assessment identified pre-retinal or vitreous haemorrhage (i.e. people with high-risk PDR). Although the specificity of the ophthalmologists (O-OPTOS specificity 80%, 95% CI 73 % to 85%; O-ETDRS specificity 77%, 95% CI 71% to 83%) was much higher than that of the graders (G-OPTOS specificity 49%, 95% CI 42% to 56%; G-ETDRS specificity 40%, 95% CI 34% to 47%), the sensitivity of the ophthalmologists was much lower than that of the graders’ pathway when using seven-field ETDRS images (O-ETDRS sensitivity 66%, 95% CI 53% to 76%; G-ETDRS sensitivity 80%, 95% CI 69% to 88%). The sensitivity of the ophthalmologists when using UWF OPTOS fundus imaging was similar to that of the graders (O-OPTOS sensitivity 81%, 95% CI 71% to 89%; G-OPTOS sensitivity 87%, 95% CI 78% to 93%) when compared with the reference standard. Additional 3 compared the ophthalmic graders’ referral for either eye by OPTOS or by ETDRS against the O-FTF assessment combined with O-OPTOS or O-FTF assessment combined with O-ETDRS of active PDR in either eye. This additional analysis was aimed at comparing the diagnostic performance of graders using OPTOS or using ETDRS as, in practice, images could be obtained with only one imaging modality. The sensitivity and specificity of the graders were similar to the set reference standard. Some other useful results on the percentage agreement among test positive and reference standard or enhanced standard can be found in Appendix 1, Table 40.

Paired comparisons of OPTOS and ETDRS assessment of proliferative diabetic retinopathy

The difference in sensitivity and specificity between OPTOS and ETDRS fundus images assessed by the ophthalmic graders in the main analysis and SENA1 were also compared with corresponding 95% CIs produced using Newcombe’s method for paired data30 and McNemar’s test. 31 The results show no evidence of statistically significant difference in sensitivity and specificity between OPTOS and ETDRS fundus images in both analyses (Tables 16 and 17). However, a difference of a meaningful amount cannot be ruled out, with 95% CI for difference in sensitivity under SENA1 in particular ranging from –21% to 5%.

Secondary sensitivity analyses on combined diseases (diabetic macular oedema and proliferative diabetic retinopathy) for all patients

In addition to the principal analyses that were conducted at the person level separately for DMO and PDR, three planned secondary analyses and one additional analysis at person level that focused on the overall referral status of a patient, irrespective of whether it was active (or unsure or ungradable) DMO or PDR that required referral due to disease, were conducted first. These four secondary analyses focused on the entire EMERALD patient population (‘All patients’) rather than only those patients who were eligible for the new pathway. Table 18 displays the reference standard for the secondary analyses and the ophthalmic graders’ grading results for both diseases and for all patients at eye level against the reference standard. The reference standards of O-FTF + OCT assessment for DMO and O-FTF for PDR of those present with active disease were used (either disease constituting active disease at the person level) or active disease which required treatment. Diagnostic performance for the secondary analyses was assessed using the same outcomes and methods as for the principal analyses and the results are given in Table 19.

The results showed an overall high sensitivity for the new ophthalmic grader pathway in detecting DMO and PDR (either or both diseases). The sensitivity of graders referral for combined diseases (SENA2A) was similar among the two combinations of images (G-OCT + G-OPTOS vs. G-OCT + G-ETDRS); however, the specificity was much higher when they used G-OCT + G-OPTOS (sensitivity 96%, 95% CI 92% to 97%; specificity 33%, 95% CI 25% to 42%) than when they used G-OCT + G-ETDRS (sensitivity 95%, 95% CI 92% to 97%; specificity 22%, 95% CI 15% to 30%). Additional 4 was also conducted in the light of the results of the pre-planned analysis, to test graders’ assessment of active disease(s) rather than graders’ referral of active disease(s) (i.e. including ‘unsure’ and ‘ungradable’). Compared to graders’ referral of active disease(s) (including ‘unsure’ and ‘ungradable’) (SECA2A), graders’ assessment of strictly active disease(s) (Additional 4) presented similar sensitivity (93% compared with 95%), but higher specificity (G-OCT + G-OPTOS, Additional 4: 48% vs SECA2A: 33%; G-OCT + G-ETDRS, Additional 4: 38% vs. SECA2 A: 22%). Both SECA2A and Additional 4 demonstrate how the new pathway would function in practice.

Three eye-level secondary analyses were also conducted, and the results are displayed in Appendix 1,Tables 41 and 42.

Sampling and recruitment strategy

The original plan was to recruit FG participants via ophthalmology clinics and according to the strategy outlined in Table 20. All patients had to have previously successfully treated DMO and/or PDR to be eligible; at the time of enrolment in the EMERALD study their disease could be active or inactive. Some patients had one or the other disease (i.e. only DMO or only PDR), whereas others were under review for both conditions.

Unfortunately, as the recruitment of patients for the main study progressed it became increasingly clear that the targets for FG recruitment were unlikely to be met. Many clinics were seemingly unable to gain consent from more than a handful of patients for a FG meeting. Naturally, some clinics were more successful than others, and towards the end of 2018 it was decided to concentrate on just five localities: Belfast, Fife, Frimley, Gloucester and London. Even then, the prospect of holding FG meetings for different age groups had to be abandoned in four of the five centres. Holding gender-specific meetings was, however, possible in three of the centres.

Consent is one thing and attending a group is another, and so it proved. Only approximately one-third of those who had consented and agreed to attend a meeting did so. As a result, our final sample was as described in Table 21. The average age of all patient attendees was 59.75 years. The lowest average age was 48 years (for a Belfast group) and the highest average age was 70 years (for the group from Fife). In the text that follows, the FG identification numbers do not reflect the order of the groups in Table 21.

What happened in the focus groups?

The activities were carried out in five stages:

1. Introduction – following a welcome and first name introductions, the sessions began with a brief outline of the EMERALD project. It was explained to participants that although proceedings would be recorded and subsequently transcribed, whatever was said within the meeting would remain confidential. It was also stated that the doctors who organised their care and treatment were ‘very keen to get their views about the service that patients were offered, and about some changes in the service that were being considered’. (Patient consent had been obtained and information was given to patients by doctors and research nurses at each clinic before the FG meeting.)

2. Phase 1: images – the investigative component of the sessions began with the moderator showing the members of the group a series of photographs. The first photograph was of a doctor and a patient engaged in a slit-lamp examination. Participants were asked what, in particular, they appreciated (or disliked) about ‘this stage’ of routine clinic visits. The second photograph was of a nurse (in uniform) and a patient undertaking an acuity test. Again, respondents were asked what they appreciated about ‘this stage of your visit’. The third photograph was of a room with a chair/bench, in which antivascular endothelial growth factor (anti-VEGF) injections (a standard treatment for people with DMO) were given; participants were asked to comment on their experiences. The fourth photograph was of fundus imaging equipment [specifically an ultra-wide field fundus camera (OPTOS) in use and an spectral domain optical coherence tomography (SD-OCT) machine in the background]. The photograph was designed to stimulate discussion about what patients appreciated (or disliked) about their engagement with the photographer. The fifth and sixth photographs were of a waiting room and an empty corridor lined with chairs: typical of almost any corridor queue in a medical clinic. Again, participants were asked to comment on their ‘experiences’ with waiting.

3. Phase 2: vignettes – three vignettes were presented to participants. All of the vignettes concerned the provision of a virtual clinic. The first presented the story of 58-year-old man (Mr Smith) with DR who attended the ophthalmology clinic, had an eye scan and was told that he would get a result by letter. The second was a story of an ophthalmologist (Dr Xi) who wished to reduce waiting times for new patients and, therefore, decided to triage his existing patients so as to concentrate on those whose condition was ‘active’. The third was of a 65-year-old woman (Mrs McWilliams) with DMO who was told that she will need to see the consultant only if the ophthalmic grader detects some changes in her eye condition. All three vignettes served as useful stimuli for discussions on virtual clinics.

4. Phase 3: a ranking exercise – participants were asked, as a group, to rank each of six statements in order of importance to them, and to arrive at an agreed order. The statements are listed in Table 22.

5. End of meeting – the meetings ended with an opportunity for participants to raise any issues that they ‘felt had not been dealt with’. They were thanked for their attendance and given £20 in cash per person for travel expenses.

Focus groups for professionals

The FG for consultants and the two FGs for ophthalmic photographers and graders took a different form. Some of the above materials were used, but only in the context of feeding back patient views to the professional group. Participants were encouraged to comment on the views of patients and to suggest ways in which patient concerns might be addressed.

Analysing the data

The audio files were listened to repeatedly. A number of key issues were subsequently identified by the researcher, and a list of codes or node labels was composed. Node labels were subsequently attached by hand to relevant parts of the data transcripts. Examples of such node labels are present in Figure 12. For example, the ‘Lampex’ node refers to talk surrounding a slit-lamp biomicroscopy phase of a clinical examination. ‘Inject’ refers to talk concerning anti-VEGF injections and ‘Wait’ refers to talk about the experience of waiting in clinic for different procedures to be conducted. Most of the nodal issues were introduced to the participants by the researcher. These included issues about waiting times, technology and tools in use, and experiences of interaction with, for example, consultants, photographers and nurses. However, a number of other issues were brought into the discussion by participants independently. One such issue was ‘Diabetes’. Thus, participants often identified themselves as ‘diabetic’, sometimes as type 1 or type 2, or as insulin dependent for X number of years and so forth, and often implied that their diabetes (rather than retinopathy) was their number one health issue. Nevertheless, the most important issue to be brought forward in discussion was related to ‘feedback’, from the professionals to the patient. We will return to the issue of feedback later. Following the identification of nodal issues or themes, the Microsoft Word^® (Microsoft Corporation, Redmond, WA, USA) version of the transcripts was searched for key terms (e.g. photographer/grader/scan/image/technician) so as to identify all potential talk about a nodal issue in any one FG meeting.

FIGURE 12.

Issues discussed by male participants in focus group 5. ANX, anxiety; AI, artificial intelligence for detecting change in retina; DIABETES, my diabetes; DRIVE, car driving; THE DOC, consultant; FEEDBAK, information and results for ‘me’; FOTOG, photographer; GP, general practitioner; INJECT, injections; LAMPEX, slit-lamp examination with doctor; OPTOS, Optos ultra-wide field imager; VIRT CLINIC, virtual consultation; WAIT, waiting during routine visits. Participants are labelled ‘Pn’.

Figure 12 provides a diagrammatic representation of issues discussed in one FG. The diagram is presented in the form of an ‘issue web’, outlining the range of issues discussed by participants in FG5. We can see how one issue links to another, which are the most frequently discussed issues and who contributes to the discussion and have a glimpse into whether something is evaluated positively or negatively (broken lines suggest a negative relationship). An explanation of how Figure 12 was constructed is provided in Box 1.

Issue webs, such as the above, offer a view on the conversational proceedings, albeit one that generates a somewhat truncated image of the entire field. Figure 12 shows, for example, that some speakers took more turns than others (P5, smaller red box, spoke least; P1, larger yellow box, spoke the most). It also shows that the major concern of these participants was with what they called ‘feedback’ (large pink box towards the centre of the diagram) or getting information on their eye condition, usually at the time of a clinic visit. Also note that participants introduced topics that were not included in the moderator’s script. Examples include references to ‘my diabetes’, driving and the GP service. More importantly, we can detect differences in attitude to, say, the acceptance of a virtual consultation (dotted lines between nodes signal negative associations) or the use of fundus imaging equipment (specifically the OPTOS ultra-wide field imager). Thus, participants 2, 3 and 4 openly disliked the idea of a virtual clinic: ‘For the peace of mind of the patient, they should be seen regularly by the consultant’. Equally, participant 1 ‘hated [OPTOS] with a passion’. Finally, we can see that participants frequently cross-reference issues (line thickness indicates a strength of association between one topic and another); indeed, they rarely talked in terms of single issues or themes. For the sake of clarity, however, the presentation of findings that follows is presented in terms of single, sequential concerns, rather than one that emphasises cross-referencing and cross-talk.

Focus group phase 1: the journey through the clinic

As stated above, phase 1 of the FG was built around a set of photographs that were intended to stimulate discussion around a version of current clinic practice in which patients move through various stages of the ophthalmology service (e.g. from the waiting room to the room in which the visual acuity is evaluated, back to the waiting room, to the photographer, SD-OCT, ‘Optos’ ultra-wide field imager, accepting an anti-VEGF injection and ending with a slit-lamp examination with the consultant). The trajectory and the objects in the pictures were seemingly familiar to all participants.

Focus group phase 2: the virtual clinic

The acceptability of a ‘virtual consultation’ or ‘virtual clinic’ was explored via the use of three vignettes. (However, the word ‘virtual’ was not used by either the moderator or the participants.) The discussion that ensued from the presentation of the vignettes moved in various directions, as is evident from Figure 12. For most participants, however, the provision of a clinic in which people were assessed for ‘stability’ and given routine assessments and treatments, but not informed of the result ‘on the day’, was unacceptable:

Yeah, I mean, anxiety, everybody’s the same, but if you are anxious about something like this, it can put your blood sugar up and that screws up your general diabetic health. So I just think it’s important to go in, if you find out there’s bad news and something needs treated, I mean, they have the stuff there, get it all done [on the day].

M, FG3

Thus, patients spoke of the virtual clinic as ‘half a service’, a ‘backward step’ and ‘retrograde’. Fears were also expressed about being ‘isolated’ and ‘cast adrift’ without an opportunity to ‘have a talk [with the consultant] and get any problems in’. One participant in FG8 spoke about the following:

Waiting for the postman to bring a letter and you’re worrying, you’re anxious. And the thing is, with diabetes, being anxious and stressed doesn’t do it any good.

The male participants in FG5 also traced a direct link between waiting and anxiety. Another female participant in FG6 talked of being ‘left in limbo’ in the absence of any information about her eyesight and the need for ‘reassurance’ before leaving the clinic:

Walking away not knowing; that’s not nice.

In a similar manner, a male participant in FG1 stated that he would not want ‘to go away [from the clinic] wondering’, and a participant in FG9 stated that he would not want to ‘go away thinking, is there something wrong?’. The following exchange occurred in FG2 with respect to the vignette on ‘Mr Smith’:

So he doesn’t know the results, does he?

He doesn’t.

So he’s sat there worried sick.

Those who regarded the provision of a virtual clinic as acceptable did so on a number of grounds. Some were accepting of the pathway as long as their eyes were stable (although there is something of a ‘catch-22’ there). A few accepted the proposal on the grounds of ‘trust’. Thus, when the moderator asked one of the male participants in FG4 how he would feel in his own case, if an ophthalmic photographer/grader said that he would get a letter about his results, he stated:

That’s fine. I trust him, he knows his stuff.

Likewise, a member of FG10 said the following of the nurse practitioner who dealt with the eye scans:

I trust her implicitly. Frankly, I trust her more than I trust some of the younger doctors.

Some participants (e.g. in FG4 and FG10) saw the proposed arrangement as equivalent to and as acceptable as the screening service that they had experienced in other medical settings.

In FG7, divergence emerged in the following exchange, which opens with a participant talking about leaving the clinic without a ‘result’, and consequently waiting for the letter that will confirm stability or suggest further investigation:

I think that’s the problem, is waiting for that letter. I’d rather, on the day, be told rather than have to wait. Like the sword of Damocles hanging over me.

Yes [several answer].

Plus, letters get lost as well.

I must admit, I’ve been on the reception of that, and it’s worked without any problems at all. I got the letter, it told me what I needed to know, and they said ‘we’ll see you in 3 months,’ and I knew exactly there was a plan and that was fine. But I could imagine, as the lady said over there . . .

And you didn’t see the consultant at all?

No. And I didn’t need to, because again, if it’s an objective test and objective results then why do I need to see a doctor?

Exchanges of that kind sometimes led to discussion as to how ‘results’ might be communicated to patients. As is evident above, patients were not always favourably disposed to traditional mail as a form of communication, and suggested that other media could be effectively used for the transmission of test results, including e-mails, text messages and cell telephone messages; these routes were considered to be quicker than communication by letter. A few participants thought that it would be advantageous to have their scanned images mailed to them in addition to any evaluations.

Linked to fears and anxieties about not seeing the consultant were concerns about the professional skills and expertise of the ophthalmic photographers and graders. As already indicated, the individuals who do the scanning are of uncertain status to the participants; they are seen only through a ‘glass darkly’, as it were. Consequently, few participants were happy about leaving the decision of whether or not DR was active to a non-medical professional of any kind (the male participant referred to in FG10, above, was unusual in this respect). This is despite the moderator indicating that the individuals who reviewed the images were highly trained. Thus, a female participant in FG2 said of the person who scanned her eyes that he was ‘just a photographer’. A participant FG3 suggested that ‘someone more trained’ should be looking at the images. A participant in FG1 was adamant that decisions of whether or not the retinopathy was ‘active’ ‘should not be left to the photographer/grader’. In FG6, some of the participants stressed the fact that consultants had expert or tacit knowledge that was simply unavailable to others.

In a number of FGs, discussion turned to the issue of whether a ‘robot’ or an advanced artificial intelligence (AI) system might be able to detect change in the retina better than a human actor (AI appears in Figure 1, for example). However, none of those who said that they would feel content with a robotic assessment imagined that the (expert) consultant could be removed from the care pathway; clinical judgements were for clinicians alone.

In nearly all FGs, participants drew a distinction between themselves and future patients. Current patients were used to and comfortable with the existing service and would not want it changed. Future patients, who might experience the virtual service, might not know any better. Changing the care pathway now, and for ‘us’, would only lead to dissatisfaction. In addition, if change were to occur then the rationale for such change ought to be fully explained to those affected. It was a position nicely summed up by a member of FG9:

People like [John] and myself, we’ve been there for some period of time now and we know the procedures and we know what we need to do and we know what they need to do as well, because it becomes a matter of fact, and we look for that all the time. If something changes you’re aware of it straightaway. If they want to change the system then they’ve got to advertise the fact to all people who are under their care so that they’re aware, otherwise it falls flat and people get concerned.

Focus group phase 3: the ranking exercise

The ranking exercise was originally intended as a means of stimulating discussion among participants about the key issues, and perhaps reaching a consensus on what was most important to them. In truth, discussion in most groups was minimal, but the final rank order of key statements did serve as something of a confirmatory signal on the ‘feeling of the meeting’. For example, in FG7 it seemed to the moderator that two distinct camps emerged on the acceptability of the virtual clinic; therefore, instead of asking everyone present to rank the six statements, the group was divided into two. As can be seen from Table 22, there was indeed a considerable divergence between the two factions on the importance of ‘seeing the eye doctor during every visit’ (highest rank = 1; lowest rank = 6).

As with most other groups, the issue of waiting was regarded as of low importance. Reducing waiting times for new patients was also of low importance for most groups. The high rank of this item in group 2 was explained to the moderator by saying that ‘it was number 1 for others’ but not for ‘us’. For most of the other FGs, the highest-ranked statement was statement 1.

Responses of consultants

The moderator began the meeting with the six consultants by asking them to outline the potential advantages of the EMERALD care plan for patients, the NHS and themselves. Various responses emerged, but foremost was the idea that the EMERALD study would reduce in-clinic waiting times for patients (as well as referral times for new patients). Given that the patient groups persistently claimed that in-clinic ‘waiting’ was not a problem, the issue deserves some consideration.

From a consultant stand point, it seems clear that failure to attend clinic appointments (did not attends) are a problem for professionals and patients. Thus, one consultant spoke of patients with very serious conditions who failed to attend the retina clinic because of long in-clinic waiting times, and who had to be chased up in other settings (e.g. the glaucoma clinic). In the light of this, a reduction of routine waiting times could well be an advantage for patients. It may be (as was suggested above) that the FG sample was biased towards people who were retired or not economically active, and for whom in-clinic ‘waiting’ was a minor inconvenience (however, FG1, FG3, FG5, FG6 and FG10 clearly contained some economically active and self-employed individuals). Either way, there appears to be a strong difference of opinion here on the virtues of reduced in-clinic waiting times. Other advantages of the EMERALD study that were mentioned by the consultants included the potential for reduced (labour) costs for the NHS, and an increased ability to provide personalised treatment to patients with active conditions. This latter advantage is in precise accord with the wishes of the patients (as expressed via the ranking exercise); however, it is equally clear that patients welcome a medical consultation even when their condition is ‘stable’.

On patient reactions to the photograph of the slit-lamp examination, consultants adopted a straightforward technical stance. The slit-lamp examination was useful for ‘active’ eyes only. There was no need to use such methods for patients who were stable, and no real need to meet with such patients. It was further argued that the use of SD-OCT images plus ultra-wide field photography was ‘as good as a personal exam’ for detecting changes to eye tissue. However, as we have noted above, patients view the clinic as a site for the exchange of information and a source of reassurance, as well as a site for diagnosis and treatment. Therefore, for patients, the slit-lamp photograph reflected an occasion in which questions could be put and answers obtained from a medical expert. Consequently, one issue that arises out of the provision of virtual clinics is where patients can get advice and reassurance about their condition. Various suggestions were made by consultants. For example, one consultant reported that she currently told the clinic photographers to tell patients that they could ask for her to contact them by telephone if they wished to discuss anything. Others suggested that more ‘patient education’ was required: informing patients of the merits of SD-OCT and ultra-wide field images over a personal examination.

On the provision of virtual clinics more generally, it seemed as if they were already in operation in some regions. Two consultants reported that although there had been some complaints when virtual clinics were originally organised, there were no complaints any more. Both new and long-term patients (who had experienced the previous care pathway) seemed to accept the reorganised service, and a view was expressed that the new care pathway as proposed by the EMERALD study will work ‘eventually’ (none of the FG patients was drawn from these clinics).

When asked to discuss how patients might get ‘results’ of clinic reviews quickly, a number of responses were offered. At the simplest level were proposals to e-mail, telephone or text message results to patients (which is exactly in accord with what patients suggest). More advanced was a proposal that patients could be offered a system whereby they could log-in to a personal account and ‘view results for themselves’, which would also be in accord with some of the more technically minded patients. Another suggestion was that patients be offered a guarantee that images would be reviewed and assessments returned within 48 hours.

In addition to communication strategies of the above kind, some consultants proposed organisational change. For example, it was suggested that if photographers were expertly trained to grader status, it would be possible for the graders to give ‘results’ to most patients immediately. One consultant suggested that a grader might sit with the photographer and offer on-the-spot assessments. Whatever technical, administrative and legal difficulties there might be with such proposals, it quickly became clear that they would meet with resource problems of various kinds. For example, one consultant claimed that there were no graders available to her retina clinic as they were all deployed in the screening service. Implicit agreement among the consultants did, however, emerge on some points. First, that it might be possible to get ‘results’ back to patients in a ‘1-day virtual clinic’ if there were sufficient numbers of trained graders to assess images. Second, that even if results were to be forwarded to patients on a day other than the day of a clinic visit, but in a timely manner, more photographers would need to be trained to grader status.

Finally, issues relating to diabetes emerged during the discussion. The problem of dealing with retinopathy in the context of diabetes was fully recognised, and a number of references were made to a Royal College of Ophthalmologists working-group on the potential provision of a ‘one-stop shop’ for patients.

Responses of photographers and graders

Two FGs containing five photographers and two graders were held. As with the consultants, the moderator began the meeting by asking participants to outline the potential advantages of the proposed EMERALD care plan for patients, the NHS and themselves. Interestingly, the photographers and graders paid quite a bit of attention to the infrastructure costs of the new pathway, for example purchase of imaging equipment, buying software licenses and training of graders. As with the consultants, however, the primary advantage of the proposed EMERALD pathway for patients was said to be a reduction of in-clinic waiting time; participants voiced surprise at the low ranking of the ‘waiting time’ item in the patient-group ranking exercise.

Photographers and graders were well aware of patient anxiety regarding getting ‘results’. Some said that, currently, they were nearly always asked for indications as to the patients ‘stability’. As a consequence, all had developed conversational strategies (or stock replies) for dealing with the problem. For example, a photographer/grader might say to a patient:

Well; we have some very good images now and they just need to be examined by the doctor.

Another response was:

As you know, you’re in this clinic because you are low-risk and this is just a routine check-up.

As with consultants, photographers and graders thought that it would be a good idea to explore the use of electronic media (as well as traditional mail) for getting results to patients. An existing time frame for getting results to patients was mentioned as ‘4 to 6 weeks’ in one of the FGs. As to the idea of graders giving results to patients, there was considerable reluctance on a number of grounds. One participant mentioned lack of indemnity insurance as a barrier. Another stated that, even if they were trained to grader status, they would feel too ‘vulnerable’ to be responsible for making final decisions on images. More fundamentally, both photographers and graders suggested that although it might be OK for suitably trained individuals to give a result to a patient who was evidently ‘stable’, in cases of active conditions a number of problems arose. Primarily, offering a judgement about new bleeds or changes in the retina would lead the patient to ask further questions about consequences and treatments: questions that the graders felt that they would not be able to answer. It was suggested that intimating to a patient that something had changed, without offering a treatment or advice response there and then, would only intensify anxiety. It was considered far better to notify patients at a later stage, telling them how they could be accommodated in a consultant-led clinic. In addition, nearly all of the photographers and graders pointed out that there were always cases in which they simply did not know whether or not there was something awry; some images required interpretation by a medical expert.

All but one of the photographers and graders seemed to have some experience of virtual clinics, such as maculopathy or glaucoma clinics, and considered them to be both highly efficient and acceptable to most of the patients who they encountered. It was suggested that a virtual retinopathy clinic was comparable to a screening clinic and fulfilled the same public health role, with an emphasis on preventative rather than curative medicine. As with the consultant group, considerable emphasis was placed on the need for ‘patient education’ if virtual clinics were to be fully acceptable.

Costs

The average time taken in minutes to complete specified activities by either the graders or the ophthalmologists is shown in Table 25.

Some of the times in the minimum column are very small, but are as recorded in the CRF. They may indicate that problems, such as recurrent PDR, were detected very quickly and a decision was made to refer.

The costs in Table 23 were obtained from the standard source, PSSRU. 33 It may seem odd that the costs per hour for consultants and associate specialists are almost identical. In addition, no cost is given for staff doctors (sometimes referred to as ‘trust doctors’).

Patients may be seen by one of four grades of staff:

• consultants – starting salary £80,000 up to £108,000 (after about 20 years; there are yearly increments at years 2–5, then larger increments at 5-yearly intervals)

• associate specialists – starting salary £56,000, potentially up to £96,000 after 20 years

• staff doctors – starting salary £40,000, up to about £76,000 after 20 years

• specialist registrars – under supervision with checking by more senior staff, so costs were not used.

Table 26 presents alternative costing that would give a slightly lower cost for ophthalmologist time. However, in the analysis, we have abided by the standard PSSRU costs. These are based on observed costs, and the absence of a difference in cost between consultants and associate specialists may relate to seniority: if the average seniority of associate specialists is much greater than that of the consultants, they will have accumulated more salary increments.

The NHS reference cost is our default cost. Where no reference cost was available, we used the difference between the NHS reference costs and the time-multiplied salary cost (£26.51) for ophthalmologist time, to derive an indirect cost weight factor of 2.2 to assess the full cost of grader pathways. However, the PSSRU costs include overheads, so the indirect adjustment factor was not necessary. Table 27 shows the costs of the different procedures.

Diabetic macular oedema: comparison of the ophthalmic grader pathway with current standard of care

The sensitivity of graders as determined in the main analysis of EMERALD was very good, 96.6% in the main analysis (see Chapter 2, Table 14), when compared with the standard of care (slit-lamp biomicroscopy undertaken by the ophthalmologist who also had access to SD-OCT images). The specificity of graders, also determined in the main analysis, was not as good, at 31%, although this could still lead to an ophthalmologist seeing approximately one-third fewer patients with inactive DMO in clinics. The main analysis included referrals from graders to ophthalmologists when they detected the presence of DMO (recurrent DMO) but also when graders were not sure if DMO was present (i.e. grading stated ‘unsure’) or when images could not be graded by the graders (i.e. grader stated ‘ungradable’). This would also be the procedure followed if this new pathway were to be used in clinical practice.

However, the savings in ophthalmologist time would apply only if neither eye had active DMO or active PDR, as under these circumstances the patient would still require an ophthalmologist evaluation and treatment. Table 28 shows the relative costs of follow-up of DMO by ophthalmic graders and ophthalmologists.

We used the results of several sensitivity analyses as reported in Chapter 2, Table 13. There were no significant differences in sensitivity, but specificity varied from as low as 21% (i.e. 79% of patients having to be referred to ophthalmologists, reducing the savings) to 56% if graders were asked to identify central-involving active DMO only.

It appears from these analyses that even with low specificity, follow-up by graders could provide useful savings in ophthalmologist time, which could be then redirected to the timely evaluation and treatment of people with sight-threatening disease.

Proliferative diabetic retinopathy: comparison of grader pathway with current standard of care

The main analysis undertaken in the EMERALD study compares the current standard of care pathway for PDR (i.e. O-FTF with slit-lamp biomicroscopy) with the ophthalmic grader pathway (review of either seven-field ETDRS fundus images or ultra-wide field fundus images by the grader). As in the DMO analysis, in the PDR analysis ‘referral’ by the grader included those patients identified as having active PDR by the grader and those identified as ‘unsure’ and ‘ungradable’ (see the main analysis in Chapter 2, Table 15).

Chapter 2, Table 16, shows a sensitivity of 82% with a specificity of 54% for graders when grading ultra-wide field fundus imaging, and a sensitivity of 85% with a specificity of 48% when grading was based on seven-field ETDRS fundus photographs. The small differences are insignificant (see 95% confidence intervals in Table 16). Ultra-wide field fundus images appear to be more cost-effective given that they provide as good sensitivity and specificity as the ETDRS ones, or in the case of PDR with high-risk characteristics, better sensitivity and specificity, but require less time to take and read, and are therefore less costly.

In SENA1, the diagnostic performance of the graders was assessed based on their specific identification of active PDR (‘unsure’ and ‘ungradable’ were not included). Their specificity improved but their sensitivity was much poorer. Therefore, this option would be less costly because the improved specificity means that fewer patients would be referred to ophthalmologists. However, many patients with active disease would be missed. However, this would not be how the grader pathway would be implemented because graders in routine care would be referring patients whether they identified active disease or if they were to be unsure or unable to grade the images.

In SENA2, the ophthalmic grader pathway was assessed by how well it identified eyes requiring treatment [some recurrences of PDR may not need immediate treatment or immediate treatment may not be possible (e.g. in cases of vitreous haemorrhage the view of the back of the eye would be poor and further laser could not be undertaken until the blood clears, which may take weeks to happen); under the latter circumstances, patients will be followed and re-assessed at the next visit]. The graders’ sensitivity for detecting these more seriously affected eyes was much better: 86% with ultra-wide field imaging and 88% with seven-field ETDRS photographs. Specificity was 52% and 46%, respectively, which was similar to the main analysis.

Analysis SENA4 shows the sensitivity of ophthalmic graders’ assessment of images when the reference standard involves the detection of pre-retinal or vitreous haemorrhages (i.e. people with high-risk PDR). The sensitivity was 87% for ultra-wide field images (95% CI 78% to 93%) and 80% (95% CI 69% to 88%) for seven-field ETDRS images. Specificities were 49% and 40%, respectively.

Table 29 shows costs of care related to PDR as drawn based on the main analysis of EMERALD (comparing the new ophthalmic grader’s pathway with the current standard of care: ophthalmologists face-to-face evaluation using slit-lamp biomicroscopy and taking into account referrals by the grader related to ‘active PDR’, ‘unsure’ and ‘ungradable’). Overall, there could be savings from the graders reading the fundus images compared with the ophthalmologists’ face-to-face examinations. Table 30 shows the same comparison but for eyes needing treatment, and Table 31 for eyes with pre-retinal or vitreous haemorrhages.

In both of the above scenarios, the savings are modest with seven-field ETDRS photographs, but higher with ultra-wide field images. However, these scenarios refer only to PDR requiring treatment and PDR with pre-retinal or vitreous haemorrhage; these would be the groups that would be more important not to be missed by the ophthalmic graders.

Proliferative diabetic retinopathy: comparison with enhanced reference standard

The cost of the ERS reflects the three elements: O-FTF with slit-lamp biomicroscopy, plus the ophthalmologist reading of both seven-field ETDRS and ultra-wide field fundus images. The ERS in EMERALD was carried out because it was accepted as possible that new vessels could be missed on the ophthalmologist slit-lamp examination but be seen on fundus images. Both sets of photographs, however, could not be carried out routinely in clinic in the NHS. For the cost–consequences analysis, the value of the ERS is that it allows us to compare the costs, sensitivity and specificity of having photographs read by an ophthalmologist or ophthalmic graders. However, for completeness, we give the costs of the ophthalmic grader pathway compared with those of the ERS in Table 32. As before, ‘inclusive referral’ means that the ophthalmic graders refer to ophthalmologists all eyes in which they diagnose ‘active PDR’, plus those about which they are ‘uncertain’ and those with ‘ungradable’ images.

One interpretation of the above figures is that, assuming that the ERS has 100% sensitivity, the marginal cost of each 1% sensitivity is (£7771.00/21.3) £364.84 for ultra-wide field images. Therefore, using the ERS might seem to be good value if it costs only £364.84 to detect each extra case of PDR. However, the cases of PDR will include a mixture of active PDR requiring immediate treatment and active PDR that is less urgent and might only be brought back at the next regular visit (or earlier).

The comparison of O-FTF with ERS shows that the ERS identifies more reactivated (recurrent) PDR than was identified using images only. This may not be surprising given that some patients have problems sitting still and focusing on the right point during slit-lamp biomicroscopic examination. However, as shown in Table 14, direct O-FTF identifies some eyes with PDR that were not identified from reading the ultra-wide field or seven-field ETDRS fundus images. This suggests that the clinically most effective method of follow-up might be to use a combination of O-FTF and photographs, of which ultra-wide field images appear to be the better choice based on the lower cost of both taking and reading the photographs, and the higher sensitivity and specificity at detecting the more severe form of PDR (i.e. with pre-retinal haemorrhage and/or vitreous haemorrhage).

If the optimal method of follow-up was a combination of O-FTF and reading of ultra-wide field images, one question would be who reads the photographs.

Table 33 shows a simple cost comparison of the reading of photographs by the grader and the ophthalmologist for 100 patients. The data compare ophthalmologist and grader sensitivity for detecting active PDR from ultra-wide field images, using O-FTF as the reference standard. Specificity is higher when ophthalmologists read the photographs, with no overlap of 95% CIs (see Table 15). Sensitivity is better by approximately 9% for ophthalmologists reading ultra-wide field images, but CIs overlap. There is little difference in sensitivity between ophthalmologists and graders with ETDRS photographs, with overlapping CIs.

However, in the ERS scenario, the ophthalmologists see all of the patients and would want the images to be available at that time to come to a definite diagnosis and to counsel the patient accordingly. It is likely that the ophthalmologist would want to check areas on the images that had looked suspicious on slit-lamp biomicroscopy. Therefore, even if all of the images came with reports from the ophthalmic graders to the ophthalmologist by the time of the slit-lamp examination, it is likely that the ophthalmologist would still spend time looking at the images and the savings in Table 33 become an overestimate. The EMERALD study cannot provide data to address this. In any case, logistically, it would be difficult to have a grader grading the images in clinic prior to patients being seen by the ophthalmologists.

Results for EQ-5D-5L, NEI VFQ-25 and VisQoL measures

Tables 34 and 35 show the mean scores for the generic health-related quality-of-life measure EQ-5D-5L and the two vision-specific questionnaires (NEI VFQ-25 and VisQoL) for patients with DMO or PDR. The patients who had active DMO status had slightly higher scores, but these were not statistically significant (see Table 34). However, patients who had inactive PDR had slightly higher EQ-5D-5L scores; NEI VFQ-25 and VisQoL scores followed a similar pattern to that observed for DMO. However, the differences were not statistically significant for any of the outcome measures.

Figure 13 shows that there were no significant differences between patients who had active DMO and patients who had inactive DMO in the different NEI VFQ-25 subscales or in the composite score.

FIGURE 13.

Mean NEI VFQ-25 scores among patients eligible for the new ophthalmic grader pathway categorised by DMO status (active vs. inactive). a, Composite score is an average of the vision-targeted sub-scale scores, excluding the general health rating question.

Figures 14 and 15 show graphically the scoring of the severity levels for the EQ-5D-5L dimensions. Overall, there was no difference between the active or the inactive eye status for patients with either DMO or PDR.

FIGURE 14.

Proportion of DMO patients split by EQ-5D-5L dimensions and levels.

FIGURE 15.

Proportion of PDR patients split by EQ-5D-5L dimensions and levels.

Diabetic macular oedema

Surveillance of people with DMO by trained ophthalmic graders using SD-OCT was found to have very good sensitivity (97%) but with much poorer specificity, ranging from 31% to 56% depending on the scenario. However, even that level of specificity would still allow useful savings in ophthalmologist time, which could be used for other patients. However, there might be some disutility owing to anxiety and the lack of reassurance gained by seeing an ophthalmologist (see Chapter 3). This could be resolved if patients wishing to talk to ophthalmologists could do so by telephone, but there would be a time cost to that.

As noted in Chapter 2, the sensitivity and specificity vary according to the comparison. For example, the term ‘inclusive referral’ meant that graders referred DMO when they consider it to be definitely active, when they were unsure about the activity of the disease and when there were ‘ungradable’ images. However, in all scenarios in DMO, sensitivity remained over 90%.

Active DMO does not progress rapidly and visual acuity is not expected to drop rapidly either; therefore, significant harm would be unlikely if active DMO were to be missed in 3% of cases at one visit, especially considering that not all eyes with DMO need immediate treatment. The proportion of eyes with DMO requiring treatment is approximately only 58% (85/147) of all active DMO (see Table 13).

Proliferative diabetic retinopathy

In PDR, comparison of the grader pathway with the standard care of O-FTF in the main analysis showed a sensitivity that may or may not be considered acceptable in clinical practice. It should be noted, however, that the sensitivity varies according to the grade of PDR, being higher in PDR requiring immediate treatment and reaching nearly 90% if ultra-wider field imaging is used in people with more severe PDR with pre-retinal and/or vitreous haemorrhage.

In PDR, an important finding, albeit from a post hoc analysis, was that the standard care of O-FTF had poorer sensitivity than the ERS. The ERS used both seven-field ETDRS and ultra-wide field images in addition to O-FTF. Obtaining both sets of images would not be feasible in routine care. Further analysis of a post hoc nature would be required to assess the accuracy of O-FTF plus ultra-wide fundus images only, as this is what could be implemented in clinical practice (ultra-wide field fundus images would be the choice when compared with seven-field fundus images considering that the former take less time to take and grade and have higher sensitivity to detect the more severe form of PDR with pre-retinal and vitreous haemorrhage).

One issue is whether or not the grading of images by ophthalmic graders would be seen as taking extra responsibility, meriting an increase in salary band. We have assumed that it would, so estimates presented herein use costs based on graders being on band 7. We have also assumed that the photographers/imaging technicians obtaining the images are on band 6.

Cost would be reduced further if the grader specificity could be improved by feedback and perhaps additional training and by other measures (e.g. patient selection prior to entering the pathway and improved monitors to visualise images).

Our mean time estimates for obtaining ultra-wide field images and seven-field ETDRS photographs were higher than those reported in an earlier study by Silva et al. 49 that, for this purpose, included only 22 patients. That study reported that the mean (± SD) time required to acquire non-mydriatic ultra-wide field images was 2.8 (± 1.3) minutes versus 6.2 (± 2.2) minutes for ETDRS photography. 49 In comparison, our study found that it took 10.4 (± 7.2) minutes to take ultra-wide field images and 15.3 (± 8.2) minutes to capture seven-field ETDRS photographs. However, the timing by Silva et al. appears to start from when the patient was positioned at the camera, whereas in EMERALD the time was recorded from when the patient entered the room to be imaged; therefore, the times are not comparable. In addition, the study by Silva et al. was undertaken in a single centre (rather than being a multicentre study as in EMERALD). Patients included in the Silva et al. study had a mean age of 54 years, which is a little lower than EMERALD patients (over half of whom were > 60 years of age). However, both the Silva et al. and the EMERALD studies found that taking ultra-wide field images was quicker than obtaining seven-field ETDRS.

Our estimate for the direct cost of the O-FTF is similar to that by Prescott et al.,50 who estimated a direct cost per slit examination of £27.29 using a similar approach.

The data from the FGs (see Chapter 3) showed that patients value being seen by the ophthalmologist. They gain reassurance, have reduction in anxiety and can ask questions. Furthermore, not seeing the ophthalmologist could lead to some disutility, which would persist until the patient was next seen. We do not have data to quantify this. As noted earlier, patients do not understand the professional standing of the photographers/imaging technicians, and have probably never heard of ophthalmic graders. This could perhaps be remedied by the provision of information. In this regard, the EMERALD results demonstrating good sensitivity of graders evaluating DMO images when compared with ophthalmologists (see Chapter 2, Sensitivity analyses 2, 4, 5 and 6 and Additional analyses 1–3) should provide reassurance to patients.

Modelling of cost-effectiveness

We had planned that, if there were to be differences among the assessment methods, a Markov model-based cost–utility analysis would be undertaken to examine the cost-effectiveness of the potential surveillance pathways.

The sensitivity of the grader pathway for DMO was very good, and no trade-off between cost savings and vision would be expected if implemented. Specificity was not high, but it is likely that this could be improved (e.g. in the EMERALD study, graders did not have access to any previous clinical information, such as visual acuity, and they did not have access to previous images: both could potentially lead to higher specificity). Poor specificity means that cost savings would be less than hoped for. However, no modelling was required for this part and, thus, it was not conducted.

The overall sensitivity of 80% for the detection of reactivated PDR in the main EMERALD analysis was within the originally accepted level. Indeed, there was the historical precedent of the 80% sensitivity, which was deemed acceptable for DR screening programmes (although it is unclear where this figure came from). Sensitivity was higher for the higher-risk group (high-risk PDR), as it was close to 90% when using ultra-wide field imaging.

Modelling was not possible, however, given that modelling would require knowledge of the following:

• The proportion of the missed reactivated PDR that would progress, and how many patients would have needed treatment if detected.

• How much vision a patient would lose before the next visit, if any.

• What proportion of people would have reactivation detected at the next visit – would some changes be missed again?

• How successful treatment was for those detected at each time point. This could be vision maintained or, for some, improved.

• What would happen if reactivated PDR was missed again at the next visit – what proportion of people would suffer irreversible visual loss and to what extent.

• The quality-of-life consequences for all of the above.

• The costs of treatment of reactivated PDR.

• Differences in costs and quality-adjusted life-years between grader and standard care pathways.

All of the above information would be required for the ophthalmic grader pathway and for the standard face-to-face ophthalmologist pathway (current standard of care). We know from the ERS that the sensitivity of the standard care pathway is not 100%, and that ophthalmologists, for example, may also miss active new vessels (i.e. active PDR) in some patients.

Unfortunately, information regarding most of the above points is not available and this precluded our ability to undertake modelling as initially planned. Furthermore, not everyone with a recurrence of PDR after PRP needs or can have treatment when the recurrence is first noted (e.g. if a vitreous haemorrhage occurs, patients are often managed by observation until the haemorrhage clears). Data on the proportion of patients needing immediate treatment, treatment at a later date or no treatment at all are not available either.

Research is needed into the natural history of treated PDR, including the timelines for recurrence, the forms that recurrence takes, the need for treatment and the effectiveness of treatment.