An observational study to assess if automated diabetic retinopathy image assessment software can replace one or more steps of manual imaging grading and to determine their cost-effectiveness

Article history

The research reported in this issue of the journal was funded by the HTA programme as project number 11/21/02. The contractual start date was in December 2012. The draft report began editorial review in November 2015 and was accepted for publication in July 2016. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

SriniVas Sadda received grants and personal fees from Optos and Carl Zeiss Meditec during the duration of the study.

Copyright statement

© Queen’s Printer and Controller of HMSO 2016. This work was produced by Tufail et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

Background

Diabetes mellitus, particularly type 2 diabetes mellitus, is a major public health problem both in the UK1 and globally. 2 Diabetic retinopathy is a common complication of diabetes mellitus and is one of the major causes of vision loss in the working-age population. 3 However, in the UK, diabetic retinopathy is no longer the leading cause of vision loss in this age group. 3 This has been attributed to the effectiveness of national screening programmes, including that of the NHS diabetic eye screening programme (DESP),4 in identifying those in need of treatment, particularly early treatment, which is highly effective in preventing visual loss. 3

The Airlie House Symposium on the Treatment of Diabetic Retinopathy established the basis for the current photographic method of quantifying the presence and severity of diabetic retinopathy. 5 The characteristic lesions of diabetic retinopathy are presently estimated to affect nearly half of those diagnosed at any given time. 6–9 The advances in the medical management of diabetes mellitus have substantially increased patient survival and life expectancy. However, in doing so people with diabetes mellitus are placed at an increased risk for developing diabetes mellitus-related microvascular complications, the most common of which is diabetic retinopathy. 7,8

The current treatment recommendations for diabetic retinopathy are highly effective in preventing visual loss. 10–12 Early detection and accurate evaluation of diabetic retinopathy severity, co-ordinated medical care and prompt appropriate treatment represent an effective approach for diabetic eye care.

A total of 2.04 million people were offered diabetic eye screening in 2014/15 in England, among whom the rate of uptake was 83%. 13 This represents a major challenge for the NHS, as current retinal screening programmes are labour intensive (requiring one or more trained human graders), and future numbers will escalate given that both prevalence and incidence of diabetes mellitus are increasing markedly. 14

The number of individuals with diabetes mellitus is projected to reach 552 million globally by 2030, so technical enhancements addressing image acquisition, automated image analysis algorithms and predictive biomarkers are needed if current diabetic retinopathy screening programmes are to manage this burden successfully. Automated retinal image analysis systems (ARIASs) have been developed. Although such an approach has been implemented in Scotland, little independent evaluation of commercially available or Conformité Européenne (CE)-marked software has been done or been undertaken for the NHS DESP using its image acquisition protocol.

Automated diabetic retinopathy screening

There is increasing interest in systems for the automated detection of diabetic retinopathy. In Scotland, one automated system reported a very high detection rate (100% for proliferative retinopathy and over 97% for maculopathy) in a large, unselected population attending two regional screening programmes. 15 The system differentiates those who have sight-threatening diabetic retinopathy or other retinal abnormalities from those at low risk of progression to sight-threatening retinopathy. However, other systems have since become commercially available and, although the diagnostic accuracy of these computer detection systems has been reported to be comparable with that of experts,16–22 the independent validity of these systems and their applicability to ‘real-life’ screening remains unclear. The external validity of such studies to date is limited because, commonly, images are not available for comparison, non-validated reading protocols were used and detection programs were developed on images and populations highly similar to the one they are tested on, including distribution of retinopathy severity, camera type, field of view and number of images per eye. The Scottish programme uses one field per eye, whereas two fields per eye are used in the NHS DESP. The limited external validity of studies to date may be in part because the majority of studies so far have been designed, run and analysed by individuals linked to the development of the software being tested. Independent expert groups without links to the commercial development of the software are uncommon and the alternative approach of using an independent, internationally recognised fundus photography reading centre is prohibitively expensive. Moreover, these image analysis systems are not currently authorised for use in the NHS DESP. Recent work has also shown that optical coherence tomography (OCT) imaging is a useful adjunct to colour fundus photography in screening for referable diabetic maculopathy and has the potential to significantly reduce over-referral of cases of macula oedema compared with human assessment of colour fundus images. 23 Some hospital eye services are already incorporating OCT assessment within their DESP. 24

Development and process of automated retinal image analysis systems

The introduction of high-resolution digital retinal imaging systems in the 1990s, combined with growth in computing power, permitted the development of computer algorithms capable of computer-aided detection and diagnosis. Computer-aided detection is the identification of pathological lesions. Computer-aided diagnosis provides a classification that incorporates additional lesion or clinical information to stratify risk or estimate the probability of disease. These recent advances have allowed ARIASs to have potential clinical utility.

Broadly, the approach to ARIASs can be categorised into two components: (1) image quality assessment and (2) image analysis.

Image quality assessment

The NHS DESP currently specifies two-field colour digital photography as the only acceptable method of systematic screening for diabetic eye disease in the NHS and define specific camera systems and minimum resolutions. 25 Currently, retinal digital photography has progressed to a stage at which colour retinal photographs can be obtained using low levels of illumination through an undilated pupil. 26 However, human factors such as movement and positioning and ocular factors such as a cataract and reflections from retinal tissues can produce artefacts. Without pupillary dilatation, artefacts are observed in 3–30% of retinal images to an extent at which they impede human grading. 27 Thus, the importance of good image quality prior to automated image analysis has been recognised and much ancillary research has been conducted in the field of image pre-processing. 28

Image analysis

The main challenge encountered with processing of colour digital images is the presence of numerous ‘distractors’ within the retinal image (retinal capillaries, underlying choroidal vessels and reflection artefacts), all of which may be confused with diabetic retinopathy lesions. As a result, much research has been focused on the selective identification of diabetic retinopathy features, including microaneurysms, haemorrhages, hard or soft exudates, cotton wool spots and venous beading. These clinical features have been described in great detail in landmark clinical trials, Diabetic Retinopathy Study and Early Treatment Diabetic Retinopathy Study. 10,29

The main approaches to image analysis of ARIASs can be found in a recent review by Sim et al. 30

Diagnostic accuracy of automated retinal image analysis systems reported to date

Commercial systems that were available at the initiation of this study and invited to participate in the study are described below. ARIASs not available at initiation of the study or that did not meet entry criteria, and that are therefore not evaluated, are summarised in a recent review. 30

Commercial systems that were available within 6 months of the start of the study include iGradingM (version 1.1; originally Medalytix Group Ltd, Manchester, UK, but purchased by Digital Healthcare, Cambridge, UK, at the initiation of the study, purchased in turn by EMIS UK, Leeds, UK, after conclusion of the study), IDx-DR (IDx, LLC, Iowa City, IA, USA), Retmarker (version 0.8.2, Retmarker Ltd, Coimbra, Portugal) and EyeArt (Eyenuk Inc., Woodland Hills, CA, USA). A comparison of the published sensitivity and specificity of each ARIAS, as well as a brief description of the published studies supporting each is summarised in a paper by Sim et al. 30 Direct head-to-head comparisons between systems to date have proven difficult, mainly because of different photographic protocols, algorithms and patient populations used for validation. A common thread among these automated systems is to identify referable retinopathy, defined as diabetic retinopathy that requires the attention of an ophthalmologist. To date, we have not progressed to a stage at which human intervention can be fully removed from screening programmes. All of the systems described below are semi-automated at some point in the workflow pathway and require the assistance of a human reader/grader.

Human grader performance and current screening

The effectiveness of different screening modalities has been widely investigated. UK studies show sensitivity levels for the detection of sight-threatening diabetic retinopathy of 48–82% for optometrists and 65% for ophthalmologists. Sensitivities for the detection of referable retinopathy by optometrists have been found in the range of 77–100%, with specificities between 94% and 100%. 43 Photographic methods currently use digital images with subsequent grading by trained individuals. Sensitivities for the detection of sight-threatening diabetic retinopathy have been found to range from 87% to 100% for a variety of trained personnel reading mydriatic 45° retinal photographs, with specificities of 83–96%. Compared with examination by an ophthalmologist, two, 45° field photographs with pupil dilatation has been reported to have a sensitivity of 95% in identifying diabetic patients with sight-threatening retinopathy (specificity 99%) and 83% sensitivity for detecting any retinopathy (specificity 88%). 44 The British Diabetic Association (Diabetes UK) established standard values for diabetic retinopathy screening programmes of at least 80% sensitivity and 95% specificity45 based on recommendations from a first National Workshop on Mobile Retinal Screening summarised in a paper by Taylor et al. 46

The standard method of grading digital colour photographs in the UK uses trained human graders who meet specific quality standards, with multiple possible levels of grading and quality control checks. One key challenge with all studies that measure diagnostic accuracy is the determination of an appropriate gold standard. It is debatable whether or not a gold standard exists in the detection of diabetic retinopathy, but various methods are thought to have better properties than others. The two methods that might possibly be considered as gold standard are seven-field stereoscopic photography and biomicroscopy carried out by a skilled ophthalmologist through dilated pupils. However, these methods have been compared in a number of studies47–50 and do not show perfect agreement. 51 Hence, it is clear that one or both allow fairly frequent errors in detecting retinopathy. It is not possible to decide objectively which is in error, but Kinyoun et al. 47 did subject disagreements to an expert review, which tended to favour the seven-field photography, with two errors, over the indirect ophthalmoscopy, with 12 errors. Moss et al. 49 also examined the disagreement closely and concluded that many involved detection of microaneurysms from photographs that were not detected by ophthalmoscopy. No matter which method was correct, this might suggest that disagreements tend to happen in milder disease states. The direct comparisons of indirect ophthalmoscopy (used by an ophthalmologist) with seven-field photography also raises questions about the standards of 80% sensitivity and 95% specificity45 seen as desirable by the British Diabetic Association (Diabetes UK) from The National Workshop on Mobile Retinal Screening in 1994 summarised in a paper by Taylor et al. 46 As these standards were not invariably met in comparisons of these two ‘gold standard’ methods, they may represent an unrealistic target for other methods.

Rationale for the study

Current retinal screening programmes are labour intensive (requiring one or more trained human graders) and with the growing burden of screening there has been increasing interest in systems that allow automated detection of diabetic retinopathy. In Scotland, one automated system is deployed in routine screening. The Scottish Screening Programme utilises one image per eye (field) taken for screening, as opposed to the two-field images used in the NHS DESP. A number of commercially available CE-marked systems can potentially differentiate those who have sight-threatening diabetic retinopathy or other retinal abnormalities from those at low risk of progression to sight-threatening retinopathy. The diagnostic accuracy of these computer detection systems has been reported to be comparable with that of experts; however, the independent validity of these systems and clinical applicability to ‘real-life’ screening remains unclear. Studies to date have tended to have a low rate of severe levels of diabetic retinopathy in the test population. This means that most studies are inadequately powered to show efficacy of the automated software in identification of severe diabetic retinopathy. For example, if proliferative diabetic retinopathy is rare or absent in the test population, a detection program unable to detect proliferative diabetic retinopathy may seem to perform well. However, in a different population with a higher number of people with proliferative disease or with a different ethnicity or age profile, it may miss patients who require immediate treatment. Moreover, these image analysis systems are not currently authorised for use in the NHS DESP.

There is a need for an independent evaluation of the clinical effectiveness and cost-effectiveness of available ARIAS with CE marking, to inform potential deployment in the NHS DESP. This study addresses this important clinical need in a relevant setting of the NHS DESP. It is based on a large population with sufficient numbers of patients with severe levels of diabetic retinopathy in the test population using the multi-image per eye photographic standard and a spectrum of different ethnicities, age groups and sexes, and a range of cameras typically approved for NHS use. The cost-effectiveness of the intervention specifically for incorporation in different parts of the screening pathway will also be addressed.

Research aim

This report presents the research methods, analysis plan and findings of work commissioned by the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) programme (project number 11/21/02) to evaluate three commercially available automated grading systems against the NHS DESP manual grading in a large population of patients diagnosed with diabetes mellitus attending a real-life diabetic retinopathy screening programme. The health economics of automated versus manual grading provides a key decision-making tool to determine whether or not automated screening should take place, and how it should take place, for diabetic retinopathy in the future.

Research objectives

Study design, population and setting

The study design was an observational retrospective study based on data from consecutive diabetes mellitus patients aged ≥ 12 years who attended the annual Diabetes Eye Screening programme of the Homerton University Hospital Foundation Trust in London between 1 June 2012 and 4 November 2013, which adhered to the NHS DESP guidelines. 52,53 The aim was to obtain around 20,000 unique screening episodes that had been graded manually.

Automated grading systems

Automated systems for diabetic retinopathy detection with a CE mark obtained or applied for up to 6 months after July 2013 were eligible for evaluation. Three software systems were selected from a literature search and discussion with experts and all three were agreed to participate in the study: iGradingM,31 Retmarker and IDx-DR. 36 For commercial reasons, IDx, LLC withdrew from the study before the analysis, and in 2013 another software system (EyeArt) asked to join the study, confirming that it would meet CE-mark eligibility criteria. Three systems (iGradingM, Retmarker and EyeArt) then processed all images from all screening episodes. Permission to extract pseudoanonymised images and process all images by the automated systems was obtained from the Caldicott Guardian. No formal ethics approval was required as all extracted images were anonymised and no change in clinical pathway occurred.

All automated systems are designed to identify cases of diabetic retinopathy of R1 or above. EyeArt is additionally designed to identify cases requiring referral to ophthalmology (diabetic retinopathy of U, M1, R2 or R3). Retmarker and EyeArt process all the images associated with a screening episode and provide a classification per episode. The iGradingM system provides an outcome classification for each image.

Reference standards

Statistical methods

Health economic analyses

Details of the methods used in the health economic analysis, together with the outcomes, are given in Chapter 4.

In brief, a health economic analysis was carried out to investigate the economic implications involved if (1) an automated system were to replace level 1 human graders and (2) the automated system were to be used as a filter prior to level 1 grading.

Data extraction from the Homerton diabetic eye screening programme

Box 1 shows the number of screening episodes extracted from the Homerton DESP for screening visits between 1 June 2012 and 4 November 2013. Box 1 shows the degree of data completeness for manual grades. Data from 28,079 screening episodes were obtained (142,018 images) and 20,258 patients were entered into the study. Data from first screening episodes (20,258 patients and 102,856 images) were included in the analysis. The data available for each episode included a unique anonymised patient ID code, episode screening date, retinal image filenames associated with each screening episode, camera type used and manual grades for retinopathy, maculopathy and associated assessment of image quality for each eye from each grader that processed the images. The final manual grade of each episode was obtained by combining information on the quality of the retinal image with retinopathy and maculopathy grades for each eye from each human grader using the highest manual grade, in accordance with the NHS DESP guidelines. 52,53 Individuals with one eye were identifiable in the data set and image quality and manual grades from one eye only were included.

Data on age, sex and ethnicity were available for 20,212 patients (Table 3). The median age of patients was 60 years (range 10–98 years), with 0.5% aged < 18 years and 37% aged > 65 years; 41% of patients were white European, 35% were Asian and 20% were black African-Caribbean. White European patients were more likely than Asian or black African-Caribbean patients to have retinopathy graded R0M0, despite being, on average, 7 years older than Asian and black African-Caribbean patients. Conversely, white European patients were less likely to be graded U, R1M1, R2 or R3 (11.2%) than Asian patients (15.0%) or black African-Caribbean patients (16.0%). Black African-Caribbean patients were marginally more likely than white European and Asian patients to have ungradable images (2.0% vs. 1.3% and 1.3%, respectively).

Table 31 in Appendix 3 summarises the screening pathways for all first screening episodes for 20,258 patients; 10,788 episodes (53%) were graded R0M0 by a level 1 grader and did not go on to the level 2 grader. The second largest group comprised 4534 episodes (22%) that were coded as R1M0 by the level 1 grader and were then passed to the level 2 grader, who confirmed the R1M0 grade. Overall, the level 1 grader classified 295 episodes as U (≈1%); this is a relatively low level because patients known to be photographically ungradable were technically failed before image capture and underwent slit-lamp biomicroscopy in the clinic. The Homerton DESP graded M1 as either maculopathy with exudates within 1-disc diameter of centre of the fovea (M1a) or as maculopathy with exudates within the macula (M1b). M1a was graded when visual acuity was 6/6 or better, or 6/9 in the past with no deterioration when the new problem emerged and there were no new symptoms, and 6-month recall was needed. M1b was graded when the level of exudation was more than modest or if the visual acuity is 6/9 or worse, and referral to the hospital eye services was needed.

Screening performance of EyeArt software

Table 4 gives the sensitivity (detection rate) and false-positive rates for the EyeArt software by the highest (worst eye) manual retinopathy grade per episode as modified by the Reading Centre arbitration as the reference standard. The specificity for episodes graded R0M0 (i.e. no retinopathy) was 20%, equivalent to a false-positive rate of 80% (see Table 4). For episodes graded manually as R3 the sensitivity was 100%. For any retinopathy (manual grades of R1M0 or higher) the sensitivity was 95% and for grades R1M1 or higher (including ungradable images) the sensitivity was 94%. Table 5 presents the diagnostic accuracy for the point estimates in Table 4 and likelihood ratios. For example, the 95% CI for the estimated sensitivity for R3 was 97.0% to 99.9% and the bootstrapped 95% CI around the lower limit was 85.5% to 97.4% and around the upper limit was 99.5% to 99.9%. The likelihood ratios for the software show that patients with manual grades of R1M1, R2 and R3 were all approximately 1.2 times more likely to be classified as ‘disease present’ than patients with manual grades R0M0. Patients graded U, R1M1, R2, R3 were 1.12 times more likely than patients with manual grades R0M0 and R1M0 to be classified as ‘disease present’. The comparable measures of screening performance, diagnostic accuracy and likelihood ratios as obtained prior to arbitration of manual grades by the Reading Centre are presented in Tables 32 and 33 in Appendix 3. Measures of screening performance are very similar when comparing results before and after arbitration of manual grades.

EyeArt provides an alternative classification that attempts to identify cases requiring referral to ophthalmology (grades U, M1, R2 and R3); Tables 6 and 7 present the findings using this alternative output from EyeArt. This showed a marked effect for patients graded R0M0. In Table 4, 20% were classified as ‘no disease’, compared with 41% classified as ‘no refer’. The impact on the other retinopathy grades was less marked, with a reduction in sensitivity for R1M1 from 95% to 91%, and smaller changes in sensitivity for R2 and R3. Table 7 shows that the precision of detection was not affected but the likelihood ratios were marginally increased. Very similar results were observed prior to arbitration (see Appendix 3, Tables 34 and 35).

Screening performance of Retmarker software

Tables 8 and 9 provide the corresponding results for Retmarker software. The specificity for episodes graded manually as R0M0 was higher for Retmarker than for EyeArt (53% vs. 20%). However, the detection of R2 or R3 retinopathies was slightly lower than that for EyeArt, with sensitivities of 96.5% versus 99.4% for R2 and 97.9% versus 99.6% for R3. Table 9 gives the measures of diagnostic accuracy around estimates of screening performance in Table 8. For example, for R2 the estimated sensitivity was 96.5% with a 95% CI of 94.7% to 97.7%. The bootstrapped 95% CI on the lower bound of this confidence limit ranged from 92.5% to 96.6% for Retmarker compared with 97.2% to 98.9% for EyeArt (see Table 5). The precision on the lower bound of the 95% confidence limit for sensitivity for R3 ranged from 92.1% to 96.8% for Retmarker and from 85.5% to 97.4% for EyeArt (see Table 5). Focusing on the lower bound for the detection of R3 suggests that the sensitivity was potentially better for Retmarker than for EyeArt, whereas for R2 EyeArt had higher sensitivity. However, when examining combined manual grades of U, R1M1, R2 and R3, the 95% CI on the lower bound for sensitivity ranged from 81.9% to 84.9% for Retmarker and from 92.0% to 94.1% for EyeArt (see Table 5). For these combined grades EyeArt had a higher sensitivity than Retmarker when examining the diagnostic accuracy around the lower bound of the 95% CI. The likelihood ratios for Retmarker were generally higher than those for EyeArt, but this appeared to be driven by the higher specificity of Retmarker for grades R0M0 rather than improved sensitivity for manual grades of R1M1 and above. Tables 36 and 37 in Appendix 3 give the corresponding results for Retmarker prior to arbitration, which are remarkably similar to Tables 8 and 9.

Screening performance of iGradingM software

Table 10 shows results for iGradingM using the outcome classification described in Chapter 2 prior to arbitration by the Reading Centre. None of the screening episodes was classified as ‘no disease’. This raised concerns about the iGradingM software processing both disc- and macular-centred images. We examined this directly on the subset of screening episodes that underwent arbitration and we were able to identify that 99% of disc-centred images were classified as ‘ungradable’ by iGradingM compared with 36% of macular-centred images. Note that Table 10 is based on the hierarchy for grading outcome at episodes level for iGradingM as defined a priori in Chapter 2. 56 If at least one image is classified as ‘disease’ the outcome at a patient level will be ‘disease’ even if the disc-centred images are ungradable. However, if at least one image is ‘ungradable’ and all others are ‘no disease‘ then ‘ungradable’ is the worst outcome at a patient level.

Arbitration on subset of screening episodes

Table 11 shows the agreement between the manual grade from the Homerton DESP and the Doheny image Reading Centre grade for 1700 screening episodes (8373 images) that underwent arbitration. Some disagreement exists for each grade, but for combinations of grades the agreement was reasonable. For example, the manual grade from the Homerton DESP was R0M0 (n = 862) or R1M0 (n = 362) in 1224 episodes. The Doheny Image Reading Centre confirmed a grade of R0M0 or R1M0 in 81% [(734 + 22 + 149 + 91)/1224] and an additional 9.8%, although graded by the Homerton DESP, were classified as U [(89 + 32)/1224] and 8.7% were graded R1M1, R2M0, R2M1, R3M0 or R3M1 by the Doheny Image Reading Centre. Episodes with manual grades R1M1, R2M0, R2M1, R3M0 or R3M1 from the Homerton DESP (413 + 32 + 18 + 10 + 3 = 476 episodes) were assigned a grade of at least R1M1 by the Reading Centre in 59% of episodes (279/476 episodes) and an additional 8.4% were classified as U.

Exploratory analyses of demographic factors on screening performance

Tables 12–17 summarise the screening performance for EyeArt and Retmarker for all manual retinopathy grades as modified by arbitration for three ethnic groups (white European, Asian and black African-Caribbean), three age groups and by sex. There was no evidence to suggest that ethnicity (see Tables 12 and 13) or sex (see Tables 16 and 17) influences the sensitivity of EyeArt or Retmarker. However, false-positive rates appeared lower in white Europeans for combined grades R0M0 + R1M0 than in Asians or black African-Caribbeans for both ARIASs. There was some evidence that as age increased, detection of combined grades U, R1M1, R2 and R3 was reduced for both ARIASs (see Tables 14 and 15). Sensitivity falls from 96% in the youngest age group to 92% in the oldest age group for EyeArt and from 93.5% to 78.5% for Retmarker. For Retmarker (but not EyeArt) there was a decline in the false-positive rate with age, from 64% in the youngest age group to 51% in the oldest age group.

Five different cameras were in use at the Homerton DESP: 415 episodes were photographed with the Canon 2CR (Canon, Tokyo, Japan), 7569 episodes with the Canon 2CR-Dgi (Canon), 1230 episodes with the Canon EOS (Canon), 4246 episodes with the Canon EOS2 (Canon) and 3432 episodes with the TOPCONnect (Topcon, Tokyo, Japan). In 3395 episodes the camera type was not recorded. The main reason for the camera type not being recorded was a network interruption or ‘processor timeout’. In such situations images are saved to a separate location for uploading later and the camera type code was not retained. Screening performance by camera type is summarised in Tables 38 and 39 in Appendix 3. It appears that both ARIASs performed least well with the Canon EOS2 out of all the cameras.

Multiple variable logistic regression models (cases being defined as those with manual grade refined by arbitration in the worst eye of R1M0, U, M1, R2 or R3 and non-cases patients with a worst eye grade of R0M0) were used to formally test for an interaction between the ARIAS outcome classification of ‘disease’ versus ‘no disease’, with age group, sex, ethnicity and camera type as categorical variables.

A multiple variable logistic regression model with adjustment for age, sex, ethnicity and camera type showed that EyeArt cases were 4.42 (95% CI 3.95 to 4.96) times more likely than non-cases to be classified as ‘disease’ rather than ‘no disease’. Inclusion of interaction terms in the model was not statistically significant, suggesting that the performance of EyeArt was not affected by age (p = 0.51), sex (p = 0.20), ethnicity (p = 0.58) or camera type (p = 0.03). The corresponding models for Retmarker show that cases were 3.19 (95% CI 2.99 to 3.41) times more likely than non-cases to be classified as ‘disease’ rather than ‘no disease’. However, for Retmarker there were statistically significant interactions at the 1% level for all covariates except sex, suggesting that Retmarker performance was influenced by age, ethnicity and camera type. Odds ratios from the multiple variable logistic regression model including all two-way interactions for Retmarker outcome with age, sex, ethnicity and camera type are given in Table 18. The reference category is white European men aged < 50 years photographed with the Canon 2CR DGi camera; cases were 3.73 (95% CI 3.06 to 4.54) times more likely than non-cases to be classified as ‘disease’ rather than ‘no disease, once we take account of all main effects and two-way interactions. This odds ratio increased to 3.98 for the 50 to < 65 years age group but reduced to 2.97 for the oldest age group (65–98 years), suggesting that the discrimination of Retmarker was marginally poorer in the oldest age group. If the man was Asian instead of white European, the odds ratio decreased to 2.63 (95% CI 2.17 to 3.19). A similar odds ratio was observed in black African-Caribbean individuals, implying slightly poorer discrimination of Retmarker in Asian and black African-Caribbean individuals than in white European individuals. The results suggest that the software performs differentially by camera type. For example cases were much more likely than non-cases, to be classified as ‘disease’ rather than ‘no disease’ with the Canon EOS than with the Canon 2CR DGi (odds ratios of 3.73 vs. 7.93). The interaction with patient sex, although not significant at the 1% level, has been included for completeness.

Altered thresholds

As part of the economic evaluation we explored the consequences for sensitivity and specificity of using different decision thresholds in image classification. In what follows, all episodes given a final grade of R0 or R1 by the human grader, after arbitration, were classified as disease free and all other categories, including unclassified, were considered as disease cases.

Using this as our working definition of ‘disease’ versus ‘no disease’ in our sample of 20,258 patients, Retmarker generated values for sensitivity of 0.85 and for specificity of 0.48. It was not possible to achieve this exact level of screening performance by setting a threshold on the decision statistic provided by Retmarker; additional information must be being used in the outcome classification. However, varying the closest obtained threshold to change the sensitivity by an absolute difference of ± 5% gave the following pairs of values for three operating points (Table 19).

EyeArt provided two classifications: (1) ‘disease versus no disease’ based on EyeArt software set to detect ‘any retinopathy’, that is, R1 or worse, and (2) ‘refer versus no refer category’ when it is set to detect ‘referable retinopathy’, that is, M1 or worse. These were implemented as two separate decision statistics rather than two different thresholds on one decision statistic. Varying the threshold used for the ‘disease versus no disease’ classification to obtain an absolute change in sensitivity of ± 5% gave the pairs of values for three operating points shown in Table 20.

Varying the threshold used for the ‘refer versus no refer’ classification to obtain an absolute change in sensitivity of ± 5% gives the pairs of values shown in Table 21.

The trade-off between sensitivity and specificity was further explored through an analysis of the breakdown of the false-negative results. Tables 22 and 23 shows how the proportion of false negatives and true negatives increased and sensitivity reduced in order to improve specificity. Sensitivity is largely influenced by episodes graded M1 and U. Both systems, but especially EyeArt, remained relatively successful at detecting the most clinically significant grades of retinopathy, even if sensitivity was relaxed in order to increase specificity, that is, the proportion of disease-free cases correctly identified.

Given the large proportion of ungradable episodes among the false negatives, we looked more carefully at how these were generated. EyeArt, for example, analysed each image and classified it as adequate or inadequate. If more than one image in the episode was classified as adequate, the episode was classified as adequate for image quality. Human graders probably use a very different approach: marking an episode as a technical failure if any area of the retina was not imaged satisfactorily. Of the cases that had a final human grade of U, EyeArt classified 145 as disease free, which reduced its measured sensitivity. In 135 out of those 145 episodes, EyeArt identified one or more of the images as a technical failure but classified the episodes as technically adequate on the strength of the remaining images. A different rule would have provided a classification that performed much better on the metrics used in this evaluation. For example, using the default threshold on the screening decision statistic and taking the final human grade after arbitration as truth, there are 291 false negatives, the latter group break down into final manual grades after arbitration of 7 R2, 2 R3, 137 M1 and 145 U. Taking the output from the EyeArt image quality assessment and applying an alternative rule that ‘50% or more of images are technically inadequate, then the episode is technically inadequate’ would reclassify 91 of the 291 false negatives as true positives. The result would be a marked improvement in sensitivity, from 90% to 93%. The same rule was applied to all episodes; 450 episodes that were correctly classified as ‘disease free’ were incorrectly reclassified as technical failures and specificity reduced from 35% to 32%. A tougher rule – ‘if more than 50% of images are technically inadequate, then the case is technically inadequate’ – reclassified only 47 episodes; hence, sensitivity increased by 1.7% only but at an absolute cost of only a 0.8% reduction in specificity.

Implementation

Health economics

The results of the health economic evaluation are given in Chapter 4.

Introduction

This chapter covers the economic evaluation of two approaches to incorporate automated screening into the English national screening programme for diabetic retinopathy. Two questions are addressed: (1) is it cost-effective to replace level 1 human graders (manual grading) with automated screening?; and (2) is it cost-effective to use automated screening as a filter prior to level 1 manual grading? Under strategy 1, no images would be seen by level 1 graders and level 2 graders only screen images that have been flagged by the automated system as diseased (see Figure 1). In contrast, only a subset of images flagged by automated screening systems would be seen by level 1 graders under strategy 2 (see Figure 2).

We considered costs only from the perspective of the NHS. The economic evaluation of these two scenarios did not include costs that would be the same for each screening method. These included costs associated with QA programmes that would be the same under all scenarios and common costs associated with running the screening clinic (e.g. reception, assessment of clinical history, eye drops, fundus photography, rent, overhead and common capital expenditures). We took a time horizon of 1 year, in terms of the patient outcomes evaluated, but costs were amortised over the expected lifetime of the equipment (5–7 years).

Our analysis focused on the outcome of cost per appropriate screening outcome (true positive or true negative correctly identified by the ARIASs). We chose this outcome measure because the objective of this screening programme is to pick up true-positive cases as well true-negative cases, in other words, to maximise the sensitivity and specificity of the screening programme. Both outcomes are important: positive cases should be identified but this should not come at the cost of overly sensitive screening systems that also inconvenience patients with incorrect diagnoses and add further testing costs. The sensitivities and specificities of the alternative screening strategies explored are established in previous chapters, using manual grade as modified by arbitration by the Doheny Image Reader Centre (Doheny Eye Institute, Arcadia, CA, USA) as the reference standard.

We assessed two different automated screening software packages, Retmarker and EyeArt. Although three systems were evaluated as part of this project, the poor efficacy of iGradingM (specified in Chapter 3) rendered economic evaluation unnecessary. We undertook identical analyses for Retmarker and EyeArt, altering the input parameters specific to each software. As these systems are not yet used in the English NHS, costing information is tentative and requires extensive sensitivity analysis.

Model

The health economic analysis uses a decision-tree model to calculate the incremental cost-effectiveness of manual grading as it is currently applied within one major provider of diabetic retinopathy screening in England (Homerton University Hospital Foundation Trust, London, UK) versus replacing level 1 human graders with an automated system (strategy 1) or acting as screening mechanism prior to level 1 grading (strategy 2). The decision tree reflected patient screening pathways, the screening levels through which images were processed, and grading outcomes (referral to ophthalmology/hospital eye services or rescreening as part of the annual screening programme).

Figure 1 depicts the processing mechanisms affecting all image sets under strategy 1. All images are first seen by a level 1 manual grader or are run through automated screening software. Patients undergoing manual screening immediately fell within one of seven disease categories based on underlying retinopathy status, as per the NHS DESP at the time:57 R0, R1, R2, R3, M1a, M1b or U (see Chapter 3, Data extraction from the Homerton diabetic eye screening programme for further details on M1a and M1b).

FIGURE 1.

Screening pathways (strategy 1). (a) Clinical practice; and (b) replacing level 1 grader by an automated system.

The screening programme prescribes that 90% of patients receiving a grade of R0 go straight to annual rescreening, whereas other grades and 10% of the R0 grades (for QA purposes) would proceed to level 2 graders. Depending on the agreement or disagreement between level 1 and level 2 graders, a patient would either have their images seen by a level 3 grader (disagreement) or be invited back for annual rescreening (agreed R0 and R1 cases). Anyone receiving a R3, U or agreed R2 or M1a/M1b between the level 1 and level 2 graders would be referred immediately to the hospital eye service to see an ophthalmologist. If a set of images had been sent to the level 3 grader, this becomes the final outcome grade and any R0 or R1 grades would be invited for annual screening and any R3, R2, M1a or M1b patients would be referred to the hospital eye service.

This decision tree reflected clinical pathways used at our local study site at the time the images were taken and followed through on the basis of existing, manual screening pathways. The manual grading arm consists of patients exactly as they were screened and their images were read at the local site.

The local site had some slight differences at the time of the study to the current national screening pathway. 58 The main difference between our local pathway and the national pathway is that the local pathway splits patients with M1 into M1a and M1b subgroups. The local pathway then rescreens M1a patients within a 6-month interval and refers M1b patients directly to ophthalmology. This subclassification did not affect health economic outcomes. In the absence of any information on diagnoses received during follow-up visits, subsequent treatment costs and clinical outcomes remained the same across patients evaluated at either 6- or 12-month intervals. Therefore, for the present analysis, 6- and 12-month rescreen costs were identical for manual grading and automated screening arms and we called them ‘12-month rescreen’ as they are part of the annual screening programme. If this model had a longer time frame, and incorporating results from the ophthalmological visit, this difference in local pathway would have to be revisited.

Moreover, Doheny arbitration grades did not divide M1 grades into M1a and M1b subgroups. Therefore, to consistently compare screening diagnoses with arbitration grades, both M1a and M1b were assumed to be referable diagnoses. This approach enabled us to refer to arbitration grades to calculate therapeutic effectiveness. It is also consistent with national screening pathways in the UK that call for ophthalmological referral for M1 cases. 58 M1a and M1b patient diagnoses were therefore costed equally. In practical terms, this means that although modelling reflects local practice, our results are nevertheless adaptable to national screening programmes in the UK.

Patients undergoing automated screening fell within three disease categories that are similarly based on underlying retinopathy status: disease present, no disease present or other lesion/ungradable (U). Patient images deemed ungradable result from either technical failures or the capture of ungradable images. Images that proceeded to manual grading after ARIAS would then be given grades as in the manual pathway (R0, R1, R2, R3, M1a, M1b, U) by a level 2 grader.

In strategy 1, when automated screening replaced manual grading for level 1, all episodes with a positive or ungradable disease outcome (as assessed by the software), along with a small proportion (10%) of episodes with a negative disease outcome, would be sent for further human grading (level 2 graders). Given our study’s design as a retrospective evaluation, we mapped the results from manual grading data onto the automated screening arms to demonstrate what would have happened to patients if they had undergone ARIAS classification first, followed by manual grading.

Figure 2 shows the screening pathways for strategy 2 in which an automated screening system would act as a screen for all images such that fewer would be seen by a human grader. The manual grading arm stays exactly the same as in strategy 1. Ninety per cent of patients whose images, according to the automated screening system, showed no disease would be invited for annual screening. Those images found to have disease, technical failures and 10% of those images found to have no disease (QA) would be referred to a level 1 grader. The pathway would then work exactly the same as for the manual grading arm.

FIGURE 2.

Screening pathways (strategy 2). (a) Clinical practice and (b) using an automated system as a filter prior to level 1 grading. D^–, disease absent classification by automated system; D⁺, disease present classification by automated system; TF, technical failure.

The health economic model consists of the following data: (1) the probabilities associated with each step of the retinopathy grading pathway, (2) the overall outcome of each screening strategy as being appropriate (true positive and true negative) and (3) bottom-up costing of manual screening strategies and costing from manufacturer data for the ARIAS. It therefore takes into account the screening performance of automated systems, the efficacy of manual screening, the likelihood of rescreening and referral rates to ophthalmologists.

Data inputs

Analysis of cost per appropriate screening outcome

Results are expressed incrementally as cost per appropriate screening outcome (true positive or true negative correctly identified by the ARIAS) and reflect the two strategies of when to implement automated screening. For the ARIAS, an ‘appropriate outcome’ was defined as (1) identification of ‘disease’ present by the ARIAS when the reference human grade indicated prescence of potentially sight-threatening retinopathy or technical failure (including grades M1, R2, R3 and U) and (2) identification of ‘no disease’ by the ARIAS when the reference human grade indicated absence of retinopathy or background retinopathy only (grades R0 and R1, resulting in annual rescreening).

Results

Deterministic sensitivity analysis

Automated screening costs

We undertook one-way sensitivity analysis to check the robustness of our findings to 50% changes in ARIAS pricing. Table 30 shows the results of varying the cost of ARIASs per patient to demonstrate that, as expected, as the cost of ARIASs rises, using ARIASs is less cost saving per appropriate outcome missed (for all strategies and both software systems).

TABLE 30 - Impact of variations in automated screening pricing

ICER is reduced cost per appropriate outcome missed.

Base case.

Strategy 1 replaces level 1 grader with an automated system.

Strategy 2 is when an automated system acts as a filter prior to level 1 grader.

Screening strategy and software	ICERa (£)
	ARIAS = £0.50 per patient	ARIAS = £0.75 per patient	ARIAS = £1.00 per patientb	ARIAS = £1.25 per patient	ARIAS = £1.50 per patient
EyeArt
Strategy 1c	5.83	5.17	4.51	3.85	3.19
Strategy 2d	4.12	3.46	2.80	2.13	1.47
Retmarker
Strategy 1c	13.90	12.86	11.81	10.76	9.71
Strategy 2d	11.85	10.78	9.71	8.65	7.58

The results of the threshold analysis testing the highest ARIAS cost per patient before becomes more expensive per appropriate outcome than manual grading are given in Figure 3. This analysis demonstrates that for strategy 1 and Retmarker this figure is £3.82; therefore, if the ARIAS cost £3.82 more per patient, it would be costlier than manual grading. In strategy 2 for Retmarker this figure is £3.28. Given that the other components of the pathway come out more expensive in strategy 2, the lower figure for the ARIAS pricing would be expected.

FIGURE 3.

Deterministic sensitivity analysis of variations in automated screening pricing. (a) Strategy 1 EyeArt; (b) strategy 2 EyeArt; (c) strategy 1 Retmarker; and (d) strategy 2 Retmarker.

For EyeArt for strategy 1, if the ARIAS cost more than £2.71 per patient, then it would be costlier than manual grading. For strategy 2, this figure is £2.05. Again, cost components in the rest of the pathway for strategy 2, namely that level 1 graders have the highest cost per patient image screened compared with other level graders, are behind this result.

Manual grading costs

In terms of other costs, the cost of level 1, 2 and 3 graders would be expected to influence the appropriate choice of ARIAS or a manual grader acting as a level 1 grader. If one was maximising cost savings and willing to accept lower effect levels, then a 50% change in all three graders’ costing levels (in both directions simultaneously) would not affect the choice of automated screening, as automated screening would still be cheaper than manual grading (strategies 1 and 2 Retmarker). Examining the cost of level 1 graders in particular, as the cost of level 1 graders increases, automated screening would present additional cost savings if used as in both strategies for Retmarker. Level 3 grader cost changes have limited effect on cost savings in both strategies but level 1 changes make a major difference. For example, in strategy 2, the ICER varies by £2.64 savings per appropriate outcome missed when the cost of level 1 graders varies by 50%. For level 2 graders the figure is £1.09, and for level 3 graders the figure is £0.01.

The same can be said for EyeArt; varying level 1, 2 or 3 grader costs by 50% does not change the cost-saving nature of the software and the costs of level 1 have the greatest impact on cost savings, level 2 having the next greatest impact and level 3 having minimal impact (strategies 1 and 2).

Quality assurance protocols

We also examined how changing the percentage of patient image sets being sent for QA by a manual grader affects cost-effectiveness. QA was modelled to occur when the ARIAS states that no disease is present. Rather than allowing for immediate 12-month patient recall, 10% of these image sets are sent to a level 2 grader under strategy 1 and 10% are sent to a level 1 grader anyway under strategy 2. This made no difference in where the ICER lies on the cost-effectiveness quadrant (south-west corner) when varying the figure between the extremes of 0% and 100%. As expected, a lower proportion of image sets requiring QA was associated with the automated systems offering more cost savings per patient (strategies 1 and 2 for Retmarker and EyeArt).

Probability of automated screening system identifying disease

For Retmarker, 56.8% of the image sets were labelled as ‘disease’. This figure would have to increase to 74.8% for strategy 1 or 2 to become costlier than manual grading.

For EyeArt, 85.5% of image sets were labelled as ‘disease’. This figure would have to increase to 98.5% for strategy 1 to become costlier than manual grading.

Summary of findings

The findings of this model suggest that both software options offer cost-effective alternatives to a purely manual grading approach to diabetic retinopathy screening. Although they are less effective in picking up appropriate outcomes overall than manual grading, they are less expensive per patient, with these cost results being robust to significant variations in automated system pricing.

The automated systems are comparable with manual grading in picking up true negatives. However, both systems are overly sensitive in classifying images as disease positive. In cases when disease is actually present, both machines are comparable with human graders in correctly identifying positive cases, but aggregate performance of appropriate outcomes is driven down for both automated systems by a relatively high false-positive error rate.

Nevertheless, the automated screening systems tested here could reduce the requirement for manual grading. Both systems offer cost savings per appropriate outcome missed and further technical advances are likely to make the software increasingly cost-effective.

Primary analysis

In this observational retrospective study based on data from consecutive diabetic patients who attended an annual DESP, our primary analysis was to quantify the screening performance and diagnostic accuracy of all three ARIASs compared with manual grading and its cost-effectiveness at an acceptable sensitivity for detecting any retinopathy, referable retinopathy and sight-threatening retinopathy. 45 The point estimates for sensitivity using the manual grade modified by arbitration as the reference standard were as follows: EyeArt 94.7% (95% CI 94.2% to 95.2%) for any retinopathy, 93.8% (95% CI 92.9% to 94.6%) for referable retinopathy and 99.6% (95% CI 97.0% to 99.9%) for proliferative retinopathy. Corresponding sensitivities for Retmarker were 73.0% (95% CI 72.0% to 74.0%) for any retinopathy, 85.0% (95% CI 83.6% to 86.2%) for referable retinopathy and 97.9% (95% CI 94.9% to 99.1) for proliferative retinopathy. For EyeArt, the 95% CI for the lower bound of the confidence limits for sensitivity was 93.7% to 94.9% for any retinopathy, 92% to 94% for referable retinopathy and 85.5% to 97.4% for R3. For Retmarker, the corresponding lower bound confidence limits were 70.9% to 73.1% for any retinopathy, 81.9% to 84.9% for referable retinopathy and 92.1% to 96.8% for proliferative retinopathy. The diagnostic accuracy of both EyeArt and Retmarker also falls within sensitivity levels set by the British Diabetic Association, achieving good sensitivity when compared with human graders for referable retinopathy. The false-positive rates were moderate for R0M0, at 80% and 47% for EyeArt and Retmarker, respectively. For iGradingM, none of the screening episodes was classified as ‘no disease’. This raised concerns about how the iGradingM software processed both disc- and macular-centred images. We examined this directly on the subset of screening episodes that underwent arbitration and found that 99% of disc-centred images were classified as ‘ungradable’ by iGradingM, in addition to 36% of macular-centred images. iGradingM does not function on disc-centred images, but it can pick up some disease on macular-centred images. Therefore, iGradingM is of no use as a screening tool because it classified all encounters as containing at least one image with disease or ungradable.

Owing to the very poor performance of iGradingM, health economic analyses were undertaken only for EyeArt and Retmarker ARIASs. This study examined the cost-effectiveness of EyeArt and Retmarker ARIASs using two different strategies (strategies 1 and 2) versus manual grading, as currently performed at the Homerton DESP. We found that, when used as a replacement for level 1 grading (strategy 1), both automated screening systems were cost saving relative to manual grading but also less effective. When used as a filter prior to level 1 grading (strategy 2), thus reducing the number of level 2 grading episodes, both automated screening systems are less cost saving than if used as a replacement for level 1 graders.

The issue with both ARIASs appears to be not cost but effectiveness. Both systems are overly sensitive and performed very well in picking up cases of disease, but both had moderately high false-positive rates (low specificity). Hence, cost-effectiveness suffers because too many disease-free episodes are classified as disease positive, necessitating further investigation. The decision of whether or not to fund and implement a system into routine diabetic retinopathy screening programmes, and which system to use, would be based on the tolerability of the level of misclassified screening episodes balanced against cost. If the ARIASs were to be used without additional input from manual graders, a positive ARIAS classification, as modelled in strategies 1 and 2, will have both positive and negative consequences for patients. If the ARIAS outcome was reported to the patient, false positives may cause patients undue stress. On the other hand, overprovision of services via additional referrals to higher-level graders may provide a safety check for clinical diagnoses and other concurrent non-diabetic eye disease. From this perspective, cost-effectiveness figures derived in this economic evaluation provide a conservative estimate of the gains from the use of ARIASs in screening programmes and justify further cost–utility analyses of technical issues affecting adoption. One outstanding issue is that this study examines the cost-effectiveness of using ARIASs in two different places along the screening pathway for a set number of patients as per our study site. In reality, however, the scalability of ARIASs is quite different in that, unlike manual graders, they do not take time off and can work 24 hours a day, 7 days a week, and therefore can grade many more images than manual graders over the same time period . This overall ability to scale up the screening programme by implementing an ARIAS should be considered for implementation.

Assumptions about patient numbers are key to this decision process. The Homerton DESP is a relatively small screening programme and an automated screening system would probably be procured for a much larger patient population.

Modes of implementation other than the pathways modelled here might handle some of these issues related to the realities of automated screening and better reflect gains to society. For example, under strategy 1, even if the automated screening system identified disease but a level 2 grader gave the conservative diagnosis of R0, implementation policy could be designed so that for this patient, the automated system reading could be ignored and no level 3 grading would be needed. This would allow the automated system to capture true-positive and true-negative cases while lowering costs associated with episodes identified by automated screening programmes as false positives.

Gold standard issues: missing proliferative disease

None of the systems had a 100% detection rate for proliferative disease. The study was not designed to look at the accuracy of human grading. No study has specifically looked at the sensitivity of human graders to detect R3 disease. It is clear that human graders at different levels differed in classifying R3, demonstrating that human graders themselves are not 100% accurate in detecting R3. It is striking that, of the 13 episodes classified as R3 by the Homerton DESP graders and sent to arbitration, only four were classified as R3 by the Doheny Image Reading Centre (three were classified as R1 or M1 and four were classified as U).

Secondary analyses

The colour of the fundus is determined by the choroidal blood supply and the amount of pigmentation in the choroid and overlying retinal pigment epithelium. Different ethnicities may have different levels of fundus pigmentation and fundus colour, which may influence the performance of an ARIAS. However, in this study there was no strong evidence to suggest that the sensitivity of the EyeArt or Retmarker ARIASs automated systems varies by ethnicity group but false-positive rates for combined grades R0M0 and R1M0 appeared to be lower in white European patients than in Asian and black African-Caribbean patients. There was some evidence that as patient age increased the ability of both the EyeArt and Retmarker ARIASs to detect combined grades U, R1M1, R2 and R3 fell. Whether or not this is an effect of poorer image quality with an ageing lens remains to be established. The performance of both ARIASs, but partularly Retmarker, differed according to camera type. The largest discrepancy for the detection of proliferative disease ranged from 100% for two out of the five cameras to 80% with another commonly used camera for Retmarker and from 100% to 97.4% for EyeArt. EyeArt appeared robust to variations in age, ethnicity and camera type, whereas Retmarker seemed to be less robust to variations in these patient-level factors. There is a need to understand why one ARIAS may be more sensitive to differences in camera type than another.

Detection of non-diabetic pathology

Although the study was not designed to evaluate the performance of the ARIASs on non-diabetic eye disease, we evaluated this in the subset of patient images sent for arbitration. A total of 467 images from 211 patients were coded with other pathologies by the Doheny Image Reading Centre. No episodes with at least one image coded with presence of age-related macular degeneration (11 patients), myopic degeneration (seven patients) or central retinal vein occlusion (two patients) were classified as ‘no disease’ by either software. Many more images were coded as having intermediate-sized drusen (298 images from 188 patients) or large-sized drusen (133 images from 70 patients). For intermediate-sized drusen 21 out of 133 episodes were graded as ‘no disease’ by EyeArt compared with 48 out of 133 for Retmarker. For large-sized drusen 10 out of 70 episodes were graded as ‘no disease’ by EyeArt, compared with 33 out of 70 for Retmarker. It is clear that a beneficial consequence of ARIASs is that other eye diseases may be detected.

Comparison of performance to previous studies

Altered thresholds

In any assessment of screening, automated or otherwise, there is a trade-off between sensitivity and specificity. Manufacturers of ARIASs fix an ‘operating point’ at which a commercial and clinical case can be made for the system’s effectiveness. As part of the economic evaluation we explored different operating points. With Retmarker, varying the closest obtained threshold to alter the sensitivity by an absolute difference of ± 5%, generated values of specificity between 35% and 52%. EyeArt provided two classifications, one with the software set to detect ‘any retinopathy’ and one when it is set to detect ‘referable retinopathy’. These were implemented as two separate decision statistics rather than two different thresholds on one decision statistic. It was the former decision statistic that was used in the economic evaluation as it seemed to offer the closest comparison with the Retmarker. However, the latter statistic perhaps offered the better basis for cost-effective screening. At thresholds set to deliver sensitivity of between 86% and 96%, it could deliver specificities of between 14% and 54%.

A breakdown of the false-negative results further illuminated the trade-off between sensitivity and specificity. Sensitivity was largely influenced by episodes graded M1 and U. Both systems, but especially EyeArt, remained relatively successful at detecting the most clinically significant grades of retinopathy, even if overall sensitivity was relaxed in favour of specificity. At a threshold set to attain a specificity of > 40%, 98.9% of R2 and 99.2% of R3 cases were correctly identified.

In assessing the metric used in the economic evaluation, ‘cost per appropriate outcome’, it is worth considering that many of the outcomes we have had to regard as ‘inappropriate’ would not have had any adverse clinical outcome. For example, we regarded an episode with a final human classification of ‘U’ as a false negative if classed as being of adequate quality and disease free by the software. Given the large proportion of these ungradable episodes within the false negatives, we looked more carefully at how these were generated. EyeArt, for example, analysed each image and if more than one image in the episode was classified as adequate, the episode was classified as adequate for image quality. Requiring that an episode be classified as U if 50% or more of images were technically inadequate would improve sensitivity, on our metric, from 90% to 93%, albeit at the cost of slightly reduced specificity (from 35% to 32%).

Conformité Européenne marking

The two ARIASs that underwent health economic evaluation in this study, Retmarker and EyeArt, have class IIa CE marking, allowing them to be legally placed on the UK market. Of note, since the initiation of this study, new versions of both software packages have been released but were not tested in this study.

Implementation in real life and generalisability

In contrast to previous studies of ARIASs, multiple systems were compared on the same data set of consecutive patient screening episodes from a relatively large NHS diabetic retinopathy screening programme, using multiple different camera types and on populations of patients from different ethnic groups spanning a wide age range. All steps of the study were undertaken by study personnel independent of ARIAS vendors. The ARIAS vendors were locked out of the systems after the testing phase of their software. We are confident that the results are generalisable to NHS screening programmes. We noted a variation in screening performance by camera type and it may be useful to run test sets of previously graded images for each camera before implementation of a particular ARIAS in a new programme.

Limitations of our study

This study is limited by its time horizon. We mapped an acute episode in the diabetic retinopathy screening programme, as in Scotland et al. ,21 but not the ongoing activities of the programme. Importantly, missed episodes of retinopathy could possibly have implications for patient sight. Recent work provides the transition probability data necessary to undertake this analysis (Catherine Egan, Moorfields Eye Hospital, 2016, personal communication) and should be an area of future examination.

Although a randomised controlled trial of ARIAS versus manual grading may seem desirable, the problem with a randomised controlled trial in the context of unproven software is that, for safety/ethical reasons, the investigative pathway would need to incorporate a safety check to ensure that cases of sight-threatening diabetic retinopathy were not missed by the ARIAS. This would likely lead to discrepancies that could result in an intervention in the screening pathway altering the outcome of missed retinopathy, which in turn would alter the referral pathway for a patient. Therefore, there would be no advantage of having a prospective randomised trial, as the intervention would obviate the ability to judge the implications of missed retinopathy over time. However, as there were three ARIASs tested, the advantage of a non-randomised retrospective study design is that all three ARIASs can be tested on exactly the image set. This created limitations in our analysis because of the need to map ARIAS findings onto manual grading results. The reference standard based on manual grading varies, in terms of if a screening episode received grading by level 1, 2 or 3 graders. According to the pathway protocol for strategy 1 (replacement of level 1 grader with ARIAS) if the ARIAS classified the episode as ‘disease’ the episode should receive a level 2 grade. However, some patients did not receive a level 2 grade in real life if the level 1 grader determined that no disease was present. In order to populate the health economics model we needed to assign level 2 and 3 grades to patients with automated screening outcomes that would have gone on to the level 2 grader. For these cases, we assumed a level 1 grader would act as a level 2 grader and a level 2 grader would act as a level 3 grader. If level 1 graders are, in reality, less experienced than level 2 graders, this may have underestimated the efficacy of automated screening programmes under strategy 1. However, comparison of results from both strategy 1 and 2 suggests that even if such a bias existed, its effect on final cost-effectiveness assessments was nominal.

In practice, level 2 graders are blind to grades received from level 1 graders, but level 3 graders are not blind to previous grades and would not be blind to automated system classifications. We assumed that level 1 graders are independent of later grades. In reality, however, it is possible that level 2 and 3 graders may be influenced by the disease/no disease classification from the automated system or from diagnoses given from the level 1 manual grader. Given the greater precision of manual grading diagnoses, one would expect level 3 grading scores to be more sensitive to level 1 manual grading scores than ARIAS classifications. This may mean that our cost-effectiveness assessment would vary slightly if ARIASs are implemented. Nevertheless, this ex ante modelling approach was necessary because ARIASs have yet to be adopted in the NHS.

Our findings are also limited by the effectiveness variable chosen. We chose ‘appropriate outcome’ because this allows us to analyse the screening goals of capturing true positives and true negatives together. It has also been used in other existing papers on this topic. 20,21 However, the end result for all patients either having automated screening or manual grading will be referral or annual rescreening. Pathway-specific safety checks by way of secondary evaluations of patient episodes by level 1, level 2 and level 3 manual graders after automated screening may avert inappropriate patient referrals or rescreens from automated screening scores. In reality, it is likely that the cost-effectiveness assessment produced here overestimates welfare losses occurring as a result of high false-positive rates in ARIASs. Although these systems do perform poorly under some circumstances, namely when disease is identified, tentative implementation protocols do not allow for direct referral from ARIAS disease classification. At the extremes, ARIASs are observed to demonstrate high rates of sensitivity and specificity, that is, true-negative and true-positive detection rates are high in ARIASs and comparable with level 1 manual graders. However, the impact on patient referability in the case of false positives is likely to be mitigated in practice through the existence of safety checks by way of level 2 and level 3 graders. Observational trials evaluating ARIAS effectiveness by way of correct patient referral or rescreen rates are needed to validate this hypothesis.

Our interpretation of results was also limited by the fact that the automated screening systems had differing ways of treating technical failures. Retmarker had two output options: disease and no disease. Technical failures were classified as disease positive to reflect manual grading protocols. Although this does not affect the main results, it did not allow us to differentiate the reasons behind ‘disease’ classifications for Retmarker. For EyeArt, technical failures were also included in the disease-positive arm of our decision tree, although this software does offer a technical failure differentiation. We treated technical failures as the same for comparability purposes but further analyses may wish to differentiate technical failures and disease-positive classifications to model the cost-effectiveness of EyeArt. Given the high rate of false positives associated with both automated screening systems, disaggregating disease-positive classifications into disease positive and technical failure may allow researchers to examine the inherent effectiveness of ARIASs after controlling for potentially confounding latent variables (e.g. dust on image).

Should these technologies be adopted, future experience with implemented automated screening systems in the NHS for diabetic retinopathy may allow for a better understanding of the costs associated with automated screening systems. Although we used sensitivity analysis to examine this point, the practicalities of implementation could be better quantified on actual experience.

Prioritised research recommendations

The following is a list of research recommendations from this study:

1. Planning for an ARIAS-based programme: we propose technical standards for image acquisition and recommendations for service provision. Testing on previously acquired images from a particular programme before implementation will be important in the development of a screening programme protocol.

2. Messaging development: develop pathways of communication for current and future ARIASs to exchange information and images between the primary screening software and ARIASs.

3. Image repository test set: set up an image repository of the test images used in this study to allow for rapid evaluation of newer software versions of currently available ARIASs, as well as new CE-marked ARIASs that have become available since this study. The data set should include images that were not read by arbitration graders to provide a resource representative of ‘real life’.

4. Image repository training set: set up an imaging repository of graded images that are not part of the definitive evaluation set, which will help accelerate the development of more effective ARIASs.

5. Image metadata: an early priority will be to develop quality standards for retinal imaging that may make the potential effectiveness of ARIASs even greater.

6. New ARIAS testing: it is important to establish a protocol that allows standardised evaluation of updated/novel ARIASs to be undertaken, optimising rapid uptake in screening practice.

7. Screen interval: it will be important to assess if annual or biennial screening after two no disease/disease cut-off points is appropriate for a future programme using ARIASs.

8. Health economics transition: probabilities of progression between grades of retinopathy have just become available. A Markov Model for evaluating ARIAS cost-effectiveness in screening for diabetic retinopathy could now be developed using these recently available data.

9. Wide field imaging modalities: if wide-field imaging becomes available for diabetic retinopathy screening, a modified training and validation set should be developed to help validate ARIAS use in this context.

Future prospects

In our opinion, a technical panel should be set up with sufficient expertise to advise Public Health England on technical and integration standards that would help create a market for ARIAS. We suggest that the panel should be independent of all commercial vendors but may seek advice from vendor representatives. A standard ‘real-life’ data set should be set up (e.g. the data set from this study) and an adjudicating panel that can evaluate the test performance of new/updated ARIASs and consider implementation into the NHS, independently of the software vendors.

Export of the images to the ARIASs in this study relied on a tool commissioned to transfer anonymised images from the Digital Healthcare screening database to a location for the ARIASs to analyse. This system was not automated. Standards to allow automated data interchange between ARIASs and the primary screening systems are needed for effective implementation. All of the software tested relied on manual instructions to initiate processing of the images. Future work should not only include DICOM compatibility but also a standardised method for automating the initiation and processing of output from the grading programmes, without the need for human intervention.

The workflow of the output from the ARIAS should be optimised to link back to the screening system, to set up a grading queue and randomly sample a proportion of images for arbitration.

The ARIASs may be even more cost-effective if fully integrated into the screening system to allow for real-time grading. This may allow for selective OCT to be done at one patient visit.

We noted a variation in grading test performance by camera type. This may be because of defects in the camera’s image quality (e.g. dust on the lens). Developing quality standards for the retinal imaging process would be helpful.

Diabetic retinopathy screening as currently implemented with manual graders is not designed to pick up other non-diabetic eye disease but may detect other pathology as a by-product of implementation. This study was not powered or designed to look at non-diabetic retinopathy eye disease, such as a cataract. It may be that a combination of a visual acuity filter and image quality assessment will help pick non-diabetic retinopathy eye disease. This should be considered in the pathways during implementation.

Steering Committee

We would like to thank the members of the Steering Committee for their time and invaluable advice.

Steven Aldington: independent member.

Mireia Jofre-Bonet: non-independent member.

Simon Jones: independent member.

Irwin Nazareth: independent member.

Adnan Tufail: chief investigator member.

Irene Stratton (Senior Statistician Gloucestershire Retinal Research Group): chairperson, independent member and statistician member.

Gillian Vafidis: independent member.

Richard Wormald (Sponsor Representative): non-independent member.

Contributions of authors

Adnan Tufail (Consultant Ophthalmologist, Moorfields Eye Hospital, and Professor of Ophthalmology Institute of Ophthalmology, University College London) was chief investigator of the study. He was involved in the conception and design of study, collection of data, interpretation of data and writing the final report.

Venediktos V Kapetanakis (previously Senior Research Fellow in Medical Statistics, St George’s, University of London): was involved in the statistical analysis, acquisition of data, database management, interpretation of data and writing the final report.

Sebastian Salas-Vega (Doctoral student in Health Economics, Department of Social Policy and LSE Health, London School of Economics and Political Science): health economic analysis, interpretation of data and writing the final report.

Catherine Egan (Consultant Ophthalmologist, Moorfields Eye Hospital NHS Trust, and Honorary Senior Lecturer Institute of Ophthalmology, University College London) was co-investigator of the study. She was involved in the conception and design of the study, interpretation of data and writing the final report.

Caroline Rudisill (Associate Professor in Health Economics, Department of Social Policy and LSE Health, London School of Economics and Political Science) was involved in the conception and design of the study, acquisition of data, health economic analysis, interpretation of data and writing the final report.

Christopher G Owen (Professor of Epidemiology, St George’s, University of London) was involved in the conception and design of the study, interpretation of data and writing the final report.

Aaron Lee (previously Retina Fellow, Moorfields Eye Hospital NHS Trust) was involved in the set up and activation of ARIASs, database management and editing of the final report.

Vern Louw (Data Warehouse Developer, IT Department, Moorfields Eye Hospital NHS Trust) was involved in the set up and activation of ARIASs and database management.

John Anderson (Consultant Physician and Endocrinologist, Homerton University Hospital Foundation Trust) was involved in the conception and design of the study, interpretation of human grading data and costs and review of the final report.

Gerald Liew (previously Retina Fellow, Moorfields Eye Hospital NHS Trust) was involved in data management, operational issues, interpretation of data and writing of the final report.

Louis Bolter (Divisional Service Manager, Homerton University Hospital Foundation Trust): was involved in the interpretation of human grading data and costs, support for the real-time study and review of the final report.

Clare Bailey (Consultant Ophthalmologist, Bristol Eye Hospital) was involved in the conception and design of the study and writing the final report.

SriniVas Sadda (Professor of Ophthalmology, Doheny Eye Institute, and Medical Director, Doheny Image Reading and Research Laboratory) was involved in the arbitration of study images and editing of the final report.

Paul Taylor (Reader in Health Informatics, Centre for Health Informatics & Multiprofessional Education (CHIME), Institute of Health Informatics, University College London) was involved in the conception and design of the study, analysis of altered thresholds, interpretation of data and writing the final report.

Alicja R Rudnicka (Reader in Medical Statistics, St George’s, University of London) was involved in the conception and design of the study, acquisition of data, statistical analysis, interpretation of data and writing the final report.

Publications

Kapetanakis VV, Rudnicka AR, Liew G, Owen CG, Lee A, Louw V, Bolter L, Anderson J, Egan C, Salas-Vega S, Rudisill C, Taylor P, Tufail A. A study of whether automated Diabetic Retinopathy Image Assessment could replace manual grading steps in the English National Screening Programme. J Med Screen 2015;22:112–18.

Owen CG, Rudnicka AR, Kapetanakis V. Automated diabetic retinopathy image assessment software: diagnostic accuracy and cost-effectiveness compared to human graders. Ophthalmology 2016; in press.

Data sharing statement

We shall make data available to the scientific community with as few restrictions as feasible, while retaining exclusive use until the publication of major outputs. Anonymised demographic data can be obtained by contacting the corresponding author. The corresponding author welcomes proposals for collaborative projects using the image data set.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health.