Home-monitoring for neovascular age related macular degeneration in older adults within the UK: the MONARCH diagnostic accuracy study

Article history

The research reported in this issue of the journal was funded by the HTA programme as award number 15/97/02. The contractual start date was in October 2017. The draft manuscript began editorial review in April 2022 and was accepted for publication in December 2022. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ manuscript and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this article.

Copyright statement

Copyright © 2024 Hogg et al. This work was produced by Hogg et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This is an Open Access publication distributed under the terms of the Creative Commons Attribution CC BY 4.0 licence, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. See: https://creativecommons.org/licenses/by/4.0/. For attribution the title, original author(s), the publication source – NIHR Journals Library, and the DOI of the publication must be cited.

2024 Hogg et al.

Background and rationale

Despite significant therapeutic advances, neovascular or wet age-related macular degeneration (AMD) remains the leading cause of blindness in older adults. 1 While the early stages of AMD have subtle visual symptoms, the advanced stages [neovascular age-related macular degeneration (nAMD) and geographic atrophy (GA)] result in substantial retinal damage with accompanying loss of central vision and reduced quality of life. 2

A range of intravitreal drugs that inhibit vascular endothelial growth factor (VEGF; anti-VEGF antibodies) for nAMD are currently available, with treatment generally starting with a loading phase of three injections over 3 consecutive months. A proportion of eyes become fluid free in the subsequent maintenance phase, but relapse is common and most patients require retreatment in affected eyes at some stage, with the disease typically becoming inactive for a period and then becoming active again. 3 Patients in the maintenance phase with inactive disease still need to be monitored in hospital by measurement of best corrected visual acuity and optical coherence tomography (OCT). When disease reactivation is detected, treatment is restarted.

A large study has shown that, while many patients have many months of treatment-free periods, a significant burden falls on hospitals (as well as patients) with respect to the need for regular and repeated review. 3 Thus, methods that might allow the patient to self-monitor at home would reduce the burden on hospitals.

When diagnosis of active nAMD is confirmed, treatment with anti-VEGF therapy is almost always initiated. In most cases, patients receive three injections every 4–6 weeks initially (loading phase) and then patients are reassessed at each subsequent visit in the treatment cycle to determine lesion activity and decide whether retreatment is necessary (maintenance phase). Monitoring visits use a combination of visual acuity, clinical biomicroscopic examination and OCT to determine if the neovascular lesion is active (wet) or inactive (dry). It is these monitoring appointments which are causing a significant strain on NHS outpatient clinics in eye hospitals.

Pre-coronavirus disease (COVID), during the maintenance phase, patients were monitored for relapse at regular monitoring outpatients’ visits at eye hospitals. The frequency of monitoring depended on the drug being used (ranibizumab or aflibercept) and the preferred treatment regimen [treat (active disease)-as-necessary or treat-and-extend]. Ranibizumab is licensed for monthly treatment as required, and aflibercept every 2 months, during the maintenance phase. For treat-as-necessary regimens, a lesion found to be active is treated and a further monitoring visit is arranged; treatment is withheld if the nAMD lesion is inactive, and a further monitoring visit is arranged. For treat-and-extend regimens, ‘prophylactic’ treatment is administered to an eye with an inactive lesion, extending the interval between monitoring visits providing the disease remains inactive; if the nAMD lesion is found to be active, the interval between monitoring visits returns to the standard interval (1 month for ranibizumab or 2 months for aflibercept until the lesion becomes inactive, and the interval is then extended again). A disadvantage of the treat-and-extend treatment regimen is that it can lead to unnecessary overtreatment.

The seminal clinical trials of anti-VEGF therapy for nAMD used change in best corrected distance visual acuity as their primary end point. 4–6 This was measured using strict protocols to standardise refraction methods, visual acuity recording, distance measurements and illumination between trial sites. Visual function was used as a surrogate biomarker for the therapeutic response to treatment and other functional tests together with retinal imaging parameters were used as secondary outcomes. The clinical trials mainly reported results at 2 years, though most continued to follow patients to provide longer-term data and information of time to reactivation was obtained from the IVAN study (personal communication). A database observational study of treat-and-extend patients in Australia reported that the risk of reactivation rose from 2.2% for a 6-week interval to 15.6% for 20 weeks. 3

In this diagnostic test accuracy study, monitoring of patients continued as usual in eye hospitals. Ophthalmologists in the six centres continued their preferred drug and treatment regimen to monitor and treat nAMD in their patients. The study added weekly home monitoring, using three different tests (time 20–40 minutes), to the usual care pathway. Where possible, data from both eyes were collected.

The reference standard was the usual care clinical decision about the activity of nAMD in the study eye at the hospital outpatient appointment. Each participant remained in the study throughout the follow-up period. Therefore, there will be some monitoring visits when the study eye is judged by the participant’s ophthalmologist to have active disease and some visits when the study eye is judged to have inactive disease. The primary quantitative analysis compared the results of the home-monitoring tests during the interval preceding the monitoring visit with the reference standard assessed at the monitoring visit.

Because of (1) the clinic workload in treating and monitoring nAMD patients and (2) the high cost of establishing a robust reference standard for people at high risk of nAMD but not currently being monitored by the NHS, we decided that the most urgent priority was to identify a home-monitoring test that could detect reactivation in the patients currently being managed in the NHS. We envisaged that, ideally, after diagnosis, patients would have their initial loading-dose injections in a hospital clinic and would then be discharged with the home-monitoring test; if the test subsequently indicated a deterioration in their vision, they would arrange an urgent appointment. The focus of NHS hospital nAMD clinics would then shift to providing urgent appointments to administer treatment, rather than regular monitoring.

Since the start of the study, several other studies have evaluated home monitoring in AMD; this has also been facilitated by the coronavirus disease 2019 (COVID-19) pandemic which necessitated a focus on home monitoring and forced an increase in digital literacy in the older age groups. A recent study which recruited patients with diabetic macular disease prior to the lockdown in October 2019 and after the lockdown in February 2020 provided patients with the Alleye (Oculocare, Switzerland) app on their own smartphone or provided them with it preloaded on an apple iPod Touch_TM (6th generation, Apple, Cupertino, CA, USA) reported that the smartphone home tests were able to indicate worsening pathology and the need for treatment. 7

The KeepSight Journal

The KSJ encouraged weekly monitoring using a paper journal. It included three different monitoring strategies, viewed one eye at a time. Firstly, near-visual acuity was assessed using a puzzle (crossword or word search) employing a variety of font sizes (an example is shown in Figure 1). Secondly, patients were encouraged to view objects with straight lines in the home to check for distortion (wall panelling, floor tiles, venetian blinds, etc.). Finally, they used a modified Amsler chart (VMS grid) to record areas of distortion or scotoma in their vision. The KSJ has been used before; 198 patients with intermediate AMD (at high risk of progression to late stage) were randomised to use the KSJ to self-monitor or usual care, with follow-up at 6 and 12 months to assess adherence. 11 The results showed significantly better adherence in the journal group with the findings supporting the efficacy of the journal for increasing vision self-monitoring adherence and confidence while promoting persistence in weekly monitoring.

FIGURE 1.

Example week in the KSJ.

MyVisionTrack

MyVisionTrack is a software application (app) on an iPod Touch device. It is a shape discrimination test which measures hyperacuity, by displaying four circles, one of which is radially deformed (‘bumpy’ rather than perfectly circular). Viewing the display monocularly, the patient has to identify the odd one out (Figure 2). Studies have shown that the task implemented on an iPod Touch can distinguish between intermediate and advanced nAMD and a survey reported that 98% of patients found the test easy to use. 18 Studies at the Liverpool site led by Paul Knox (co-investigator) successfully used the test in macular clinics and patients have found the test straightforward to complete. 19,20

FIGURE 2.

Diagram illustrating the steps when self-monitoring with the mVT app.

MultiBit test

MultiBit test (MBT) is also an app on an iPod Touch. It is a near-acuity threshold test of neuroretinal damage. Traditional tests fail to detect such damage because they are suprathreshold. The MBT displays receptive field-sized dots or ‘rarebits’, which provide a miniscule amount of information to the visual system compared to conventional targets (Figure 3). Patients are presented with pairs of numbers; they state the numbers that they see out loud and the numbers are then represented at high contrast together with a recording of the patient’s responses. MBT is the only test with published data describing its performance to alongside changes in nAMD activation. 21 It was used to track 29 patients during treatment and monitoring in NHS outpatient clinics (average 39 weeks follow-up), with patients monitoring themselves at home with an iPod Touch. 21 MBT performance improved gradually after treatment, stabilised during periods of disease inactivity and deteriorated gradually preceding reactivation. MBT performance also agreed well with retinal imaging clinical assessments but not with visual acuity (known to be an insensitive test of reactivation).

FIGURE 3.

Diagram illustrating steps when self-monitoring with the MBT app.

Potential inequalities in uptake

The study also aimed to address the question: How do demographic, socioeconomic and visual function factors influence the uptake of home-monitoring tests for detecting active nAMD?

A survey by Age UK in 2013 found that internet use among people aged 65 years or over varied across the UK, with a ‘north–south’ divide; more than 50% in the south (Surrey, Bedfordshire, Buckinghamshire, Suffolk and Oxfordshire) used the internet, but less than a third in the north (Cumbria, Yorkshire, Hull, Tyne and Wear). 23 With respect to smartphone use, only 20% of 65- to 74-year-olds used such a device to access the internet in 2013;24 perhaps more importantly, this percentage had increased from only 12% in 2012, suggesting that the situation is changing rapidly over time. The potential importance of failure to access the internet has been highlighted by a study of men and women in the English Longitudinal Study of Ageing from 2004 to 2011 internet use was found to be significantly ‘protective against health literacy decline’. 25

During the study design phase, the small percentage of regular internet and smartphone users was considered a potential threat to the study; we were especially concerned that potential participants might feel alienated by the technology and would not be prepared to try out the solutions we proposed. Therefore, we sought to determine the extent to which the technology was a barrier to consent and participation in order to project wider adoption of home monitoring in the future if it is found to have satisfactory performance. Moreover, among participants, it is possible that some tests will be easier to do for participants with limited experience of smart devices and the internet. This was an important factor to weigh against test performance if differences in test performance were found to be small.

We designed the study to include the following features to try to minimise the extent to which technology could be a barrier to home monitoring:

1. We included a simple paper-based home-monitoring test, which we hoped would feel familiar to participants. This test involves a series of puzzles which require participants to use their near-vision correction.

2. We also provided a mobile broadband device so that participation was not limited by the lack of home Wi-Fi. The device had a simple on/off switch; the only things that a participant needed to remember to do were to keep the device charged (a main micro-USB charger was provided) and to switch on the device before performing the home-monitoring tests that use the iPod. The iPod interacted with the mobile broadband device automatically to transmit data.

We explained the use of the devices during an initial training and information session with each potential participant and also provided a help line for participants to call in the event of the experiencing difficulty.

Aims and objectives

The aim of the MONARCH study is to quantify the performance of three non-invasive test strategies for use by patients at home to detect active nAMD compared to diagnosis of active nAMD during usual monitoring of patients in the Hospital Eye Service (HES).

The study had five objectives:

1. Estimate the test accuracy of three tests to self-monitor reactivation of nAMD compared to the reference standard of detection of reactivation during hospital follow-up with OCT imaging, clinical examination and Early Treatment Diabetic Retinopathy Study (EDTRS) visual acuity.

2. Determine the acceptability of the tests to patients and carers and their adherence to home-monitoring testing regimens.

3. Explore whether inequalities (by age, sex, socioeconomic status and visual acuity) exist in recruitment to the study and impact the ability of participants to do the tests during follow-up and the adherence of participants to weekly testing.

4. Provide pilot data for the use of home monitoring to detect conversion to nAMD in the fellow eyes of patients with unilateral disease, compared to the reference standard of detection of conversion during hospital follow-up with EDTRS visual acuity and OCT imaging.

5. Describe the challenges of implementing and evaluating home-monitoring technologies (see Changes to trial design after commencement of the trial).

The study population recruited for Objective B (the integrated qualitative study) differed from that required for Objectives A, C and D. Only selected sites, Belfast, Moorfields and James Paget, participated in procedures and data collection for the patient and carer aspects of Objective B. All participating NHS centres took part in the healthcare professional (HCP) aspect of Objective B.

Study design

The MONARCH study was a multicentre diagnostic test accuracy cohort study to estimate the sensitivity and specificity of home-monitoring tests to detect active nAMD in patients previously diagnosed with nAMD and quiescent after treatment.

MONARCH was designed to compare the results of the home-monitoring tests being evaluated (‘index tests’, see Index tests) with the results of a reference standard (see Clarification of reference outcome). These comparisons allowed the accuracy of the index tests to be quantified with respect to the reference standard.

Participants were followed for at least 6 months, accruing on average six clinic attendances at which home monitoring and reference test results were compared.

The nature of active nAMD may change over time since diagnosis, if the disease progresses despite monitoring and treatment. Therefore, the study population was stratified by time since first treatment of nAMD in the first-treated study eye (see Stratification of study population). This design was to avoid the prolonged duration of follow-up which would be required if, instead, we were to follow participants from diagnosis to an equivalent time in the natural history of their condition. These design features are shown in Figure 4.

FIGURE 4.

Study schema: Objectives A, C and D.

All study data collections were planned before the first participant was recruited, although additions were made to the case report forms (CRFs) in the first month to clarify some data items, for example, eye classification. Case report forms are available at: https://fundingawards.nihr.ac.uk/award/15/97/02.

Participant eligibility

The study population for the quantitative part of the study were patients with at least one study eye being monitored by HES for nAMD, stratified by time since starting treatment in the first-treated study eye (6–17 months; 18–29 months; 30–41 months) to ensure test performance was estimated across this range of duration of nAMD.

Settings

The study was run in the homes of patients being monitored by HES for nAMD at participating NHS hospitals and in the participating NHS hospitals.

Participants were recruited in secondary care (HES clinics). During the study, the reference standard for an eye being monitored was determined at HES clinic visits. During intervals between clinical visits, participants used home-monitoring tests to test their vision themselves. Participants were asked to complete the home-monitoring tests themselves weekly at home.

Index tests

There were three home-monitoring (‘index’) tests spanning low to moderate cost and complexity. These were:

1. KSJ test: a paper-based booklet of near-vision tests presented as word puzzles developed in the USA by KeepSight and adapted by the study team for use in the study (UK wording and defining test scores for estimating diagnostic test accuracy).

2. mVT: electronic vision test, owned by Genentech Inc., a member of the Roche Group, intended to be viewed on a tablet device.

3. MultiBit test (MBT): electronic vision test, developed by Visumetrics, licensed by Novartis, intended to be viewed on a tablet device.

Each electronic test took approximately 5 minutes (2–3 minutes per eye) to complete, though some participants took longer. The KSJ took between 10 and 20 minutes (5–10 minutes per eye) depending on the difficulty of the puzzle. Therefore, weekly monitoring was expected to take 20–40 minutes (10–20 minutes per eye). Participants were asked to try to complete all three of the index tests for both eyes but could opt to stop up to two of the three tests and continue with the study.

Each KSJ booklet lasted approximately 6–7 months. Every 6 months, participants were sent a new booklet and instructed to return the used booklet in a pre-paid envelope to the study management team at Clinical Trials and Evaluation Unit (CTEU) Bristol.

Data were collected automatically from the electronic tests.

Sample size

Outcomes

Data collection

Retinal images

Patients were asked to provide consent to retinal image collection (optional). For participants who agreed to retinal image collection, all retinal images (e.g. CF, OCT) taken during follow-up were submitted to CARF to be stored for use in future ethically approved research.

Given the higher-than-expected proportion of active lesions, we decided that the stored retinal images should be assessed for presence of activity by senior staff from the NetwORC UK reading centre (www.networcuk.com/) according to a standardised protocol. All available images were used to make the assessment. Presence of the following features was assessed in both eyes:

• Presence of subretinal fluid.

• If present, was it more or less than previous visit?

• Central subfield thickness (Figure 5) is recorded from the OCT machine automatic algorithm.

FIGURE 5.

Red arrow shows central subfield thickness from OCT scan.

Features to minimise bias

Risk of bias was considered with respect to bias domains described in an appraisal tool for diagnostic accuracy studies. 30 Information here should be read in conjunction with information about the proposed methods of analyses (see Methods for Objective E).

Qualitative methods (Objective B)

Semistructured interviews were conducted face to face and via telephone. The interview schedule (see Appendix 1, Table 29) was based on the experience of the research team and was informed by models and theories of technology acceptance. The members of the team that collected and analysed the data have extensive experience in the application of qualitative methods in healthcare research. The study followed the consolidated criteria for reporting qualitative research (COREQ) criteria. 31

Methods for Objective E

Due to the age range of the patient population, the study team anticipated that participants with little or no experience in using electronic devices may have difficulty operating and interacting with the study equipment [iPod and Mobile Wireless-Fidelity (MiFi) router]. As described, participants’ exposure to various types of electronic technology was recorded at consent. Reasons for not consenting were also recorded when participants did not want to engage with the technologies required for the study.

Throughout the study, at or before each follow-up appointment, the local research staff asked participants if they were experiencing any issues with the equipment and additional training sessions were offered. This information was captured in the CRFs and was used as a means of troubleshooting participants technical issues and maintaining participant testing.

Participants were encouraged to contact the study team on the technical support to resolve technical issues (incoming calls). Details of all calls received were logged, including call length, issues discussed, actions taken and resolution status. The helpline acted not only to capture the details and frequency of various technical issues, but also to resolve them and maintain participant testing.

The study team realised early on during the study that data were not being received from some participants, often immediately following consent. To try to identify and resolve any technical issues that participants may be facing, periodic calls were made from the helpline to participants for whom data had not been received in more than 3 weeks or more than 2 weeks since consent (outgoing calls, only to participants who had consented to being contacted by telephone). Logging these calls documented the challenges participants were experiencing, and also meant that members of the research team could try to resolve any technical issues, or prompt participants to test. The same details were recorded as for incoming calls.

In addition to periodic calls to participants, a SMS text message notified a participant if test data had not been received in more than 3 weeks since last test or more than 2 weeks since consent, or to thank them if their data were being regularly received (when consent to send these messages had been given). These messages were intended to act as a prompt to participants to test or to contact the study team if they were experiencing technical issues. All notifications sent to participants were logged.

The study team experienced a range of different logistical and technical issues relating directly to the study equipment and applications used. All issues with the study equipment and the apps were logged with as much detail as possible and are described in the results, including time frames, impact on participants and resolutions, where possible.

Statistical methods

A pre-specified analysis plan was written before analyses were carried out.

Patient and public involvement

From the beginning we recognised that effective patient and public involvement (PPI) was central to the delivery of high-quality research. Discussions with patients, clinical and academic colleagues informed the initial proposal and, in particular, the frustration articulated by patients about the burden of monthly monitoring hospital visits, both for them and their carers. This was the stimulus for our focus on home monitoring for patients already being treated for nAMD. During the design phase, both the Royal National Institute for the Blind (RNIB) and the Macular Society were contacted about the proposal and provided helpful feedback. One of the tests was also demonstrated to, and discussed with, a group of current AMD patients. They welcomed the development of technologies for home monitoring and said that they would be interested in using the test at home to monitor their vision. Feedback from a larger group of patients who had used the test as part of a formal study was also sought, and they confirmed that the test was easy to use, even for older patients with little or no experience of using ‘smart’ devices like the iPod Touch. Throughout the study, we sought to use the INVOLVE principles and resources to guide our active involvement with patients and the public. A representative from both the RNIB and the Macular Society agreed to sit on the SSC as these are both key stakeholders for patients with AMD and severe sight loss.

A group consisting of both four patients and two carers was also established [managed by Dr Hogg and a post doctoral research associate (PDRA) based in Belfast] provided crucial feedback on appropriateness of the study materials, with particular focus on the design of the training materials for using the apps and the study equipment. They also provided feedback on content of visits and frequency of procedures. Input from the PPI group was very helpful for the content of the KSJ, firstly in adapting the original one for a UK audience and then for the development of the subsequent versions.

Specific feedback that proved very useful included:

• Removal of the motivational quotes from the KSJ as the PPI group member did not like them and felt they were not appropriate for a UK audience.

• Requested removal of italicised text in the equipment instruction sheet.

• Members did not like the term ‘carer’ as they felt very few patients would describe their spouse or friend/family member who helps them as a carer and requested the patient information be changed accordingly.

• They felt there was too much reading in the PIL and so it was explained that a lot of it was necessary for legal purposes.

• They requested removal of coloured text from the study newsletter and replaced by either black on white or black on yellow text.

• They expressed no strong preference as to how they were contacted throughout the study but felt that updates via a study website would not be effective, so it was decided to not produce a dedicated study website.

Recruitment

Details of patients approached and screened are described in Appendix 1 (see Table 31). The flow of patients approached and screened, and the numbers of patients who were willing in principle, trained to use the index tests, eligible and who consented to take part are shown in Figure 6 and summarised in Table 1. Recruitment over the lifetime of the study (21 August 2018 to 31 March 2020) is shown in Figure 7. The last patients were recruited in March 2020, before the first UK lockdown due to COVID-19 (see Effect of COVID-19 pandemic on the study). A total of 297 patients (participants) consented to take part.

FIGURE 6.

Patient flow chart. (1) Figure counts all participants with at least one management decision visit, irrespective of whether the management visit was complete (includes imaging and lesion status information) or whether they withdrew before the end of the study. (2) Percentages above are all calculated with respect to the number approached.

FIGURE 7.

Cumulative number of participants recruited for Objectives A, C and D.

Overall, eligibility was high at screening (943/974, 97%), about two-fifths of screened patients were willing in principle to participate (312/814, 38%), and of those willing, the majority attended training and consented (297/312, 95%).

Reasons for ineligibility at screening and training are shown in Appendix 1 (see Table 32). The most common reasons for ineligibility at screening were due to patients not being previously treated or monitored for active nAMD (20/31, 65%) or to vision being limited by another condition (18/31, 58%). The most common reason for ineligibility at training was due to patients being unable to perform the MultiBit (MBT) electronic test (8/9, 89%).

Reasons for being unwilling in principle to participate are shown in Table 2. Being ‘put off by technology’ was the most common reason (19%, 95/502); other commonly cited reasons included ‘personal reasons’ (17%, 86/502), not interested (17%, 83/502) and ‘other’ (17%, 87/502).

The time taken from screening to a receipt of a willingness decision, and from expressing willingness to training, are shown as a survival analysis by site and overall in Appendix 1 (see Figures 21 and 22). Most centres received a decision from around 80% of their patients within 200 days. Exceptions were James Paget, with received decisions from 80% of patients after < 100 days, and Moorfields with a maximum of 60% of decisions received in total. About 90% patients had received their training within 50 days of notifying their willingness; patients from Moorfields and James Paget received their training on the day when they expressed willingness.

Recruitment by stratum of time since first treatment is shown in Table 3. Half of recruited participants were first treated for nAMD 6–17 months before consenting, 28% 18–29 months prior to consent and 22% 30–41 months prior to consent. Eighteen of 297 participants (6%) were retrained (see Appendix 1, Table 33), most of whom were being managed at James Paget (11/18, 61%).

Monitoring visits

At the end of the study, data for at least one monitoring visit after starting to use the index monitoring tests were available for 357 study eyes in 297 patients. Data were available for at least one complete monitoring visit after starting to use the index monitoring tests for 317 study eyes in 261 patients. The cumulative numbers of monitoring visits, and complete visits (including an OCT), over time are shown in Figure 8. The number of complete visits decreased after the first UK COVID-19 lockdown in March 2020 because many appointments after this date were virtual consultations or treatment was administered without formal review. This situation slowly improved in the following months. Despite the challenges of the COVID-19 pandemic, 91% of the recorded monitoring visits were classified as complete for study eyes. Further reporting of the results focuses on the complete visits, except where indicated. Further information about total visits is described in Appendix 1 (see Table 34).

FIGURE 8.

Number of monitoring visits recorded on database.

The percentages of monitoring visits classified as complete are shown overall and by site in Table 4. Moorfields had the lowest proportion of complete monitoring visits for study eyes (259/331, 78%) and Southampton (153/156, 98%) and James Paget (563/563, 100%) had very high proportions. For fellow eyes, the overall proportion of complete visits was 92%, with a similar pattern across sites. The rates of total monitoring visits per participant per year (not influenced by site-specific decisions about how to provide care during the first lockdown) was 4.01 consistent with the assumption set out in the sample size estimate in the protocol. However, this rate varied markedly by site (2.03–6.39 management visits per patient per year; see Appendix 1, Table 34). Sites also varied with respect to agreement between intended and actual review schedules (see Appendix 1, Table 35). Overall, 53% of monitoring visits occurred on or before the intended review date.

The reference standard was available for all complete management visits for both study and fellow eyes. For study eyes, the classifications were: active = 932 (60.1%), inactive = 576 (37.2%), uncertain = 40 (2.6%) and no lesion = 1 (0.1%). For fellow eyes, the classifications were: no lesion = 561 (90.6%), active = 34 (5.5%), inactive = 19 (3.1%) and uncertain = 5 (0.8%).

Baseline characteristics

Overall baseline demographic characteristics and exposure to technology of consented participants are described in Table 5. More participants were women (58.6%). The mean age was 74.9 (6.6) years (SD) and the mean visual acuity of study eyes (better seeing eyes if participants had two study eyes) was 0.2 (0.2) Logarithm of the minimum angle of resolution (LogMAR) (SD). Many participants had known risk factors for nAMD; for example, 10.1% were current smokers and 53.2% had hypertension requiring treatment. Participants reported a range of comorbidities, including 20.1% malignancy, 2.4% type 1 diabetes, 10.4% type 2 diabetes and 5.1% other conditions that may have affected their ability to adhere to home monitoring. Most participants had substantial exposure to technology before entering the study; 66.6% had a smartphone, 66.2% had a tablet and 85.1% had internet at home.

Objective C investigated inequalities that may have arisen with home monitoring (see Statistical analyses and report content). The demographic characteristics of patients who were willing in principle versus those who were unwilling in principle are reported in Table 6 by site.

Details of baseline ocular histories of study eyes are shown in Table 7. (Note that the total of 347 study eyes includes two study eyes for 86 participants.) At baseline, 75% of study eyes were recorded as having an active nAMD lesion, 71% of study eyes were being treated with or were most recently treated with Aflibercept and 10% of study eyes had a cataract.

Participant withdrawals

In almost all instances, decisions by participants were the reason for withdrawal (88/94, 94%; see Appendix 1, Table 36). The reason for withdrawal was very rarely discharge from NHS review (2/94, 2% of all withdrawals). One participant withdrew due to inadequate mobile phone network coverage and one died. The reason for withdrawal was missing for the other two.

The most common reason given by participants was personal reasons (50/88, 57% of participants’ decisions to withdraw and 53% of all withdrawals), followed by testing being too time-consuming (25 participants, 28%) or other reason (25 participants, 28%; Table 8).

Time to withdrawal overall is shown in Figure 9 (and by site in Appendix 2, Figure 23). Overall, 32% of participants withdrew.

FIGURE 9.

Kaplan–Meier graph of time to withdrawal.

Protocol deviations

Protocol deviations were few apart from monitoring visits without OCTs during the last months of follow-up, due to the COVID-19 pandemic. Other protocol deviations comprised:

1. patients who were recruited when the mVT app was not working. These patients trained to use the mVT at a later date;

2. retinal images that were not sent to CARF; percentages of monitoring visits for which images were expected are shown by site in Table 9;

3. patients’ self-reported data that were not collected before patients attended monitoring visits.

Adherence to testing

Frequencies of testing and test completion for the paper KSJ home-monitoring test are shown in Appendix 1 (see Table 37). It was intended that participants should report KSJ results once each week. Overall, participants completed the booklet fully for 87% of weeks, with < 1% of weeks with all KSJ data missing. The median time between adjacent completion dates was 7 days.

Frequency of testing and weeks when tests were completed for the electronic home-monitoring tests (mVT and MBT) are also shown in Appendix 1 (see Tables 38 and 39, respectively). The median testing frequencies for both tests for the three strata of time since starting treatment were almost all 3 or 4 per month. For study eyes, 56.2% and 59.3% of expected weekly tests were completed for MBT and mVT, respectively. There was evidence that testing frequencies/text completion differed by site: 33.7% (MBT) and 39.8% (mVT) of expected tests were performed by participants at one site compared to over 60% for both tests for all other study sites. Times to stopping testing with each home-monitoring test are shown in Figures 10, 11 and 12; tests were being completed for about 60% of study eyes by 18–24 months after starting to test.

FIGURE 10.

Time to stopping home monitoring with the mVT.

FIGURE 11.

Time to stopping home monitoring with the MBT.

FIGURE 12.

Time to stopping home monitoring with the KSJ.

Retinal images collected

The numbers of baseline and follow-up images submitted to and received by CARF are shown in Table 9.

Effect of coronavirus disease 2019 pandemic on the study

The study was not greatly affected by the COVID-19 pandemic. Recruitment was scheduled to end on 31 March 2020. A small number of patients were not recruited who might otherwise have been. However, even if they had been recruited, they might not have had a complete management visit due to the lockdown (so they could not have contributed to the primary analysis of Objective A).

The first lockdown had a more noticeable effect on the number of complete management visits. We estimate that 100–150 additional complete visits would have been documented if the lockdown had not come into effect (see Figure 8). Outgoing calls to participants to prompt testing or to find out why a participant was unable to test, and SMS text notifications, were suspended from March to June 2020 because the research team could not access the NHS-hosted MONARCH database when working from home.

Details of site-specific strategies for managing patients during the first 6 months of the pandemic were not collected (after which data collection stopped) and were not evident from the data collected (apart from the absence of an OCT). Anecdotally, strategies seemed to vary across sites. Some continued to provide face-to-face outpatient clinics; some reviewed patients virtually (or in person) but without performing an OCT to minimise patients’ time in the hospital; some treated patients (a quick procedure in an aseptic room) without assessing activity status, again to minimise potential exposure in hospital.

Sensitivity analysis 2 – primary outcome based on Central Administrative Research Facility grading

The analysis population was again the same as for the primary analysis, but with lesion activity at the management visit based on OCT images taken during the visits rather than ophthalmologists’ decisions. The same number of complete visits was available, but models including data for all study eyes could not be fitted due to the failure to converge. Instead of sampling one eye randomly (see Appendix 3), separate models were fitted for left and right study eyes; we decided to do this to make best use of the data collected. The numbers of visits included in models fitted for each index test varied because some participants had a right study eye, some a left study eye and some had two study eyes (Tables 12 and 13). Test summary scores for the whole preceding interval since the previous management visit were modelled.

The baseline VA strata of ‘Worse than or equal to 6/24’ was seen to predict CARF-graded lesion activity perfectly in left eyes, resulting in these observations (about 5%) being excluded from the model.

Results for the no-test model and for all home-monitoring test models are shown in Tables 12 and 13 for right and left eyes, respectively. In the baseline model, age was a significant predictor of activity for right eyes but not left eyes. No test summary score was seen to have a significant association with lesion activity. Days since baseline treatment was not significant for either model, contrary to the primary analysis.

Area under the receiver operating characteristics and model performance at thresholds identified by Youden’s index for each model are shown in Figure 15; further details of each model are shown in Appendix 1 (see Table 45). As in the primary analysis, the AUROCs from the models that included the home-monitoring tests were only marginally improved from the model that excluded the tests, with 95% CIs that overlapped across all models irrespective of eye. AUROCs and sensitivity were marginally better in models based on right eyes.

FIGURE 15.

Receiver operating characteristic curves and AUROCs (95% CIs) for summary test scores for all index tests; sensitivity analysis 2.

A significant association between the MBT summary score and CARF-graded lesion activity was seen in right eyes (OR 1.05, 95% CI 1.00 to 1.1; p: 0.038), but not in left eyes (OR 0.98, 95% CI 0.94 to 1.02; p = 0.366). For the mVT summary score, a significant association was seen in left eyes (OR 1.04, 95% CI 1.01 to 1.08; p: 0.015), but not in right eyes (OR 0.98, 95% CI 0.95 to 1.00; p: 0.097). An increase in mVT score of 0.01 was associated with a 1.04 increase in odds of lesion activity in the left eye (i.e. in the expected direction). No significant associations were seen for the KSJ home-monitoring test.

As in the primary analysis, the direction of the association observed for the MBT for right eyes is contrary to expectation; the average test score above the cut-off threshold was higher than below the threshold (see Appendix 1, Table 46). The direction of this association was inverted for right and left eyes. For the mVT, the direction of the association observed for left eyes is contrary to expectation; the average test score above the cut-off threshold was lower than below the threshold (see Appendix 1, Table 46). The direction of this association was also inverted for right and left eyes.

Secondary outcome – change in lesion activity

The ability of test summary scores to predict the secondary outcome, a change in lesion activity from inactive to active as determined by management decisions at pairs of management visits adjacent in time, was quantified. Lesion activity at baseline was used to define the first pair of management visits. Only complete visits, that is, visits on which an OCT image was taken, were included.

Models were constructed using the same approach taken in the primary analysis, except days since baseline was not natural log transformed. A model for the mVT summary scores did not converge. Therefore, there are no results for this test.

The analysis population was the same as for the primary analysis. There were similar numbers of pairs of visits as single visits due to using lesion activity classification at the baseline visit for the first pair in a participant’s follow-up. All available dates were included, that is, visit records did not depend on having complete data. The number of visits included in models fitted for each index test varied because participants did not always complete all tests at all visits (Table 14). Results from the models for this secondary outcome are shown in Table 14. No significant associations were seen for any home-monitoring test or other covariate.

TABLE 14 - Model results for the secondary outcome of change from inactive to active lesion in study eyes for all home-monitoring test models and for a no-test model

p < 0.05.

Transformed by multiplying by 100.

Proportion of scores across the respective interval that reported vision as being ‘worse’.

Natural log transformed.

Note

mVT test not shown because no model achieved convergence.

	No test		MBT		KSJ
Visit pairs	1413		1213		1238
Inactive to active transitions	131 (9.3%)		104 (8.6%)		120 (9.7%)
Participants	252		233		224
Predictor in model	OR (95% CI)	p-value	OR (95% CI)	p-value	OR (95% CI)	p-value
Mean MBT score			1.00 (0.99 to 1.02)	0.601
Median KSJ VA score					1.11 (0.89 to 1.38)	0.372
% KSJ VA worseb					1.21 (0.43 to 3.40)	0.715
% KSJ Amsler grid worseb					0.64 (0.15 to 2.71)	0.543
% KSJ household object worseb					1.82 (0.37 to 9.05)	0.464
Sex
Male	1
Female	1.05 (0.69 to 1.61)	0.815	1.18 (0.75 to 1.84)	0.475	1.00 (0.64 to 1.54)	0.989
Time since first treatment at baseline strata
6–17 months
18–29 months	0.79 (0.49 to 1.28)	0.349	0.76 (0.45 to 1.26)	0.284	0.81 (0.50 to 1.29)	0.37
30–41 months	0.95 (0.54 to 1.68)	0.860	0.96 (0.53 to 1.75)	0.903	0.91 (0.50 to 1.64)	0.747
Age	1.02 (0.99 to 1.05)	0.190	1.02 (0.99 to 1.06)	0.194	1.03 (0.996 to 1.06)	0.091
Baseline VA strata
Better than or equal to 6/18
Worse than 6/18 and better than 6/24	0.56 (0.21 to 1.52)	0.255	0.77 (0.28 to 2.17)	0.625	0.66 (0.24 to 1.85)	0.428
Worse than or equal to 6/24	0.93 (0.31 to 2.82)	0.904	0.80 (0.23 to 2.82)	0.727	1.06 (0.33 to 3.42)	0.928
Days since baseline*	1.001 (0.999 to 1.002)	0.236	1.001 (1.000 to 1.003)	0.112	1.001 (0.999 to 1.002)	0.452

Receiver operating characteristic curves and AUROCs for summary test scores for tests (but not for mVT) are shown in Figure 16; further details of each model, including model performance at thresholds identified by Youden’s index, are shown in Appendix 1 (see Table 47). AUROCs from the models that included the home-monitoring tests were only marginally improved compared to the no-test model, with 95% CIs that overlapped across all models. Sensitivities and specificities were broadly similar for the no-test model and the MBT model. The KSJ model showed decreased sensitivity and increased specificity compared to the no-test model. Average test scores above and below the predicted probability thresholds are shown in Appendix 1, Table 48. As in previous analyses, the average MBT score above the threshold (signifying lesion activity transition) was higher than below the threshold, contrary to expectations.

FIGURE 16.

Receiver operating characteristic curves and AUROCs (95% CIs) for summary test scores for index tests (except for the mVT) for a change from inactive to active lesion status.

Impacts of inequalities on willingness in principle to participate

During mapping to IMD, six patients were identified as having erroneous IMD ranks: four due to residing in the Isle of Man and two due to having out of range IMD ranks. These patients were excluded from analysis. Due to the small number of participants under 60 years of age (11 participants; no participation = 6, participation = 5) and over 89 years of age (32 participants; no participation = 30, participation = 2), participant age was split into three categories: under 70 years, 70 – 79 years and 80 years and older.

This analysis was performed on 936 patients (not willing, including 129 for whom willingness was pending = 645, willing = 291). Characteristics of these patients are described in Table 6. The results of the analysis are given in Table 15.

There was no significant effect of time since first treatment for nAMD on the odds of willingness to participate in the study. There was a 21% decrease in the odds of women participating in the study, although this was not significant (OR 0.79, 95% CI 0.58 to 1.09). An overall effect of IMD quintile was highly significant (χ² = 24.3, p < 0.001). Patients from the most deprived areas (IMD quintile 5) had a 47% decrease in odds of being willing compared to those from the middle quintile-deprived areas (OR 0.53, 95% CI 0.32 to 0.88); those in the second quintile had a 1.8-fold increase in odds of participation (OR 1.76, 95% CI 1.12 to 2.76).

Older patients were seen to have significantly decreased odds of being willing compared to patients < 70 years old (overall of the effect of age category, χ² = 50.5, p < 0.001). Patients between 70 and 79 years of age had 37% decreased odds (OR 0.63, 95% CI 0.41 to 0.97) and patients aged 80 years and over had 79% decreased odds (OR 0.21, 95% CI 0.13 to 0.35).

Decreased odds of participation were seen when potential study eyes had visual acuity worse than 6/18 at baseline compared to better than or equal to 6/18. However, this impact was not significant overall (χ² = 1.90, p = 0.387) or when either of the two strata with poorer vision were compared with the stratum of potential study eyes with vision better than or equal to 6/18.

Significant effects of study centre were seen on participation when compared with Belfast (overall of the effect of age category, χ² = 98.0, p < 0.001), most notably with Moorfields showing a 41% decrease in odds of patient willingness (OR, 0.59, 95% CI 0.35 to 0.97) and James Paget showing a 7.83-fold increase (OR 7.83, 95% CI 4.62 to 13.28).

Impact of inequalities on recruitment

Figure 6 shows that, although 312 patients were willing in principle, only one patient declined to take part when formal consent was sought. Of the other 14, 9 were found to be ineligible when attending for information and training and eligibility was pending for 4 (did not attend for information and training). Thus, 297/298 patients consented. No further analysis of the outcome consent to participate was undertaken.

Impact of inequalities on ability and adherence to weekly testing

Ability to perform an index test was defined as the proportion of qualifying management visits for which some valid index test data were available. Adherence was defined as the proportion of weeks in between qualifying management visits for which some valid data for an index test were available. The ability and adherence models were performed for each test separately at the level of the patient. Most participants used a smartphone, tablet, laptop/home computer, internet, e-mail or social media at least weekly [232/259 (89.6%)].

Frequency of testing and weeks when tests were completed are also shown in Appendix 1 (see Tables 37–39). Analyses of the impact of inequalities on ability to perform test and adherence to weekly testing are based on the population of 297 consented participants. The results of the analysis of participant ability to perform tests are shown in Table 16 and the results of the analysis of participant adherence to perform tests weekly in Table 17.

For ability to test, there were overall associations with VA stratum at baseline for the KSJ and MBT (χ² = 768 and 672, respectively, both p ≤ 0.001, although the associations were not in a consistent direction), and with centre for the mVT (χ² = 21.7, p < 0.001). Participants from James Paget showed a significant decrease in odds of ability when compared to participants from Belfast, with a 79% reduction in odds of ability for MBT (OR 0.21, 95% CI 0.08 to 0.54) and an 82% reduction in odds of ability for mVT (OR 0.18, 95% CI 0.07 to 0.47). There were no associations of ability to test with time since first treatment for nAMD, sex, age, IMD or exposure to technology on participant ability to perform home testing was seen for any test.

For adherence to testing, there were overall associations with VA stratum at baseline for the MBT and mVT (χ² = 20.9 and 11.3, p < 0.001 and 0.004, respectively, although the associations were not in a consistent direction), and with centre for the MBT and mVT (χ² = 21.8 and 17.8, respectively, both p < 0.001). Participants from James Paget showed a significant decrease in odds of adherence when compared to participants from Belfast, with a 71% reduction in odds of adherence for MBT (OR 0.29, 95% CI 0.16 to 0.52) and a 69% reduction in odds of adherence for mVT (OR 0.31, 95% CI 0.17 to 0.57). IMD had a significant impact on KSJ testing adherence (χ² = 12.5, p = 0.016), with participants in the most-deprived quintile showing 2.1 times increased odds of adherence compared to those from the middle quantile (OR 2.11, 95% CI 1.07 to 4.16). There were no associations of adherence to testing with time since first treatment for nAMD, sex, age or exposure to technology.

Analytic approach and methods

The test accuracy of the index tests was examined for two outcomes:

• Primary analysis outcome: the reference standard of an ophthalmologist’s classification of a fellow eye having active disease at a management visit, that is, conversion to active nAMD.

• Sensitivity analysis 1: the reference standard of the classification of a study eye at a complete management visit as having either active or inactive disease, using test data for the 4 weeks preceding the management visit.

• Sensitivity analysis 2: the alternative reference standard of classification of a fellow eye having active disease at a management visit, that is, conversion to nAMD, based on grading by CARF of OCT images taken during the management visits (Table 18).

A sensitivity analysis of the primary outcome that using test summary scores calculated for the 4-week period immediately prior to management visit was also carried out.

Out of 132 participants with a fellow eye and some home-monitoring test data, 17 (12.9%) were recorded as having nAMD during at least one management visit. This rate of conversion was higher than expected based on epidemiological studies of conversion rates in unaffected fellow eyes, potentially due to study eyes having had nAMD longer ago. No lesion, uncertain and inactive reference standard classifications were combined for all analyses; only active classifications were considered to represent conversion. (Each participant could only have one fellow eye because each participant had to have a study eye.) All available dates were included, that is, visit records did not depend on having complete data. However, participants did not have home-monitoring data for all tests. The numbers of participants with home-monitoring data and fellow eye conversions to nAMD disease for the three tests are shown in Table 18.

Separate models were generated for each test, summary test score and outcome and for a no-test model. Baseline VA strata could not be included because the better than or equal to 6/18 stratum predicted success perfectly and the worse than or equal to 6/24 stratum exhibited collinearity. Days since baseline was not included as a fixed component because its inclusion suggested model misspecification. For the KSJ models, only the household object and median near VA score could be included due to convergence issues or high SDs with the % Amsler grid worse and % VA worse scores. In addition, postestimation of the KSJ model suggested model misspecification that could not be controlled. All models were constructed with independent covariance, as other covariance structures resulted in failure of convergence. Each model fitted a random effect of participant and allowed random slopes for days since baseline nested within participant. As for Objective A, test summary scores for the mVT were multiplied by 100 before fitting the models, so that OR estimates correspond to changes in ORs for every 0.01 change in mVT score. No interactions were tested. For eyes that converted, follow-up data (reference classifications and test summary scores) were not included after conversion to active nAMD was identified.

Model performance in all cases was examined by assessing OR estimates along with their CIs and by examining model AUROCs. AUROCs were calculated conditional on the random effects and on the fixed effects only.

Primary analysis

Table 19 shows the numbers of participants and visits with data for models fitted for each index test.

Odds ratios estimates for index test summary scores and AUROCs for each model are also shown in Table 19. Neither participants’ sex nor time since first treatment at baseline was associated with conversion of fellow eyes to nAMD disease in any model. Increasing age at baseline was associated with conversion, significantly so in the model that included the KSJ summary test scores (OR 1.18 per year of age, 95% CI 1.02 to 1.36; p = 0.023) and with borderline significance in the MBT and no-test models.

Only the KSJ home object summary score was significantly associated with conversion; patients who reported more often that their vision was ‘worse than baseline’ were more likely to have a fellow eye conversion (OR 15.33 for a household object percentage worse summary score of 100 vs. 0, 95% CI 1.19 to 196.83; p = 0.036). However, this estimate should be interpreted with caution, due to the imprecise CI and potential misspecification of the model.

AUROCs and model performance at thresholds identified by Youden’s index for each model are shown in Figure 17, with further details in Appendix 1 (see Table 49). The AUROCs from the models that included the MBT and mVT home-monitoring tests were only marginally higher than for the no-test model, while the AUROC from the KSJ model was moderately higher. All AUROCs had wide 95% CIs that overlapped across all models. Sensitivity and specificity for all models were reasonable (≥ 0.69).

FIGURE 17.

Receiver operating characteristic curves and AUROCs (95% CIs) for summary test scores for index tests for conversion of a fellow to an active lesion; primary analysis.

Average test scores above and below the predicted probability thresholds are shown in Appendix 1 (see Table 50). Mean MBT test scores were lower above the identified threshold (prediction of fellow eye conversion) than below, with non-overlapping CIs. Mean mVT scores were higher above the identified threshold (prediction of fellow eye conversion) than below. In contrast to several previous MBT and mVT models, these differences in means above and below test thresholds were in the expected direction associated with predictions fellow eye conversion, again with non-overlapping CIs. The percentage KSJ home object summary score above the threshold as 12% (home object test ‘worse than baseline’) compared to 1% below the threshold. The median self-scored VA assessment was very similar above and below the threshold.

Sensitivity analysis 1 (data for preceding 4 weeks only)

Table 20 shows the numbers of participants and visits with data for models fitted for each index test. ORs estimates for index test summary scores and AUROCs for each model are also shown in Table 20.

As in the primary analysis, baseline VA strata and days since baseline treatment could not be included in the models. The no-test model was the same. For the KSJ models, the Amsler grid percentage worse summary score was not included due to a very high SD and model misspecification that could not be controlled. All models were constructed with independent covariance, as models with other covariance structures did not converge.

Neither sex nor time since first treatment at baseline was associated with conversion of fellow eyes to nAMD, in any model. Increasing age at baseline was associated with conversion, again significantly so in the model that included the KSJ summary test scores (OR 1.21 per year of age, 95% CI 1.04 to 1.40; p = 0.014) and with borderline significance in the no-test model (OR 1.13 per year of age, 95% CI 1.00 to 1.28; p = 0.050).

As in the primary analysis, only the KSJ home object test summary score was significantly associated with conversion (OR 15.33 for the percentage worse summary score of 100 vs. 0, 95% CI 1.19 to 196.83; p = 0.036). This estimate should be interpreted with caution, due to the imprecise CI and potential misspecification issues.

AUROCs and model performance at thresholds identified by Youden’s index for each model are shown in Table 20 and Figure 18, with further details in Appendix 1 (see Table 51). The AUROCs from the models that included the home-monitoring tests were similar to those in the primary analysis and, again, only marginally improved from the model that excluded the tests, with wide 95% CIs that overlapped across all models.

FIGURE 18.

Receiver operating characteristic curves and AUROCs (95% CIs) for summary test scores for index tests for conversion of a fellow to an active lesion; sensitivity analysis 1.

Average test scores above and below the predicted probability thresholds are shown in Appendix 1 (see Table 52). The pattern was the same as in the primary analysis.

Sensitivity analysis 2 (lesion activity classified by Central Administrative Research Facility grading)

Table 21 shows the numbers of participants and visits with data for models fitted for each index test. ORs estimates for index test summary scores and AUROCs for each model are also shown in Table 21.

Model fitting was constrained as for the primary analysis and first sensitivity analysis, except that days since baseline could be included. Predictors that were not to do with the tests showed similar associations; the association of age with conversion was of borderline significance. In this analysis, the mVT and KSJ median VA test summary scores were significantly associated with conversion (respectively, OR 1.06 per 0.01 logMAR unit, 95% CI 1.01 to 1.12; p = 0.012 and OR 0.19 per one VA step increase, 95% CI 0.04 to 0.97; p = 0.046). The latter estimate was unexpected, as higher KSJ VA scores relate to better VA.

Area under the receiver operating characteristics and model performance at thresholds identified by Youden’s index for each model are shown in Figure 19, with further details in Appendix 1, Table 51. The AUROCs from the models that included the home-monitoring tests were worse than those in the primary analysis, including the no-test model. However, as in other analyses, AUROCs for models including summary test scores were not markedly larger than for the no-test model, with wide 95% CIs that overlapped across all models.

FIGURE 19.

Receiver operating characteristic curves and AUROCs (95% CIs) for summary test scores for index tests for conversion of a fellow to an active lesion; sensitivity analysis 2.

Average test scores above and below the predicted probability thresholds are shown in Appendix 1 (see Table 52). The pattern was the same as in the primary analysis and the first sensitivity analysis.

Participant helpline

Issues affecting the availability of the electronic tests

Frequencies of testing and test completion for electronic tests described in Adherence to testing include occasions when participants were unable to complete the electronic tests. Details of downtime for each test are shown in Table 26. These occasions accrued to 15 days of testing for the MBT test and 30 days for the mVT test. To put these numbers in context, total testing time in the study spanned 1318 days, although the number of participants testing varied over the course of the study.

On 25 October 2018, the company which created the mVT app was acquired by the Roche Holding AG (Basel, Switzerland). Upon acquisition, the mVT servers were switched off as they prepared to move to the Roche severs. Consequently, the mVT app could not be activated on new iPods and existing iPods with the app activated were not compatible with the latest version of iOS installed on the iPods. This issue meant that prospective participants attending an information and training session were unable to have the mVT app activated on their iPods. In discussion with the SSC, it was decided that recruitment should continue and affected patients should have their mVT app activated at the earliest opportunity when the new servers were switched on. This issue coincided with an iOS update from v11.0 to v12.0. The mVT app was incompatible with v12.0 during the server switchover period, preventing the mVT app from working on any iPod that was updated to v12.0 at that time.

The mVT team assured the research team that both issues would resolve when the server transfer was complete. They had been unable to provide advance notice due to the confidentiality of the acquisition and had not anticipated that there would be any impact on the study. The server switch over was completed on 16 November 2018 when the mVT app was again available. iPods were updated to v12.0, a process that was quite time consuming for both sites and the post-doctoral fellow at Queen’s University Belfast (QUB) and multiple device management software (MDMS) was set to install future updates automatically. The study team updated the equipment instructions for patients with what to do if their iPod needed to install an update. For patients who joined the study without an activated mVT app, the study team phoned the patients to talk them through activating the app remotely or the local site team arranged to see the patients when they next attended clinic.

Around this time, sites and participants reported a variety of technical issues such as error messages, inability of the iPod to connect to the internet, the MBT app failing to work on two iPods following an update, iPods of a certain batch being unable to update to v12.0 and three iPods with an mVT app error preventing activation of the app. The team’s ability to guide a participant through problem solving remotely over the telephone varied substantially by patient.

On 15 April 2019, a participant reported that the two-port USB charger had exploded in a power socket while charging the devices. The charger casing had split into two pieces and the electricity to the participant’s house was tripped. The participant was unharmed and there was no damage to the house or the iPod and MiFi devices. Subsequently, all participants were posted a letter with new instructions on charging the study devices, advising that devices are charged for a maximum of 3 hours per week while at home (i.e. not to leave the charging devices unattended) and to unplug the charger once charging is complete. The Sponsor, site research teams and the study management group were also informed of the incident and kept informed about actions taken. The letter was subsequently submitted as a non-substantial amendment and approved.

Feedback from the charger supplier reported that such an incident had not been reported previously. They specified that the chargers were fitted with a surge protector which was designed to disable the plug to preserve the devices in the event of a power surge. The Sponsor arranged for all chargers (approximately 200) that had not yet been dispatched to sites to have a portable appliance test; all these chargers passed testing except for one charger which split similarly during testing. The research team was subsequently made aware that seven other chargers being used in the study had broken apart: one at a participant’s home, one at a site during equipment set up and five when returned to testing and being unpackaged by the Sponsor. The Sponsor decided that all participants should have a tested charger. Tested chargers were posted to all sites to replace their local stock and chargers belonging to their participants. The chargers were either swapped at clinic visits or posted to participants.

For both applications, a security certificate was required for the servers that host the online portals. The certificates expired, causing both test portals to become temporarily unavailable. Renewal of the certificate was the responsibility of the app providers. For the MBT app, the portal was down for less than 1 day (9 August 2019) and the issue was resolved by 3 p.m. The helpline received three phone calls about this issue. For the mVT app, the portal was down for 1 week (from 6 August 2019 to 13 August 2019). The delay was due to administrative processes within Roche. During this time, participants could test as normal, but new devices could not be activated. This issue effected four participant training sessions, causing sessions to be rearranged, participants being sent home without equipment or without the mVT app activated. In addition, patient test data were not received by the study team during this week but were stored locally on the device and became available after the issue was resolved.

Test data for the MBT app were unavailable to be downloaded for 11 days in October/November 2019 due to the portal server being at full capacity. The server capacity was increased, causing the cost of hosting the portal to rise.

Participants could not test using the MBT app and the online portal could not be accessed from 14 August 2020 to 28 August 2020. The issue was caused by the expiry of a domain certificate required for the host database. The delay in resolving the issue arose because Novartis had to approve the update, leading to over 50 participants contacting the study team during the affected period.

Logistical issues with the electronic equipment provided for the study

As we supplied a limited data contract with the MiFi device, we were keen to limit the use of the iPod to study testing only; therefore, we decided to the use a MDMS which meant we retained control of all devices during the study. We chose to use the free Apple product as commercial custom-designed versions appeared expensive in comparison; however, on hindsight, the limited functionality of the basic version was a disadvantage, and a commercial version would have been better. In particular, we were unable to stop Apple updates being pushed to devices which caused a lot of confusion for technology-naïve participants and calls to the helpline. The MDMS also meant all the iPods had to be loaded with the apps in Belfast while connected to the QUB network and then transported to the sites. This was problematic as there are strict regulations governing the transport of the lithium-ion batteries by air and ferry with only specific carriers able to accept them and only two batteries could be present per package. We had budgeted to send the equipment packs to sites in batches of 10 by a standard carrier, whereas the subsequent regulations meant we had to send individual equipment packs (both iPod and MiFi device contained a battery) at a cost of £6.27 each with a specialist carrier, significantly increasing the costs and time required to administer the process.

Mobile phone connectivity using the MiFi device was required for app data to be transmitted. Mobile phone coverage was checked at the outset, but connectivity was still a problem for some participants some of the time, either due to variability in the strength of mobile phone coverage or difficulty in using the MiFi device. We also had considerable problems at one of our sites which had no network coverage within the hospital preventing set up occurring at the training visit. This required negotiation with the hospital IT department to provide special permission to access a sufficiently secure Wi-Fi network. In September 2019, the cost of a MiFi device increased from £33 to £63. The cost of the Vodafone tariff also rose on 16 September 2019 from £6 per month to £9 per month for each activated MiFi device. The study was able to bear these cost increases only because the total number of participants recruited (and hence packs of equipment in operation) was smaller than expected.

Theme 1: the role of home monitoring

Theme 2: suitability of procedures and instruments

The iPod Touch and app-based index tests were viewed as novel innovations in eye care while recognising that they reflected the increasing pervasiveness of technology. Participants’ pre-HM use of such devices varied, and experience with more recent devices, including smartphones, was relatively limited. Despite this experience, even those with minimal or no previous exposure to similar technologies reported that they viewed HM as realistic and suitable. Overcoming unfamiliarity with technology or any hesitancy with using it was regarded as ‘just something that needed to be done’ as part of their eye care.

Participants suggested demonstration of the device and index tests, prior to, and separate from the formal ‘training’ in HM procedures, provided reassurance that it was ‘simple enough to do’, and increased belief in their capacity to undertake HM. In addition to this technological apprehension, participants reflected that being older and unfamiliar with technology might also result in hesitation about engaging in HM and generate concern that HM was too complex or a ‘burden’.

Theme 3: experience of home-monitoring procedures

Theme 4: feasibility of regular home monitoring in usual service delivery

Theme 5: impediments to home monitoring

Views of informal ‘carers’ and healthcare professionals – summary

In general, there was a high level of concurrence between the perspectives of informal ‘carers’ and HCPs and views of patients (see Table 28). HCPs suggested that increasing age may be linked to initial reluctance or hesitancy about HM while acknowledging that this view may also be related to their underlying assumptions about older people and use of technology. HCPs agreed with the patients’ views that training was essential to the successful uptake and use of HM, and that training needed to be adapted to the needs of each individual patient and their previous experience of technology. There was a concern that HM could be resource intensive in terms of the technical support needed (see Appendix 1, Table 28 HCP quotes). Informal ‘carers’ or family members saw their role in HM as a supportive and facilitative one, and they valued the feedback from a test performance as a way of assessing the response to the nAMD treatment that was received by their family member (see Appendix 1, Table 29 carers quotes).

Main findings: study conduct

To our knowledge, this is the first multisite deployment of home-based visual function monitoring in nAMD patients in the UK adhering to best practice for diagnostic test accuracy studies. All index tests had supporting peer-reviewed evidence of their potential to identify active nAMD lesions in similar populations when the study was conceived. Both app-based tests had formal spin-out companies and at the time of trial initiation, both had been acquired by pharma companies for further application and development. Hence, they were in a relatively well-developed stage when we began the trial. Despite this, we experienced a wide range of practical and logistical challenges in the delivery of the trial:

1. Device-based challenges

   1. During the initial recruitment period, the company that created the mVT app was acquired by a pharma company, upon acquisition, the mVT servers were switched off and for a period the mVT app or servers were not available for new participants nor were compatible with the iOS update which happened around the same time.

   2. Regulations limiting the transport of lithium-ion batteries occurred after the grant submission. Rather than sending a bulk pack of 10 sets of equipment to sites, we had to send them individually. Packaging and arranging postage proved very time consuming for study staff.

   3. Malfunctioning of chargers – iPod Touch devices do not come with their own charger unlike iPhones, and these needed to be purchased and provided separately. Since both the MiFi and iPod Touch needed charging, we opted for a double USB charger rather than an Apple version. Despite checking all the appropriate certifications were in place, we had a report of a device overheating and exploding. This required a temporary stop of the trial while all chargers were replaced, and Portable Appliance Testing (PAT) tested.

   4. iOS management – we used the standard Apple MDMS for the study devices. With hindsight, a more expensive custom version tailored to the trial would have been much better. We were unable to block iOS updates going to the devices and for technology-naïve participants this caused a lot of confusion and concern and calls to the helpline. One of the updates made the apps stop working and so there was a delay while they were updated appropriately. We promised participants they could keep the devices if they completed the required follow-up as an incentive, but this proved very problematic as the MDMS meant they had to be returned to Belfast and connected to QUB network for files to be deleted, and this took considerable time when staff were working from home. For as many participants as possible, we sent new devices which were spare given the sample size was smaller than expected.

2. Different approaches at sites to patient recruitment/inclusion.

   There was some variation in how different sites managed patients in terms of frequency of monitoring visits and treatment strategies that may have contributed to the variation in recruitment across sites. During study initiation visits, we also encountered several research staff who were very sceptical about providing older nAMD patients with electronic devices and we felt this may have contributed to how they approached potential participants in clinics.

3. Recruitment timescale.

   It quickly became apparent that sites had to wait for patients to attend routine visits before approaching them, and if they were willing in principle to take part, wait until the next routine visit to attend an information and training session. Such a protracted recruitment pathway was not factored into the recruitment prediction. Therefore, recruitment was slower than expected. We also ended up with fewer participants in the two strata of longer time since diagnosis because, once the study was underway, patients were recruited when they became eligible. Sites were able to recruit patients eligible for the 30- to 41-month stratum mainly early on during recruitment; recruitment became increasingly difficult as the study progressed as eligible patients did reach this length of time from diagnosis.

1. 1 Index tests generated many scores between management visits. These needed to be summarised to fit models because models could not fit raw data, as well as respecting the structure of eyes and management visits nested within participants. Ultimately, we calculated very simple summary scores (averaging multiple scores) but considered several issues, for example, controlling for the time between management visits and number of scores being averaged, weighting proximal scores (closer in time to the reference standard visit) more highly than those that were more distal/older (addressed in the first set of sensitivity analyses) alternatives to a simple average such as mode.

2. 5. Feedback to participants on performance. The MBT provided a numerical score on performance, whereas mVT did not; this was unable to be altered at study onset and as a study team, we would have preferred that neither provided a score to prevent concern or anxiety for the participants. However, some participants really liked getting feedback from MBT and it seemed to motivate adherence and retention, whereas for others it caused concern.

3. 6. Pandemic: Recruitment was due to end on 31 March 2020. Therefore, the COVID-19 pandemic only impacted the final 2–3 weeks of recruitment. The frequency of the monitoring visits was significantly impacted with different sites adopting different strategies to managing care pathways, including one site which stopped retinal imaging altogether.

Main findings: primary outcome (Objective A)

Main findings: Objective C

Main findings: Objective D

Main findings: Objective B

Main findings

Equality, diversity and inclusion

Patient and public involvement

Age-related macular degeneration patients who were members of our PPI group provided useful feedback regarding the KSJ, the style, format and content of the PIL and study newsletter. PPI feedback was most useful for the KSJ as the US style motivational quotes were strongly rejected by the PPI group. Feedback on the design and operation of the electronic tests was not very useful as the PPI participants had minimal experience of smartphones or computer access and so seemed unwilling to criticise their content. Future studies evaluating mobile devices should ensure PPI groups include participants with a range of exposure and familiarity with the potential technology. The strong rejection of the term ‘carer’ by the PPI group members was also an important criticism which we acted upon and should be considered for future studies.

Strengths and limitations

Lessons for the future

It is clear from the outcome of this study that independent validation of such tests is essential. Robust pilot data about the DTA of potential tests are essential before designing a full-scale diagnostic test accuracy study. Most pilot studies describe associations rather than predictions, with analyses that have not been directed by a prespecified data analysis plan. Therefore, there is a high risk of selection of the reported result from multiple analyses,43 for example, reflecting post hoc manipulations of the data with respect to outliers, uncertain or missing reference classifications or other missing data. In addition, studies are typically designed to optimise test performance or in populations or settings that bias test performance. The topic under which this study was commissioned was advertised twice. The commissioning process was abandoned on the first attempt when the manufacturers of a test of interest declined to collaborate. The topic was readvertised despite this experience and this study was commissioned on the basis of the peer-reviewed evidence cited in our application. In our view, the development of the tests and the evidence to support them was below the usual level required by the Health Technology Assessment (HTA) programme for a commissioned topic.

Given the additional complexity that provision of the iPod device and MiFi added to the execution of the study, it is likely that future studies will concentrate on providing access to an app that can be loaded direct to patients own smartphone. However, some patients may find that intrusive or unacceptable and those without devices will face exclusion.

It is clear from the outcome of this study that independent validation of such tests is essential. However, evaluations need to be agile to react quicker to poor performance or add/swap included tests as technologies emerge. Adaptive study designs may be considered in future,44 in conjunction with an ongoing review of literature (akin to a Cochrane living review) to identify promising tests and efficiently evaluate them.

Future research

Given the clear potential for digital exclusion in these types of studies, further research should concentrate on interventions to improve participation in mobile health research beyond providing access to hardware.

Further research is also warranted to identify pros and cons of different methods to summary test score when the results of multiple tests are likely to contribute to DTA for a subsequent reference standard. Considerations include: the best summary measure, how time over which monitoring has occurred/number of tests should be considered, whether test scores to application of the reference standard should be weighted more strongly than test scores that are more remote in time.

Further research should also focus on the nature of personal feedback to home-monitoring tasks and how it can be used to help or hinder adherence and retention. It was clear that different patients reacted differently to the feedback on test performance with some really liking it and others finding it stressful.

Given the superiority of retinal imaging biomarkers over functional testing in the Early Detection of age-related macular degeneration (EDNA) study, further research focusing on low-cost portable home OCT devices would be worthwhile. Several devices are in development,45,46 but none have been deployed in a large-scale study yet.

The longitudinal imaging data collected in MONARCH can also be further analysed later. Of particular interest is whether central macular thickness on the visit prior to an active clinical decision is predictive of lesion reactivation.

Contributions of authors

Ruth E Hogg (https://orcid.org/0000-0001-9413-2669) (Senior Lecturer, Co-chief Investigator) co-led the conception and design of the study, contributed to the interpretation of the results, led the writing of the introduction and discussion and contributed to the writing and editing of the report.

Robin Wickens (https://orcid.org/0000-0003-4374-3747) (Trial management, Bristol CTU) was responsible for the day-to-day management of the study, contributed to the interpretation of the results, led the writing of the methods and challenges section and contributed to the writing/editing of the report.

Sean O’Connor (https://orcid.org/0000-0001-6805-8899) (Post-doctoral fellow, QUB) was responsible for the day-to-day management of the qualitative component, managed the device logistics and MDMS, contributed to the interpretation of the qualitative results, led the writing of the qualitative chapter and contributed to the writing/editing of the report.

Eleanor Gidman (https://orcid.org/0000-0002-5261-5213) (Statistician, Bristol CTU) conducted the statistical analysis study, contributed to the interpretation of results, led the statistical methods section and contributed to the writing/editing of the report.

Elizabeth Ward (Study management, Bristol CTU) was responsible for the day-to-day management of the study, contributed to the interpretation of the results, led the writing of the methods and challenges section and contributed to the writing/editing of the report.

Charlene Treanor (https://orcid.org/0000-0002-3209-8060) (Post-doctoral fellow, QUB) was responsible for the day-to-day management of the qualitative component, managed the device logistics and MDMS and contributed to the writing/editing of the report.

Tunde Peto (https://orcid.org/0000-0001-6265-0381) (Consultant Ophthalmology) contributed to the conception and design of the study, recruitment and follow-up of participants, contributed to the interpretation of the results and contributed to the writing/editing of the report.

Ben Burton (https://orcid.org/0000-0001-9579-9078) (Consultant Ophthalmology) contributed to the conception and design of the study, recruitment and follow-up of participants, contributed to the interpretation of the results and contributed to the writing/editing of the report.

Paul Knox (https://orcid.org/0000-0002-2578-7335) (Reader, Vision Scientist) contributed to the conception and design of the study, recruitment and follow-up of participants, contributed to the interpretation of the results and contributed to the writing/editing of the report.

Andrew J Lotery (https://orcid.org/0000-0001-5541-4305) (Consultant Ophthalmology) contributed to the conception and design of the study, recruitment and follow-up of participants, contributed to the interpretation of the results and contributed to the writing/editing of the report.

Sobha Sivaprasad (https://orcid.org/0000-0001-8952-0659) (Consultant Ophthalmology) contributed to the conception and design of the study, recruitment and follow-up of participants, contributed to the interpretation of the results and contributed to the writing/editing of the report.

Michael Donnelly (Professor, Health Services research) contributed to the conception and design of the qualitative study, management of the qualitative study, contributed to the interpretation of the results and contributed to the writing/editing of the report.

Chris A Rogers (https://orcid.org/0000-0002-9624-2615) (Professor, Statistics) contributed to the conception and design of the study, was responsible for the statistical analyses and statistical analysis plan, contributed to the interpretation of results and contributed to the writing/editing of the report.

Barnaby C Reeves (https://orcid.org/0000-0002-5101-9487) (Professor, Co-chief Investigator) co-led the conception and design of the study, contributed to the interpretation of the results, led the writing of the final report and contributed to the writing and editing of the report.

Acknowledgements

Data-sharing statement

All available data can be obtained by contacting the corresponding author.

Ethics statement

Main REC: Northern Ireland Health and Social Care Research Ethics Committee A

Reference number: 17/NI/0235

Date of favourable ethical approval: 29 January 2018

Information governance statement

Queen’s University Belfast is committed to handling all personal information in line with the UK Data Protection Act (2018) and the General Data Protection Regulation (EU GDPR) 2016/679. Under the Data Protection legislation, Queen’s University Belfast is the Data Controller, and you can find out more about how we handle personal data, including how to exercise your individual rights and the contact details for our Data Protection Officer here: https://www.qub.ac.uk/privacynotice/Research/ListofResearchPrivacyNotices/PrivacyNoticeforResearchParticipants.html.

Disclosure of interests

Full disclosure of interests: Completed ICMJE forms for all authors, including all related interests, are available in the toolkit on the NIHR Journals Library report publication page at https://doi.org/10.3310/CYRA9912.

Primary conflicts of interest: Barnaby C Reeves reports former membership of: the Health Technology Assessment Commissioning Board (from January 2010 to 31 March 2016); the Health Technology Assessment Efficient Study Designs Board (October to December 2014); the Health Technology Assessment Interventional Procedures Committee B Methods Group (December 2016 to September 2022); Systematic Reviews Programme Advisory Group (Systematic Reviews National Institute for Health and Care Research Cochrane Incentive Awards and Systematic Review Advisory Group), dates unknown. He has current membership of a CTU funded by the NIHR up to 31 August 2021. He has no other competing interests.

Chris A Rogers reports membership of a Clinical Trials Unit funded by the National Institute for Health and Care Research. She also reports membership of the Health Technology Assessment Funding Committee Policy Group (formally CSG) and the Health Technology Assessment Commissioning Committee. She has no other competing interests. Ruth E Hogg reports attendance at Roche Digital Health Advisory Meeting July 2019. She also received partial PhD Studentship funding from Okko Health 2021 for home monitoring of Diabetic Retinopathy. Paul Knox reports software support from Vital Art and Science Inc who produced the My Vision Track App. Andrew J Lotery reports receiving consulting fees from and owning stock or stock options of Gyroscope Therapeutics. Sobha Sivaprasad reports grants from Boehringer Ingleheim, receiving consulting fees from Boehringer Ingleheim, Novartis, Apellis, Bayer, Oculis, Oxurion, Roche and Biogen. She also received payment or honoraria from Boehringer Ingleheim and Bayer, support for attending meetings from Bayer and participation in an advisory board with Bayer. She is also a Macular Society Trustee (unpaid). She was also a former committee member of HTA Commissioning Committee from 7 August 2017 to 30 September 2021.

Publications

Ward E, Wickens RA, O’Connell A, Culliford LA, Rogers CA, Gidman EA, et al. Monitoring for neovascular age-related macular degeneration (AMD) reactivation at home: the MONARCH study Eye (Lond) 2021;35(2):592–600. https://doi.org/10.1038/s41433-020-0910-4

O’Connor SR, Treanor C, Ward E, Wickens RA, O’Connell A, Culliford LA, et al., Monarch Study Group. The COVID-19 pandemic and ophthalmic care: a qualitative study of patients with Neovascular Age-Related Macular Degeneration (nAMD). Int J Environ Res Public Health 2022;19(15):9488. https://doi.org/10.3390/ijerph19159488

O’Connor SR, Treanor C, Ward E, Wickens RA, O’Connell A, Culliford LA, et al. , Monarch Study Group. Patient acceptability of home monitoring for neovascular age-related macular degeneration reactivation: a qualitative study. Int J Environ Res Public Health 2022;19(20):13714. https://doi.org/10.3390/ijerph192013714. PMID: 36294292.

Papers in progress

1. Diagnostic test accuracy of home-monitoring tests.

2. Uptake of digital health interventions, and inequalities in uptake, in an elderly population living with sigh impairment.

3. Challenges in the evaluation of digital health interventions requiring participation in an elderly (visually disabled) population.

4. Association of morphological features of AMD lesions with lesion status classifications by ophthalmologists monitoring patients for reactivation.

Disclaimers

This article presents independent research funded by the National Institute for Health and Care Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, the HTA programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, the HTA programme or the Department of Health and Social Care.