Automated tests for diagnosing and monitoring cognitive impairment: a diagnostic accuracy review

Article history

The research reported in this issue of the journal was funded by the HTA programme as project number 15/67/01. The contractual start date was in August 2015. The draft report began editorial review in March 2016 and was accepted for publication in August 2016. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

Rumona Dickson is on the Health Technology Assessment Evidence Synthesis Board.

Copyright statement

© Queen’s Printer and Controller of HMSO 2016. This work was produced by Aslam et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

Population

This review addresses patients in two specific diagnostic categories: those with MCI and those suffering from early dementia.

Epidemiology

Obtaining accurate incidence and prevalence figures for MCI is difficult, as people with memory decline may go undiagnosed. Prevalence and incidence estimates can also vary significantly depending on the definitions that are used. The variance in these estimates then poses a challenge to the understanding of the social burden of this disease. For example, the authors of a study utilising data from the Medical Research Council Cognitive Function and Ageing Study of people aged ≥ 65 years29 estimated the prevalence of MCI to range from 2.5% to 41.0% in the UK. In addition, the rates of progression from MCI to dementia varied from 3.7% to 30.0%. 29

The most common form of dementia in the UK is AD. 34 There are an estimated 163,000 new cases of dementia identified each year in England and Wales. The risk of being diagnosed with dementia rises with increasing age; however, a significant portion of people who are diagnosed with dementia are younger than 65 years. 35 The incidence of dementia ranges from 6.7 per 1000 person-years at age 65–69 years to 68.5 per 1000 person-years at age ≥ 85 years. Prevalence increases with age; it is estimated to be 3% by 70 years and then to double every 5.1 years thereafter. 36 A report published by the Alzheimer’s Society predicts that there will be 1 million people living with dementia in the UK by 2025. 35

Current diagnostic practice

The 2006 guidelines from the National Institute for Health and Care Excellence (NICE)8 place emphasis on the early diagnosis of people with dementia to allow for effective management and planning with patients and carers. The projected increase35 in the prevalence of dementia by the Alzheimer’s Society highlights the importance of equitable and easy access to diagnosis in the UK patient population.

A schema of the pathway for assessing cognitive impairment in the UK is presented in Figure 1. The first point of contact with health-care services for a person with suspected cognitive impairment is with a health-care professional in primary care or in an acute hospital setting. In primary care, the general practitioner (GP) usually takes a brief history, conducts a physical examination and conducts a short test of cognitive function to establish a differential diagnosis for cognitive impairment. The NICE guidelines8 recommend the use of the MMSE when aiming to diagnose people with dementia. It is possible to offer a diagnosis of dementia at this point if it is in an established state. 37 However, MMSE is insensitive to early-stage dementia38 and does not effectively map the transition from MCI to early dementia. 39 The NICE guidelines8 also recommend a number of pen-and-paper-based tools as suitable tests for screening people for cognitive impairment. Some of these tests, along others with others used in clinical practice, are further outlined in Table 3.

FIGURE 1.

Adapted pathway for assessing cognitive impairment.

TABLE 3 - Screening tests for cognitive impairment

6CIT, Six-Item Cognitive Impairment Test; AMT, Abbreviated Mental Test; GPCOG, General Practitioner Assessment of Cognition.

Source: NICE guidelines8 and clinical advice.

Test	Administration time (minutes)	Sensitivity (%)	Specificity (%)
GPCOG8	5	82–8540	82–8540
6CIT8	3–4	78.5–8341	77–10041
Mini-Cog assessment instrument42	2–4	76–9940	89–9640
AMT8	2–4	Not validated in a primary care setting	Not validated in a primary care setting
Memory Impairment Screen43	4	74–8640	96–9740

After this initial screening, the GP refers patients with suspected MCI or early dementia to a memory assessment service, which is usually based in secondary care and can involve the examination of older adults by community mental health teams. Memory assessment service teams play an important role in clarifying the diagnosis (MCI or dementia, subtype and severity of dementia), identifying which patients with MCI are at greatest risk of developing dementia and determining who are most in need of follow-up. Diagnoses are clarified by taking a detailed clinical history from the patient and a family member or carer, interpreting scan results (if needed) and interpreting the findings from cognitive function pen-and-paper diagnostic tests. 44

In an acute hospital setting, patients can be ‘incidentally’ discovered to be living with cognitive impairment through routine testing for another medical condition, and/or patients may present with acute confusion secondary to a medical problem. 44 There are many different pen-and-paper tests used to aid diagnosis for MCI and early dementia available for use in a secondary setting; three of the tests most commonly used in the NHS are described in Table 4.

TABLE 4 - Tests used to aid diagnosis of MCI and early dementia

Test	Administration time (minutes)	Sensitivity (%)	Specificity (%)
DemTect45,46	8–10	9245	8645
The Montreal Cognitive Assessment: A Brief Screening Tool For Mild Cognitive Impairment46	10	9047	8747
Saint Louis University Mental Status48	7	98–10049	98–10049

Reference standard

A reference standard can be described as being the best available method for identifying patients who have the target condition. 50 The reference standard for this research is the clinical diagnosis of MCI and early dementia. It is recognised that clinical diagnosis itself has a degree of variability, but this is not unique to dementia studies and does not invalidate the basic diagnostic test accuracy approach. Any recognised diagnostic criteria (e.g. International Classification of Diseases, Tenth Edition;2 Diagnostic and Statistical Manual of Mental Disorders-Fourth Edition51 or -Fifth Edition52) can be used. 53 Dementia diagnosis may specify a pathological subtype. Clinicians may use imaging, biomarkers or other data to aid diagnosis, for example the Clinical Dementia Rating54 which is a gold-standard research criterion against which most rating scales have been compared. 55 However, in this report, diagnoses based only on these tests, without corresponding clinical assessment, were not included. It is recognised that different iterations of diagnostic criteria may not be directly comparable and that diagnosis may vary with the degree or manner in which the criteria have been operationalised (e.g. individual clinician vs. algorithm vs. consensus determination).

Index test

For the purpose of this report, the index test is any automated computerised assessment of cognitive impairment, which can either be self-administered or interviewer administered. In self-administered tests, patients may require help with accessing the necessary computer programs, login identification and simple start-up explanation. It is important to ensure that the patient can see and/or hear the instructions and test stimuli.

There are several automated tests available to help identify patients with MCI and early dementia. An accurate automated cognitive assessment tool would be clinically valuable if it were shown to work as well as clinician-delivered tests. However, there is limited clinical evidence to demonstrate their equivalence or superiority over standard practice.

The authors of a UK-based review56 investigated the use of several available computerised automated tests and assessed their sensitivity and specificity for detection of MCI compared with two well-validated pen-and-paper tests: the Hopkins Verbal Learning Test and the MMSE. 56 The authors of the review concluded that the Hopkins Verbal Learning Test had better sensitivity for the detection of MCI in older adults than the computerised tests. They also identified that one automated test, CogState, enabled the identification of cognitive deficits beyond mild impairments in memory; for example, CogState detected more functional deficits than the Hopkins Verbal Learning Test or MMSE. 56 The authors, however, did not address if these tests had the potential to facilitate timely diagnosis or if they were effective in monitoring disease progression. Appendix 1 explains measures for assessing an index test against a reference standard.

Table 5 provides a list of automated computerised tests that were identified during the initial scoping search that was conducted to inform the development of the protocol for this review.

Search strategy

The following electronic databases were searched for the period from 2005 to the latest available version (August 2015).

• MEDLINE (via OvidSP)

  • MEDLINE In Process & Other Non-Indexed Citations (via OvidSP)

• EMBASE (via OvidSP)

• The Cochrane Library

  • Central Register of Controlled Trials

  • Cochrane Database of Systematic Reviews

  • Database of Abstracts of Reviews of Effects

  • Health Technology Assessment database

  • NHS Economic Evaluation Database

• ISI Web of Science

  • Proceedings (via Index to Scientific and Technical Proceedings)

  • ISI Web of Science – Science Citation Index Expanded

• PsycINFO.

The search terms for electronic databases comprised a combination of medical subject heading terms and free-text words. The search had no language restrictions and there were no limits on specific study design. Animal studies were excluded, as were case reports, comments, editorials and letters. The details of the search strategies and the number of references retrieved are provided in Appendix 2.

Study selection

The citations identified were assessed for inclusion through two stages using Covidence systematic review software (Veritas Health Innovation Ltd, Melbourne, VIC, Australia). 61 Two reviewers independently scanned all of the titles and abstracts and identified the potentially relevant articles to be retrieved. Full-text copies of the selected studies were subsequently obtained and assessed independently by two reviewers. Disagreements were resolved by consensus or in discussions with a third reviewer.

The eligibility criteria are listed in Table 6 for the diagnostic accuracy studies and in Table 7 for the studies monitoring disease progression.

Data extraction

Two reviewers developed, piloted and standardised two sets of data extraction forms. One form was designed to collect data on diagnostic accuracy and the other was designed to collect data related to monitoring disease progression. One reviewer extracted details of study design, participants, index and reference standard tests, outcome data and other relevant data, and a second reviewer checked the data extraction. Any discrepancies were resolved by consensus or in discussions with a third reviewer.

Assessment of methodological quality

The quality assessment of studies meeting the inclusion criteria was carried out by one reviewer and independently checked for accuracy by a second reviewer. The methodological quality of the included studies was assessed using the QUADAS-2 (Quality Assessment Tool for Diagnostic Accuracy Studies) tool. 62 This tool is designed to evaluate the risk of bias and applicability of primary diagnostic accuracy studies. The results of the quality assessment are presented in summary tables and as a narrative synthesis.

Outcomes

We recorded the following diagnostic accuracy outcome measures of automated computerised tests for MCI and early dementia:

1. sensitivity and specificity

2. area under the receiver operating characteristic curve (AUC).

Method of analysis/synthesis

Patient and public involvement

The review team was guided during the review by an Advisory Group comprising clinicians and service users. Building trusted contacts with service users led us to believe that the most effective way to obtain engagement with service users was through frontline agencies, for example the Alzheimer’s Society and Dementia UK. We sent out a call for participation through these frontline groups to identify people interested in giving feedback on the results of the review and on the final report. We took guidance from these agencies when we planned and facilitated our meetings and consulted the guidance available from INVOLVE on the principles of involving the public in research from the National Institute for Health Research. 63

Initial searches and application of inclusion criteria

The results of the searches and study selection are presented in Figure 2. The initial electronic search was conducted in August 2015 and 13,352 records were retrieved. An additional 5444 records were identified through ProQuest, hand-searching and citation tracking. After deduplication, 13,542 titles and abstracts were screened and 399 articles were shortlisted for full-text assessment. Four trials were identified during hand-searching:64–67 two are still recruiting participants,64,67 one has been completed but the results have not been published66 and the status of one trial65 is unknown. The authors of these studies were approached by e-mail and telephone for results, but no responses were received. These studies are tabulated in Table 21 in Appendix 2.

FIGURE 2.

The PRISMA flow diagram.

The reasons for excluding studies are tabulated in Appendix 3.

All of the tests (computerised and pen-and-paper) identified during screening are listed in Appendix 4.

Included studies

In total, 16 studies68–72,74–81,83,85,86 were included in the diagnostic accuracy review.

No studies met the review inclusion criteria for monitoring progression in MCI or early dementia and, therefore, there is no further mention of monitoring disease progression in the results section.

The details of the 16 included studies68–72,74–81,83,85,86 are summarised in Tables 8–10. Five studies reported results for more than two groups of participants: three included healthy controls, MCI and early dementia,71,76,86 and two included healthy controls, early dementia and early dementia/MCI combined. 70,74

In total, 10 studies68,69,71,72,75,76,78,79,85,86 evaluated the use of automated computerised tests to detect MCI alone, seven studies70,71,74,76,80,81,86 reported results for early dementia, three studies70,74,77 reported results for combined MCI/early dementia and one study83 reported results for human immunodeficiency virus (HIV)-associated neurocognitive disorder (HAND).

The information in Table 8 shows that the index tests assessed in the included studies varied. However, the references tests used in the studies were generally consistent across studies for patients with MCI and for patients with early dementia.

Study quality assessment

A summary of the quality assessment conducted is presented in Figures 3 and 4.

FIGURE 3.

Risk of bias and applicability concerns: summary.

FIGURE 4.

Risk of bias and applicability concerns graph: presented as percentages.

We used the modified form of QUADAS-2,62 which is recommended by the Cochrane Diagnostic Test Accuracy Reviews Guidelines,87 to assess the quality of the included studies. Concerns regarding the risk of bias and applicability have been presented as a summary and as percentages in Figures 3 and 4, respectively.

The risk-of-bias criterion for patient selection was high for seven studies,71,72,74–76,81,83 unclear for one study86 and low for eight studies. 68–70,77–80,85 The studies were judged to be at high risk for this criterion because a case–control study design was used.

Regarding the index test criteria for risk of bias, seven studies70,71,74,76,78,83,85 were judged to be at unclear risk. In these studies, the threshold values for the index tests were not pre-specified. However, for all of these studies, except Dwolatzky et al.,71 it was clear that the index test results had been interpreted without knowledge of the results of the reference standard.

We gave a judgement of high concern regarding applicability of the index test for all of the studies68–72,74–81,83,85,86 because the interpretation of the index test was different from the review question, as it is not possible to diagnose MCI and early dementia using automated computerised tests in isolation; specialist expertise is necessary to establish a diagnosis.

The reference standard domain for the risk of bias was unclear in eight studies,68,69,72,75,79–81,86 as it was not possible to ascertain whether or not reference standard results were interpreted without knowledge of the results of the index tests. However, the reference standard used in each of the studies was likely to correctly classify the target condition. All studies but one76 were judged to have low concern for applicability regarding the reference standard. The one study76 used a consensus of two clinicians’ opinions as the reference standard.

In the flow and timing domain for the risk of bias, a judgement of unclear risk of bias was given to two studies72,83 as a result of there being no details of timing or attrition described in the published papers. Not all of the studies (n = 11)68–71,74,76,77,79,80,85,86 reported whether or not there had been an appropriate interval between the index test and the reference standard. However, 14 studies68–71,74–81,85,86 were assessed as being at low risk owing to all patients having received the same reference standard and all patients being included in the analysis.

Although only one study77 was judged to be at low risk of bias across the four domains, the studies were considered to be of good quality. Patient selection issues were the most likely to introduce bias.

The only concern for applicability was the one previously mentioned, that is the interpretation of the index test was different from the review question as it is not possible to diagnose MCI and early dementia using automated computerised tests in isolation.

Index test details

The time required to self-administer the different index tests ranged from 10 minutes79,81 to 45 minutes. 71 A range of cognitive domains was tested across the different tests, memory and executive function being the most common. Even though all of the tests were self-administered, a practice session was offered to participants in five studies69–71,79,86 to make sure that patients were familiar with the software and platform for the index test. Nine studies68,69,71,72,75,77–80 reported that support staff were present in the room to address any questions from participants about the software or platform.

Study and participant characteristics

The characteristics of the included studies are presented in Table 10. A total of 2832 people were included in the studies, with the number of people in each study ranging from 5085 to 766. 81

Two studies were multicentred71,85 and one study71 was carried out internationally, in Canada and Israel. The earliest study was published in 200180 and two were recently conducted in 2014. 72,78 The study participants were generally similar across the studies (owing to relatively similar study inclusion and exclusion criteria), although four studies69,80,81,86 did not report their exclusion criteria.

The participants were mainly recruited through primary care68,72,79,85,86 or from a memory clinic. 71,75,76,81 Six studies were based in the USA,70,76,77,80,83,85 three were based in the UK,68,69,86 two were based in Australia,75,79 one was based in Ireland,81 one was based in Canada and Israel,71 one was based in Canada,78 one was based in Spain72 and one was based in Finland. 74 Participants in 13 studies68,70–72,74,75,77–80,83,85,86 had 874 to 2280 years of education; for the other three studies, data describing the education of the participants were not reported. 69,76,81 Only five studies70,72,77,83,85 reported data on ethnicity. There is a lack of demographic data in two studies69,76 for participants who were tested for MCI. Fifteen studies reported on cognitive impairment without comorbidities. 68–72,74–81,85,86 One of the included studies reported on cognitive impairment with a comorbidity (i.e. HIV). 83

Results from studies on cognitive impairment without comorbidities

The diagnostic accuracy of 11 automated computerised tests for the detection of MCI and/or early dementia without comorbidities was evaluated in 15 studies. 68–72,74–81,85,86 One study76 evaluated a test which did not have a name. Three studies72,74,81 reported on multiple or singular domains of the Cambridge Neuropsychological Test Automated Battery (CANTAB). The pooling of data from these 15 included studies was considered inappropriate as there were few studies evaluating the same index test in the same population, and it was possible to extract 2 × 2 data from only five72,74,80,81,85 of the 15 studies.

Results from studies on cognitive impairment with comorbidities

The diagnostic accuracy of one automated computerised test was evaluated in one study83 that included people with cognitive impairment with comorbidities. This study examined HAND and used the automated test CAMCI. CAMCI assessed multiple domains with different tasks, which included simple reaction time, recurring picture, go/no go rule 1, go/no go rule 2, word recall, digit span forward, digit span reverse, shopping trip directions task, shopping list task, errand – bank, errand – post office, completion of the shopping list task and incidental recall task. The study set out to examine a range of diagnostic accuracy outcomes, but did not report the values for all of them.

The study authors state that the AUC analyses indicate that the digit span, forward digit span size and functional driving task differentiated between HIV-positive individuals with and without HAND. 83 However, the results presented show a low to moderate sensitivity but poor specificity. The authors of the study presented values for several, but not all, tests. It was reported that, in the forward digit span, a raw score cut-off point of ≥ 6 gives a sensitivity of 63.0% and a specificity of 19.0% and a forward span size raw score cut-off point of ≥ 5 gave a sensitivity of 80.0% and a specificity of 9.0%. The study also reported that the raw score on the shopping trip directions task of ≥ 13 gave a sensitivity of 75.0% and a specificity of 18.0%. 83

Patient and public involvement

Data from the included studies describing the index test details, characteristics of study participants and diagnostic accuracy outcomes were presented and discussed with a service user. The structure of this meeting is described in Appendix 5. As some of the index tests showed high sensitivity and specificity when used to examine one or two cognitive domains, the service user thought that all of the index text domains ought to have been explored by the authors of the studies to enable a comprehensive overview of any cognitive impairment identified. In addition, the service user considered that more information on key domains would help clinicians and patients to address the challenges faced by patients with MCI or early dementia as they carry out their everyday activities. The service user thought that the studies covered different populations very well, but raised concerns about the age of the study participants, as none of the tests evaluated cognitive impairment in people older than 90 years of age. A further concern was that the level of education of the person being tested might affect his or her performance on the test.

The discussion with the service user was mostly centred around the importance of the index tests being user-friendly and acceptable to patients. In a conversation about the different kinds of platforms used for the tests, the service user noted that if a patient had tremors then a desktop computer would be preferred to a touchscreen test, especially if the touchscreen test was very sensitive to touch.

When discussing the different types of tests available, the service user also highlighted the importance of ensuring that the test had a clear contrast between colours because it is likely that older people will have cataracts, colour blindness or other problems with their eyesight.

The service user also stated that some people might become frustrated with tests that lasted longer than 40 minutes, especially people who are not familiar with modern technology and people who are considered to be very elderly. The service user suggested that a patient’s GP might be the best person to make a decision about whether or not the patient would be able to complete an automated test and generate meaningful results, as some people are more comfortable than others when using technology.

Individual test costs

After identifying the individual index tests via searching, a second search was run to identify the total costs of using these computerised tests and, where possible, to determine the acquisition costs of the tests. A number of grey literature sources generated some cost data, but it was not clear if these costs were yearly subscription costs for an organisation that had planned to use the test or if they were one-off costs for the use of one test by one person. 88–91 The authors of the publications did not state whether or not the costs also included the costs associated with training or administering the test, and the need for a health-care specialist to score/interpret the test was not discussed.

Summary of main results

We identified 16 studies68–72,74–81,83,85,86 for inclusion in the diagnostic accuracy review of automated computerised tests to detect MCI or early dementia; only one study83 included patients with comorbidities. No studies met the review inclusion criteria for monitoring disease progression.

Ten studies68,69,71,72,75,76,78,79,85,86 evaluated the use of automated computerised tests to detect MCI alone, seven studies70,71,74,76,80,81,86 reported results for early/mild dementia, three studies70,74,77 reported results for combined MCI/early dementia and one study83 reported results for HAND.

Eleven different index tests were evaluated. The only study which evaluated the diagnostic accuracy of automated computerised tests for people with cognitive impairment with comorbidities was conducted by Rosenthal et al. 83

Most of the included studies had small sample sizes; 10 studies68,69,71,72,74–77,81,86 had fewer than 150 participants and only two studies79,85 had more than 500 participants. Differences in study design, primary outcome measure cut-off points, the likelihood of bias, the reporting of summary statistics (e.g. 2 × 2 data) and disparity in the index tests precluded a meta-analysis of the data reported in the included studies. Meaningful synthesis of the diagnostic outcomes from the included studies was not possible.

Of the 16 studies,68–72,74–81,83,85,86 only one77 was judged to be at low of risk of bias across the four domains examined; despite this, the overall quality of the included studies was considered to be good. Patient selection issues were the most likely to introduce bias. The only concern for applicability was the one previously mentioned, that is, the interpretation of the index test was different from the review question as it is not possible to diagnose MCI and early dementia using automated computerised tests in isolation.

Applicability of findings to the review question

Ideally, for diagnostic purposes, an index test with high specificity is preferable, and high sensitivity is preferred for screening. 92 When diagnosing patients with MCI and/or early dementia, an index test with both high sensitivity and specificity is needed to be able to appreciate a distinctive pattern of cognitive impairment in MCI and early dementia.. This distinctive pattern of cognitive impairment is different from the cognitive impairment caused by other disease processes (e.g. cognitive impairment as presented in depression or HIV). The distinctive pattern of cognitive impairment caused by other conditions such as depression requires the use of elaborate scales such as the Cambridge Cognitive Examination,93–95 the Alzheimer’s Disease Assessment Scale96,97 or the Repeatable Battery for the Assessment of Neuropsychological Status. 98,99 Current pen-and-paper tests screening instruments for MCI cannot do this.

In the case of HIV, however, it is difficult to draw a clearer distinction in the pattern of cognitive impairment, as no attempt has been made to adapt the diagnostic criteria for HAND to the pattern of cognitive domains that are affected in HIV. 84,100,101 Although it is difficult to find robust data, the dominant theory is that HAND causes deficits primarily in executive function, psychomotor speed and attention. Language is not thought to be affected, and neither are learning nor recall, although deficits in the other domains can present themselves to the patient as communication and memory difficulties in the ‘real-life’ scenario (as opposed to the testing environment). 102 This makes it difficult to separate it from or equate it to MCI or early dementia.

Most studies were not conducted in samples representative of the usual clinical population in which these tests might be used (e.g. patients coming to memory clinics with a mix of MCI and dementia of various aetiologies, some were the ‘worried well’ and others were depressed) but were conducted in convenience samples of patients with limited diagnoses (mostly MCI and AD). Furthermore, only three studies68,69,86 out of 16 were conducted in the UK.

It is difficult to draw a clear picture of the diagnostic accuracy of the index tests that were assessed in this review, as there is not enough evidence to support the use of one test over another.

It is not clear from reviewing the included studies if these computerised tests ought to be used in primary and/or secondary care. In the UK, some GP practices take part in ‘case finding’ for dementia, for example specifically targeting ‘high-risk’ groups (e.g. older adults, or patients with high vascular risk, learning disability or Parkinson’s disease), and hospital staff undertake brief cognitive assessments during all acute admissions for older adults.

The pen-and-paper tests currently used in clinical practice not only help clinicians differentiate between normal cognition, MCI and dementia,45,46,48 but also assist in staging severity of illness. In contrast, CANTAB was the only automated test in this review that could stage severity. 72,74,81 The time taken to complete these computerised tests varied between 10 and 45 minutes but was not reported in two studies. 72,74 In contrast, pen-and-paper-based tests typically range from 7 to 10 minutes to complete. 45,46,48 Concern for the time it takes to complete the tests was raised by the service user, who pointed out the possibility of people becoming frustrated with tests that lasted for more than 40 minutes, especially if they were not familiar with using technology.

The data in the included papers did not describe the time required for training, administration, scoring and interpreting the test results.

Comparisons with previous research

None of the previously conducted relevant reviews56,57,103 in this area carried out a diagnostic accuracy review. They were narrative reviews56,57,103 that provided a summary of the battery of tests used and then rated this evidence on validity and reliability, comprehensiveness, and usability. Our review focused on automated computerised tests that were self-administered and had a minimum level of involvement from health-care professionals. In line with our review findings, the authors of the other reviews56,57,103 concluded that there are significant differences in automated computerised tests and hence they must be judged on a case-by-case basis. 57

Strengths and weaknesses of the review

The search strategy for this review was extensive and included multiple databases and grey literature sources. The majority of studies were identified by terms related to the index test, the terminology for which is reasonably standardised. To capture the difficult-to-locate studies where the index test is not referred to in the parts of the electronic record available for search retrieval, we searched the Cochrane Dementia and Cognitive Improvement Group’s register of diagnostic test accuracy studies. The methodological rigour of the review process was enhanced by the use of two assessors to perform citation screening, quality assessment and data extraction/checking. We also contacted all of the primary study authors and asked them to fill in the actual values of a 2 × 2 table. Out of 15 authors, only two were able to send in the requested details. We also conducted a patient and public involvement exercise, but were able to receive feedback on our review findings from only one service user representative.

We excluded studies in which the automated tests or the studies evaluating the automated test were not described in English. The utility of this review is limited in part by the heterogeneity of the automated computerised tests assessed in the included studies. Owing to substantial diversity in the index tests and the characteristics of the participants, it was not appropriate to pool the data. The poor reporting of the diagnostic accuracy outcomes by the authors also hampered data extraction, quality assessment and meaningful synthesis.

We were also unable to identify any studies that reported on any measure of test acceptability. We did not identify any studies that reported outcomes related to monitoring disease progression.

Implications for practice

The overall quality and quantity of information is insufficient to be able to make recommendations on the clinical use of the computerised tests for diagnosing and monitoring MCI and early dementia progression. The suitability of these tests also depends on the costs of the test, training, administration and scoring. Increased effectiveness of a test could offset the costs of the test, equipment and staff training.

The diagnosis of patients with MCI and early dementia is currently based on clinical judgement and medical history as well as on the results of cognitive tests. However, a ‘definitive’ diagnosis of dementia can be made only after a post-mortem analysis of the brain. We also note that autopsy studies in dementia have shown that there is little correlation of these results with the clinical diagnosis made when patients were alive. Therefore, the reference standard remains a clinical diagnosis based on history and physical and cognitive examination. For this reason, we would recommend against approaches that use computerised tests in isolation. Even with pre-specified cut-off values for a particular population, any cognitive testing measure alone is insufficient to render a diagnostic classification: other relevant clinical information must be taken into account.

Older people are more likely to have motor limitations (e.g. as a result of strokes, pain or tremor) or vision and hearing impairments that may render computerised testing impractical. This raises the issue of whether or not these tests can be truly self-administered and valid. For these patients, automated computerised tests may not offer benefits over current practice.

Implications for research

Further research is required in order to establish stable cut-off points for each automated computerised test used to diagnose patients with MCI or early dementia. These cut-off points need to be tested in specific patient populations, for example in patients of different age groups or education levels and from different geographical regions.

The prevalence of dementia and alternative diagnoses in the study populations should be clearly reported and make reference to standardised checklists for diagnostic reviews such as Standards for Reporting Diagnostic Accuracy104 or Standards for Reporting Diagnostic Accuracy – dementia. 105 Investigators might consider evaluating automated computerised test performance at different cut-off points, across populations with varying prevalence of MCI and early dementia. Once sufficient studies in the defined populations are conducted, further reviews and meta-analyses can be carried out.

Future research in this area should also focus on providing more information on the costs of computerised tests and include time for training, administration and scoring of the different tests, as these are important factors for their use in routine clinical practice. This type of information is currently lacking in the published studies describing computerised tests used to diagnose or monitor people with MCI or early dementia.

Contributions of authors

Dr Rabeea’h W Aslam drafted the review protocol, managed the process of study selection, data extraction and data checking, facilitated the service user feedback and drafted the final report.

Dr Vickie Bates commented on the protocol, contributed to study selection and data extraction and provided input into the final report.

Dr Yenal Dundar contributed to the initial search strategy, commented on the protocol, contributed to data checking and provided input into the final report.

Ms Juliet Hounsome commented on the protocol, contributed to data extraction and provided input into the final report.

Miss Marty Richardson commented on the protocol, contributed to the discussion on possible statistical analysis and provided input into the final report.

Mrs Ashma Krishan commented on the protocol, contributed to the discussion on possible statistical analysis and provided input into the final report.

Professor Rumona Dickson commented on the protocol and provided input into final report.

Dr Angela Boland provided input into the draft and final reports.

Ms Eleanor Kotas conducted the literature searches for studies and costs.

Miss Joanne Fisher contributed to data extraction.

Dr Sudip Sikda was clinical consultant, reviewed the protocol and provided input into the final report.

Professor Louise Robinson was clinical consultant, reviewed the protocol and provided input into the final report.

Data sharing statement

All available data can be obtained by contacting the corresponding author.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health.

Receiver operating characteristic curve

When an intervention test has a range of possible thresholds that could be used to divide results into test positive and test negative, the relationship between the threshold used and the performance of the test can be examined in a receiver operating characteristic curve. This is a graphical plot of the sensitivity (TP rate) against 1 – specificity or the FP rate for each threshold; examples of a receiver operating characteristic curve are shown in Figure 5, with the associated distribution of the index tests in diseased and non-diseased populations. An ideal test would have a point in the top-left corner with 100% specificity and 100% sensitivity.

FIGURE 5.

Examples of a receiver operating characteristic curve. (Image reproduced from Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy with permission. 107)

Area under a receiver operating characteristic curve

The receiver operating characteristic curve can be used to assess the degree to which sensitivity changes at different levels of specificity or vice versa. Some studies report AUC as a proportion of the total area of the graph. This is a measure of the predictive accuracy or discrimination of the diagnostic test, that is, the ability of the test to discriminate between those who have (or will develop) MCI from those who do not have (or will not develop) MCI.

The AUC can also be expressed as the probability that someone with the disease will have a higher test result than someone without the disease. It is also referred to as the c-statistic. An AUC of 1.0 indicates a perfect test, and an AUC of 0.5 (the diagonal line) indicates that the test is no better than chance (i.e. 50% probability) in predicting whether or not the disease is present. An AUC of 0.5 to 0.7 is considered as poor discrimination, 0.7 to 0.8 acceptable discrimination, 0.8 to 0.9 excellent discrimination and > 0.9 exceptional discrimination.

Positive predictive value and negative predictive value

The PPV is the probability that subjects with a positive screening test truly have the disease.

The NPV is the probability that subjects with a negative screening test truly do not have the disease.

The PPV and NPV are clinically significant, as they give probabilities that an individual is truly MCI/early dementia positive given that they tested positive or truly MCI/early dementia negative given that they tested negative.

Likelihood ratio

The LR gives another measure of performance for the disease, and is described in chapter 10 of the Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy,107 as follows:

The LR+ describes how many times more likely positive index test results were in the diseased group than in the non-diseased group. The LR+, which should be > 1 if the test informative, is defined as:

(where T+ is index test positive, T– is index text negative, D+ is diseased, D– is non-diseased, P means probability and | means ‘given that’ or ‘on condition that’) and is estimated as:

The LR– describes how many times less likely negative index test results were in the diseased group than in the non-diseased group. The LR–, which should be < 1 if the test is informative, is defined as:

and is estimated as:

Advisory group members: computer based tests for mild cognitive impairment and dementia

By Aslam, 19 August 2015.

Do you want to get involved in research to find out whether computer-based tests for assessing a person’s memory and thinking work?

The Liverpool Reviews and Implementation Group specialises in using a research method called a systematic review of the literature. This method provides an overview of the relevant research in the subject area.

The Liverpool Reviews and Implementation Group is currently working on a research project funded by the National Institute for Health Research. They are currently looking for people living with mild cognitive impairment (MCI) or early dementia as well as their carers who have an interest in assessment tools for thinking and memory.

As part of the research process, they would like to form an advisory group. The advisory group will have 2–3 service users or carers who have an interest in assessment tools for thinking and memory. The advisory group would be expected to attend two meetings between August 2015 and December 2015, and will need to be available for consultation via e-mail or face to face between meetings. They will pay a fee for consultation and travel expenses for all meetings. To apply, please complete the application form and return to the Liverpool Reviews and Implementation Group.

For more information contact:

Dr Rabeea’h Waseem Aslam, Liverpool Reviews and Implementation Group, University of Liverpool.

Telephone: XXX

E-mail: XXX

Deadline: 20 September 2015.