The 4 'A's test for detecting delirium in acute medical patients: a diagnostic accuracy study

Health economics

This study also wanted to compare the implications of any differences in diagnostic accuracy between the CAM and the 4AT with respect to health-care costs. Delirium is known to be economically costly,21,62 and the prevention of delirium is cost-effective. 63,64

Survey development

The surveys (survey A is in Appendix 1 and survey B is in Appendix 2) were developed in multiple stages. They were initially drafted in web form and revised by Antaine Stiobhairt, Alasdair MJ MacLullich and Susan D Shenkin based on previous literature and personal clinical experience of delirium. The revised surveys were reviewed by members of the study team, who assessed face validity, structure and clarity, and further revisions were made. Subsequently, survey A was piloted with 19 additional health-care practitioners outside the study and survey B was piloted with five. In each case, participants were asked to comment on content and any technical problems. Minor amendments were made to both surveys.

In both surveys, the items were presented in a fixed order (with no randomisation) and included multiple choice, ranking, five-point Likert scale and open-comment response formats. Response options including ‘N/A’ (not applicable), ‘don’t know’ and ‘other (please specify)’ were provided throughout the survey, and the majority of questions were optional in order to minimise response coercion and attrition. The surveys began with six demographic items, two of which (career stage and specialty) were presented only to respondents who had a primary qualification in medicine. Some additional questions were presented to those with this qualification (see Appendix 1). Both surveys finished with an open-comment box in which respondents were invited to ‘comment on any of the issues raised in this survey or additional issues surrounding the detection and assessment of delirium that have not been addressed’. No incentives were offered for survey completion.

The surveys were considered to be service evaluations, as the participants were anonymous health-care practitioners and the surveys were of current practice; thus, formal research ethics approval was not required. The surveys were hosted on the internet using Survey Monkey (www.surveymonkey.com). Invited recipients who clicked on the hyperlinked URL in the e-mail were initially presented with a web page that described the study in greater detail and explained that participation was anonymous, that no computer location information or cookies would be collected, and that the results would be published and may involve direct quotations. Potential respondents were then presented with the question ‘Do you agree to participate in this survey and consent to the potential use of your anonymised responses as described above?’. Those who chose ‘I agree’ proceeded to the survey, whereas those who chose ‘I do not agree’ were directed to a page that explained that they had to agree if they wanted to participate. As it was anonymised, the survey did not allow for checking for multiple responses from single users. No time-stamp data for individuals were analysed.

Participants

For Survey A, the health-care practitioners of interest were medical practitioners, nurses, occupational therapists and physiotherapists working in the UK who came into contact with delirium as part of their daily routine. These professional groups were selected as they were numerically large and came into frequent contact with patients with delirium. A list of e-mail addresses of potential respondents was generated through networks of study authors and through internet searches focused on, but not limited to, EDs, ICUs, acute assessment units/medical assessment units (MAUs), elderly care, orthopaedics, oncology, stroke units and palliative care, as delirium is common in these settings. Contact details of relevant associations/societies and trusts/health boards were also obtained through internet search engines.

A standard invitation e-mail was sent to individual practitioners on 30 September 2014. Amended versions of this e-mail were sent to trusts/health boards across the UK, and to relevant societies/associations (geriatrics, psychiatry, acute medicine, nursing, palliative care, etc.). All recipients were asked to forward the e-mail to relevant practitioners in their own contacts list and to consider displaying an A4 poster highlighting the survey in their staff areas. Reminder e-mails were sent on 21 October 2014 and 18 November 2014. Each of the study collaborators forwarded the e-mail to their contacts informally during this time. The survey was closed on 3 January 2015, by which time participation had tapered off.

Survey B was intended for any health-care practitioners worldwide who had used the 4AT to screen patients for delirium. A convenience sample of suitable practitioners was obtained through multiple methods. A standard invitation e-mail was distributed to practitioners working in units where the 4AT was known to be in use on 4 November 2014. Practitioners who completed survey A (Delirium Detection and Assessment in Clinical Practice), who had confirmed that they had used the 4AT and who had provided their e-mail address were e-mailed on 6 November 2014. All recipients were asked to forward the e-mail to 4AT users in their own contacts list and to consider displaying an A4 poster highlighting the survey in their staff areas. Reminder e-mails were sent on 2 December 2014. Each of the study collaborators forwarded the e-mail to their contacts informally during this time. Posters were also displayed in staff areas of units in the Royal Infirmary of Edinburgh that use the 4AT, and paper slips containing the link were handed to practitioners informally. The survey was closed on 22 January 2015, by which time participation had tapered off.

Data analysis

Quantitative data were analysed using R version 3.0.2 (2013; R Core Team, The R Foundation for Statistical Computing, Vienna, Austria). The majority of the data had non-normal distributions and heterogeneous variance across groups; therefore, medians and interquartile ranges (IQRs) are reported throughout and non-parametric statistical tests were used.

For comparative data in survey A, the threshold for statistical significance was a p-value of < 0.05. Between-group analyses were carried out using chi-squared tests where both variables were categorical, using Kruskal–Wallis tests with Holm–Bonferroni method followed by Mann–Whitney U-tests. For items whose responses were given on five-point Likert scales, verbal responses were converted to the corresponding numeric responses 1–5 to facilitate between-group analyses, and analysed as continuous variables. Mean scores and standard deviations (SDs) were reported for these converted responses, as the median and IQR rarely varied from 3 and 2–4, respectively, owing to the restricted scoring range of 1–5. Between-group analyses involving work settings were carried out on participants who worked in single inpatient settings only, in order to avoid the possible confounding effects of working in multiple settings. Owing to the large number of significant pairwise comparisons, details of significant effects were not reported in text when the effect size r < 0.15, but were included in tables only. Unanswered items and sample attrition resulted in missing data and fluctuating sample sizes across and within items; therefore, the results are based on the total number of responses for each item, excluding cases of missing data and when respondents chose ‘N/A’ or ‘don’t know’, unless stated otherwise. No statistical corrections were applied.

Qualitative data submitted through open-comment fields were reviewed informally and the decision on which quotations to report was based on what was judged to contribute to improve the implementation or refinement of the 4AT.

Survey A

A total of 2671 practitioners agreed to participate in the survey. Of those, 172 stopped before completing the core demographic questions; 137 completed these questions but did not continue; 12 worked outside the UK; 41 had a primary professional qualification in an area other than medicine, nursing, occupational therapy or physiotherapy; for two their primary professional qualification could not be verified; and one was retired. These cases were excluded. Data from 2306 (86%) respondents were retained. A summary of the respondent characteristics is shown in Table 1.

All estimates relate to the frequency of respondents except where stated otherwise. Numbers in cells pertaining to settings are not equal to the total sample size, as respondents could choose multiple settings.

Survey B

A total of 117 practitioners agreed to participate in the survey. Fourteen stopped before completing the core demographic questions (items 1–4) and three stated that they had never used the 4AT in clinical practice. These cases were excluded, giving a final sample of 100 (88%) respondents. The geographical distribution was Scotland (n = 62), England (n = 28), Italy (n = 5), the USA (n = 3) and Australia (n = 2). Appendix 3 gives a summary of respondent characteristics.

In addition to the 4AT, a large proportion of respondents reported that they had used the CAM (68%, 61/90). Among the 28 respondents who named additional tools they had used, the Abbreviated Mental Test (n = 15) and the MMSE (n = 8) were the two most commonly mentioned (see Appendix 4).

All respondents reported having at least moderate confidence in their ability to detect delirium (moderate 28%, 28/100; high 52%, 52/100; and very high 20%, 20/100). The interval from which the 4AT was first used by the individual to the time of completing the survey varied across the sample [‘< 1 month’, 10% (9/90); ‘1–6 months’, 21% (19/90); ‘7–12 months’, 18% (16/90); and ‘> 1 year’, 51% (46/90)].

A detailed summary of user opinions on the 4AT, collected as part of survey B, is provided in Appendix 5. Regarding general opinions on the 4AT, 84% (59/70) had positive views, 14% (10/70) had neutral or mixed views and one respondent (1/70) had a negative view. When respondents were asked how often they used the 4AT with patients at risk of delirium, 33% (30/90) stated ‘never/rarely’ or ‘sometimes’, 14% (13/90) said ‘about half of the time’ and 52% said ‘frequently’ or ‘almost always/always’. When respondents were asked if they thought that using the 4AT as part of routine assessment was feasible in their unit, most (95%, 81/85) said ‘yes’, with many referring to the ease (39%, 15/38) and speed (29%, 11/38) of use. One question addressed the extent of knowledge of delirium that respondents thought necessary for health-care practitioners to have in order to use the 4AT effectively: 7% (6/84) stated ‘none/very little’, 58% (49/84) stated ‘some’ or ‘a moderate amount’ and 35% (29/84) stated ‘quite a bit’ or ‘an extensive amount’. With respect to specific training in use of the 4AT for health-care practitioners to be able use the tool effectively, 17% (14/84) stated ‘none/very little’, 60% (50/84) stated ‘some’ or ‘a moderate amount’ and 24% (20/84) stated ‘quite a bit’ or ‘an extensive amount’.

When respondents were asked whether or not the 4AT was used as part of routine assessment by them or others in their unit, 69% (57/83) said ‘yes’, with two respondents commenting that they were currently carrying out audits of 4AT use in their units. Regarding barriers to using the 4AT in respondents’ units, several were identified (Table 3). The use of an alternative tool was not considered by most to be a significant barrier. Opinions on other barriers showed a broad distribution across the sample, suggesting that implementation of delirium assessment tools is complex. Two particular comments from the free-text comments emphasised that the extent of 4AT use depends on broad systemic and cultural factors and not simply on the merits of its utility:

We tried it but then moved back to AMT + CAM as AMT already performed by Nursing staff as part of basic assessment.

Consultant, ED/MAU

We have done some improvement work with it but there was some resistance from colleagues about using it as a screening tool.

Speciality trainee level 3 or above (ST3+), MAU/Medicine of the elderly (MoE)

To gather information about ease of use of the 4AT, respondents were presented with descriptions of three patient groups and asked to indicate their typical experience of using the 4AT with each group (see Appendix 2). For ‘drowsy patients who cannot produce verbal responses’, 51% (36/70) reported that using the 4AT was ‘very easy’ or ‘easy’, 17% (12/70) reported that this was ‘neither easy nor difficult’ and 31% (22/70) reported that this was ‘difficult’ or ‘very difficult’. For ‘patients with dementia who are alert and able to converse’, 77% (60/78) reported that this was ‘very easy’ or ‘easy’, 17% (13/78) reported that this was ‘neither easy nor difficult’ and 6% (5/78) reported that this was ‘difficult’ or ‘very difficult’. For ‘patients who are agitated and distressed’, 44% (33/75) reported that this was ‘very easy’ or ‘easy’, 20% (15/75) reported that this was ‘neither easy nor difficult’ and 36% (27/75) reported that this was ‘difficult’ or ‘very difficult’.

Respondents were also asked how long items 1–3 (alertness, abbreviated mental test-4, attention) of the 4AT typically take to complete: 12% (10/81) stated ‘< 1 minute’, 54% (44/81) stated ‘1–2 minutes’ and 33% (27/81) stated ‘3+ minutes’. Some respondents added that the time taken is ‘affected by deafness’, that they usually have a ‘conversation at the same time’ and that ‘longer time usually hints at worse performance’. Most of those who commented explained that the time taken to obtain collateral history from patients’ family, carers, GP or medical records was highly variable. These findings confirm that the bedside components are brief in most patients but that item 4 (determination of acute onset) can be time-consuming.

A detailed account of responses to survey B questions along with examples of free-text responses is reported in Appendix 3. In addition to giving feedback on the 4AT itself, respondents were asked whether or not they would suggest making any changes to it. The majority did not propose any changes, with 6–10 comments on each item and the guidance notes. Respondents’ comments could be grouped into the following themes: changes to item content, clarity, visual presentation, the scoring system and general comments. Examples are shown in Box 1. There were also some queries about validity and diagnostic accuracy of the individual items. Similarly, a small number of respondents suggested including ‘time’ alongside the four items of the AMT4.

The findings of survey B were discussed with members of the study team, and considered in the light of other information, including additional external validation studies that had been published, the clinical service evaluations that had been collected as part of the study process, and other feedback about the 4AT from outside the study. The team decided that, in the absence of consistent feedback concerning a potential specific change to the 4AT, the study process had determined that no change to the 4AT was required in advance of the diagnostic accuracy study.

Survey A

Survey A had a substantial number of respondents and is, to our knowledge, the largest survey on delirium to date. Because of the nature of the recruitment process, respondents were likely to have had an interest in delirium and/or to have been interested in the management of cognitive impairment more generally. Nevertheless, given the sample size, the results do suggest that there is increasing awareness of delirium among hospital staff in the UK. Additionally, respondents indicated that the majority of units in which they worked had guidelines for delirium detection. These findings indicate potentially positive trends in improving delirium detection and, therefore, care in the UK. However, other findings suggest that substantial challenges remain.

Nine out of 10 of respondents agreed or strongly agreed that formal diagnosis of delirium is important, that distinguishing between delirium and dementia is important, and that delirium treatment improves patient care. Although these data are from a potentially biased sample of health-care professionals, they support the notion that a substantial proportion of practitioners believe that delirium is worth diagnosing. This view in the context of the well-documented poor rates of delirium detection demonstrates a challenging paradox in mainstream clinical practice.

Some of the findings of this study help to address the question of why this paradox exists. One of the striking results is the range of terminology used to describe a patient with ‘recent onset drowsiness’ who is ‘not producing responses but is responding intermittently to one stage commands’. This clinical scenario is clearly consistent with a diagnosis of delirium according to DSM-5 criteria and the accompanying guidance notes, especially considering that delirium is much more common than alternative diagnoses. Yet only one-third of respondents favoured the formal term ‘delirium’ being applied to this case, with several non-diagnostic, ill-defined terms, such as ‘obtundation’, collectively being suggested more often. Notably, a follow-up question asked respondents if the patient described at the beginning of this paragraph had delirium, to which two-thirds of respondents responded that this was ‘likely’ or ‘very likely’. Additionally, a different question directed at medical practitioners found that 72% believed that the patient needed to be able to produce verbal responses to allow bedside assessment of delirium; this view is not aligned with guidance in the DSM-5, which deems it possible to assess delirium in non-comatose patients with acute mental status disturbance who are incapable of speech. These findings indicate that there is remarkable inconsistency in the terms used to describe the clinical states most compatible with an initial diagnosis of delirium. The results in relation to the variable terminology used in the organisations of respondents parallel the findings from individual practitioners. The consequences of using inconsistent terminology are potentially serious, including failure to apply agreed treatment pathways, impaired communication among staff and an absence of adequate communication of the diagnosis to patients and carers. Moreover, an incoherent approach to diagnosis among senior staff makes attempts to provide effective training to junior practitioners and students much more challenging. Thus, these findings have broad implications for education and training, including continuing professional development.

Practitioners showed a range of levels of confidence in their own ability to detect delirium. Given that the sample is likely to be biased towards those with an interest in the condition, it is of concern that at least a substantial minority of respondents, including medical practitioners, have very low to moderate confidence. Given that delirium is very common, affecting at least one in eight hospital patients, this lack of confidence does not stem from unfamiliarity with the condition; rather, it is likely to result from insufficient education and training. The general incoherence around approaches to delirium detection may also contribute to the lack of confidence in many practitioners. The findings showed that just over half of respondents had ever used a specific delirium detection tool; non-specific cognitive tests were more widely used. Related to this, although around two-thirds of units had delirium guidelines, respondents reported that in only a minority of these were the guidelines followed.

Another informative finding concerns the perceptions of the responsibilities of different disciplines with respect to delirium. In particular, respondents mostly felt that nurses were responsible for screening for delirium but not for diagnosing it. It is not rational or pragmatic to restrict diagnosis of such a common condition to medical staff alone, when the diagnosis can usually be made readily with bedside assessment and informant history (or personal knowledge of the patient). On the contrary, it can be argued that nurses who have appropriate training (including an awareness of the DSM-5 criteria) are in a strong position to make a diagnosis, because they have the most direct contact with the patient of any health-care practitioner. Notably, the prompt diagnosis of delirium is advocated by many policy-makers, such as Healthcare Improvement Scotland. This finding has implications for both policy and clear decision-making around the explicit roles of doctors and nurses in detecting delirium and initiating early care. With respect to physiotherapists and occupational therapists, an awareness of the main features of delirium, perhaps coupled with the use of a screening tool such as the 4AT, could lead them to report potential delirium to colleagues qualified to make a diagnosis.

Survey B

Survey B was directed at respondents who had experience of using the 4AT. The final sample size was 100 and comprised mostly doctors and nurses. A large majority of respondents reported carrying out delirium assessments ‘frequently’ or ‘almost always/always’, meaning that this is a sample that is likely to include a substantial proportion of professionals experienced in delirium assessment. In addition to responses to the structured survey questions, multiple free-text comments were provided.

Taking the findings as a whole, respondents generally viewed the 4AT as a useful, rapid and practical tool. Several comments were made about potential changes. However, no clear problem with the 4AT emerged consistently. Many of the issues raised relate not to the 4AT specifically but to general challenges in the assessment of delirium, such as the availability of informant history and the time it can sometimes take to get this history, and basic knowledge of the features of the syndrome itself. Other issues relate to possible modifications of the cognitive tests used, the time frame over which altered mental status is considered to indicate delirium, and so on. Many of these suggestions are reasonable and reflective of variations in accepted, mainstream practice. However, given that the current version of the 4AT is in wide use, supported by several validation studies and mentioned in several policy statements and guidances (see Chapter 3), in the absence of strong positive evidence in favour of these modifications being introduced, retaining the current version is the most pragmatic option.

Around one-third of practitioners reported that it was ‘difficult or ‘very difficult’ to complete the 4AT for ‘drowsy patients who cannot produce verbal responses’. Items 2 and 3 of the 4AT allow for scoring such ‘untestable’ patients, and item 1 allows for scoring drowsy patients. This finding suggests that education about the features of delirium with respect to reduced arousal would be helpful, and specific training on the 4AT concerning this issue would reduce the uncertainty reported by some practitioners. With respect to training, most respondents stated that at least some training in using the 4AT would be needed for the tool to be used effectively. The proportions of respondents stating this were similar to the proportions responding to the question about whether or not training in delirium in general would be required to use the tool. Although the 4AT was designed to be simple and practical, and usable without specific training, the survey findings suggest that users must have a basic understanding of delirium if they are to use the 4AT. This should include understanding that a reduced level of arousal is commonly seen in delirium and contributes to that diagnosis. A number of comments suggested that more guidance is needed about what to ask carers or contacts in terms of changes in patient behaviour. This knowledge is best classed as part of a practitioner’s general information about delirium detection; nevertheless, given the lack of other approaches to delirium detection in routine care, more information on how this should be done in the context of using the 4AT would be valuable. Additionally, in conjunction with this, some training in the use of the 4AT would appear to be beneficial in promoting its effective use. Given the lack of space on the one-page 4AT form, such additional education and training on delirium in general and on the 4AT would best be provided on easily accessible websites, including www.the4at.com.

Some queries arose about the scoring of the cognitive test items. Cognitive tests used in hospital practice show variable diagnostic accuracy for dementia and delirium,46,52,95,96 and no single test provides sufficiently good performance for it to be used in all contexts. In addition, cognitive tests performed in isolation inform clinical judgement, but cannot be performed as diagnostic tests alone. Therefore, practitioners using the AMT4 and Months Backwards tests need to be aware of both the value and the limitations of the information provided by the test results. For practitioners seeking further information about the cognitive test items in the 4AT, the URL of the 4AT website is given on the form (and, indeed, is easily found through an internet search), and an up-to-date list of specific 4AT validation studies, as well as relevant studies relating to the cognitive test items, is provided on that website.

A need for both the scoring and the guidance to be on the paper or electronic documentation was also identified. This sometimes seems to have been lost when the 4AT has been incorporated into a larger assessment such as an electronic patient assessment form. This could be addressed by providing clear links to the 4AT guidance notes or the website.

The current findings emphasise that the use of the 4AT is not based solely on the utility of the tool itself, but also depends on broader systemic and cultural factors. All of those participating in survey B had some experience of delirium assessment, and yet there was some evidence of lack of knowledge about some general aspects of delirium, which had an adverse effect on participants’ understanding of how to use the 4AT. Furthermore, respondents identified external factors that hindered their use of the 4AT, and presumably their use of any other delirium assessment tool.

Policy websites and documents that include the 4AT

• Scottish Standards of Care for Hip Fracture Patients 2016: the 4AT is the recommended delirium assessment tool. 100

• Healthcare Improvement Scotland’s Delirium Toolkit: the 4AT is the recommended delirium assessment tool. 101

• Managing Falls and Fractures in Care Homes for Older People – Good Practice Resource: the 4AT is the recommended delirium assessment tool. 102

• Dementia Revealed: What Primary Care Needs to Know: the 4AT is included as one of the recommended tools. 103

• Assessing for Cognitive Impairment in Older People: Clinical Audit 2014–15: the 4AT is one of the recommended tools. 104

• Falls and Fracture Consensus Statement: the 4AT is now included in the list of ‘quality metrics required to pass best practice tariff’. 105

• London Major Trauma Systems’ Management of Elderly Major Trauma Patients: the 4AT is one of two recommended delirium assessment tools. 106

• US Department of Veterans Affairs’ Delirium Information for Professionals: the 4AT is among the recommended tools for delirium assessment. 107

• American Nurses Association: the 4AT is one of the included tools for delirium assessment. 108

• Early Identification and Initial Management of Delirium in the Emergency Department/Acute Medical Assessment Unit: the 4AT is included as the main delirium assessment tool. 109

• Australian Commission on Safety and Quality in Health Care’s Delirium Clinical Care Standard: the 4AT is included as a recommended tool. 110

• Health Navigator New Zealand (website providing clinical information for clinicians in New Zealand): the 4AT is the recommended delirium assessment tool. 111

Examples of websites and documents documenting 4AT use in clinical practice

• Identifying Critical Success Factors for Improved Outcomes for People With Dementia and Their Carers in Acute Care. 113

• Improving Older People’s Acute Care: Impact Report: summary report of multiple quality improvement projects including delirium in several hospitals in Scotland, involving use of the 4AT. 114

• National Hip Fracture Database Annual Report 2016: the 4AT is documented as having been introduced as a standard clinical tool for delirium detection in hip fracture patients. 115

• Think Delirium: Recognition of Delirium in an Acute Medical Unit: clinical audit, Ninewells Hospital, Dundee, showing improvement in delirium detection following introduction of the 4AT. 116

• Improving Identification and Management of Delirium in the Orthopaedic Trauma Unit: quality improvement project on increasing the use of the 4AT in an acute hip fracture ward. 117

• Derbyshire Healthcare NHS Foundation Trust’s annual report and accounts 2015/16: the report documents that the 4AT was introduced to improve delirium detection. 118

Case study: City Hospitals Sunderland NHS Trust

City Hospitals Sunderland NHS Trust is a large district general hospital with ≈900 beds, of which ≈200 are for acute care of the elderly. The 4AT has been embedded in the trust since 2013. It is used for all acute admissions, and in the ED it automatically appears in the nursing admission documentation to be completed for all admitted patients who are aged ≥ 65 years. Score distribution is approximately 65% of patients with a score of 0, 20% of patients with a score of 1–3 and 15% of patients with a score of > 3. All patients scoring ≥ 1 (and patients about whom there is a clinical concern) are reviewed by the specialist delirium dementia outreach team, comprising four band 7 practitioners (three nurses and one physiotherapist), a 0.5 whole-time equivalent senior pharmacist and a 0.5 whole-time equivalent band 3 administrator, with supervision from 1.5 whole-time equivalent consultant geriatricians. The team assesses 300–400 new and 100–200 review patients each month (total to June 2017, n = 1218; mean age 78.1 years, range 61–102 years). This has resulted in improvements in achievement of the CQUIN target and in delirium diagnosis recording, and a reduction in referrals to the mental health liaison team. Of note is that approximately 75% of patients seen by the delirium dementia outreach team are identified after a positive score on the 4AT, but 25% are also referred by clinical staff as a result of concerns. The majority of these patients score 0 on the 4AT, but, on review, are confused and/or drowsy, and so the score is likely to have been inaccurate. Written feedback from the team was provided: in summary, the 4AT has been widely adopted and is considered easy to use and quick to complete. Patients with scores of 0 may be ‘underscored’; and local review has concluded that this is a staff training issue, rather than an issue requiring a change to the tool.

Case study: King’s College Hospital NHS Foundation Trust, London

King’s College Hospital NHS Foundation Trust is a large acute care teaching hospital with 950 beds. The 4AT is embedded as the screening test for all acute admissions. A delirium and dementia team was established to review patients who screened positively on the 4AT. 119 It has been difficult to validate because of issues with retrieving accurate information from primary care. However, there is a good correlation between higher 4AT scores and delirium features. Just over 80% of individuals screened go on to receive a diagnosis of dementia or delirium at follow-up. Written feedback from the team is that the 4AT is easy to use and score.

Research objectives

The study objectives were to:

1. examine screening practices for delirium at particular locations in the patient journey into and through hospital

2. explore staff knowledge of delirium, the value attached to identifying delirium, and the strategies called forth to manage it, at these same service delivery locations

3. determine the organisational, environmental and system-level barriers at locations along the patient journey that have an impact on the systematic identification of patients with possible delirium

4. consider general staff use of screening tools and the 4AT specifically, the contexts in which the 4AT is used, and the speed, ease and value attached to its use.

We performed a multimethod, qualitative, mixed-methods study comprising in-depth interviews with health professionals and observation of practice in different locations along the patient journey: the ED, the MAU, the surgical assessment unit (SAU), the elderly assessment unit (EAU), and selected acute wards. The rationale for combining interviews and observation was that whereas interviews can explore staff understanding of delirium, and the meaning of, and value attached to, delirium screening in particular settings, observation provides a picture of what staff actually do in the real-life context of health-care delivery in time and space. This includes what people take for granted or are unable to articulate. The overall aim of the study, therefore, was to provide a descriptive and explanatory account of how hospital staff from different disciplines and at different levels of seniority understood, made sense of and attached value to identifying patients with delirium, in the context of the organisation of work within settings intended to achieve different purposes.

Sampling strategy

Data collection

Data analysis

All audio-recorded interviews were anonymised and transcribed verbatim. As our focus was on interview content, transcription notations were used only to indicate pauses, cross-talking and hesitations. Researchers conducting the interviews listened to the audio-recording alongside the transcribed text for accuracy of transcription. They also completed a pro forma for each interviewee, which included the interviewee’s salient characteristics (profession, grade/seniority, gender and organisational setting/ward type). A summary was produced as soon as possible after the interview; this included the main issues raised, the research questions on which the contact was most focused, new ideas generated to pursue in the next interview, and/or areas that were insufficiently explored. Audio-recordings, transcriptions and summaries were shared and discussed within each local team; transcriptions and summaries were shared between the two local teams. We did not send transcriptions to interviewees for comment; although this offers potential for further discussion and elaboration, its use as a method of validation is contested, reflecting varied methodological and epistemological stances.

Contemporaneous notes from informant interviews and field notes were written up as expanded accounts as soon as possible after these events. 131–133 The researchers’ impressions of, and reactions to, what they had seen and heard were recorded separately in a chronological journal. All data sets were stored electronically and organised by site and setting. Research teams involved in the English and Scottish sites met regularly to discuss the emerging data and what to make of them, and to compare and contrast findings within and across sites; team members also communicated regularly by e-mail and telephone.

Data from all sources were analysed using grounded theory methods,134 including simultaneous data collection and analysis, constant comparison, search for negative cases and memo-writing. Multiple readings of all interview transcripts, summaries and field notes by team members individually familiarised researchers with the data, including their variability and range within and across settings and at individual and organisational levels. Through discussion, we developed tentative ideas, captured in memos, of what was going on at both levels. For example, what were the distinctive features of clinical work undertaken in different organisational settings; how did these inform the delirium-related work carried out by individual staff; and how did delirium knowledge across sites and settings influence the value attached to the work of delirium identification, assessment and action?

Data sets (transcripts and field notes) relating to each organisational setting were subject to initial open coding by local research team members, individually and together. This first phase involved synthesising, integrating and organising data segments into named meaning units on hard data copies, with each data source examined on its own terms. In this way, we sought to retain the coded segments in the context of the whole interview or field note. Initial codes were provisional, primarily descriptive and close to the data, and aimed at understanding participants’ meanings, accounts and actions. Coded transcripts were shared, compared and discussed among the research team, differences in interpretation often giving rise to new insights. For example, participants in Denbury often prefaced their accounts of action on delirium in terms of what they should do, suggesting the need to examine how normative practices relating to delirium identification were introduced, and what was shared, subverted or sustained over time.

From the initial codes, we agreed which to pursue for more in-depth focused coding. The open code ‘knowledge of delirium’, for instance, was developed as an analytic category during focused coding comprising several components (formal knowledge of diagnostic criteria; practice knowledge based on exposure to patients with delirium and its effects on them; and tacit knowledge from practice experience, emotions, insights and observation). Analytic categories, components and descriptors, and the research objectives they related to, were displayed and managed in Microsoft Word (Microsoft Corporation, Redmond, VA, USA) files organised by site and setting. With regard to knowledge, for example, we compared and contrasted different forms of knowledge to examine how they variously shaped meaning and action on delirium identification between settings, between professionals and across sites, and created diagrams of these relationships. These were captured in memos and further tested out with the data.

In addition to coding and categorising data, we explored the data narratively to examine processes and context systematically. For example, regarding delirium identification, we perused larger sections of transcripts and notes to examine the narrative structure of accounts: what preceded a particular account/incident and what followed from it to capture the processes involved. Comparing and contrasting narratives between settings cast light on how features of organisational context affected these processes. Each stage of analysis involved discussion within the research team: codes and categories were reviewed; memos and diagrams were shared; and explanations developed were refined and verified, moving iteratively between the empirical data and sense-making in relation to them.

Research team

Two investigators (EL and AS) conducted the interviews and observations, with academic support from Mary Godfrey and Janet Hanley (both experienced qualitative researchers). Organised as two local teams, Elizabeth Lavender and Mary Godfrey pursued the research in the English sites and Antaine Stíobhairt and Janet Hanley did this in the Scottish site. They came from different disciplinary backgrounds – psychology (EL and AS), sociology (MG) and nursing (JH) – and from varied research areas: applied health and social care research relating to older people, including with cognitive impairment (MG), adult mental health (EL), learning disability and delirium (AS), and the introduction of new health technologies particularly in primary and community care (JH). None was a medical practitioner. Their varied backgrounds and perspectives generated openness to exploring different ways of considering issues and problems.

The whole team met together for a day at critical points: at the beginning of the research to develop the fieldwork plan; during fieldwork to reflect on emerging data, consider variation within and between sites and what shaped it, and identify new lines of enquiry; and during analysis to discuss and refine codes, categories, analytic concepts and narrative content. Regular telephone conversations, e-mail discussions and sharing of memos, diagrams and matrices occurred between investigators in an ongoing dialogue and to ensure consistency of analytic approach in the three sites. Within local teams, formal meetings and informal discussions took place approximately weekly.

Ethics

Ethics approval for the study in the two English sites was obtained from the National Research Ethics Committee (reference 14/SW/1095). The main ethical issue was not seeking formal consent for general observations of routine practice. As observations were unobtrusive and carried out in public or semi-public spaces, and did not identify individual staff, patients or visitors by name, we considered that it would be impractical, and probably more intrusive, to seek formal written consent from all those present. Instead, we sought informal verbal consent. At the beginning and end of each observation, the researcher ‘signed in’ and ‘signed out’ with a senior staff member. No difficulties were encountered in any of the sites.

Advice from the National Research Ethics Committee in Scotland was that as the focus was on examining the 4AT in use, the study was more appropriately characterised as audit and, therefore, required only local clinical governance approvals; these were obtained. The conduct of the study in this site conformed to the same ethical principles as informed research in the English sites.

The emergency department

Studies examining recognition, detection and delirium occurrence in EDs have been a particular focus of research interest on delirium. EDs are both a patients’ first point of entry into general hospitals and a location in which decisions about appropriateness of acute care are initially made. From systematic reviews135,136 of observational and prospective studies over two decades, a yawning gap exists between the occurrence of delirium among older patients in EDs as assessed by researchers using various measurement tools, and what is identified by ED physicians, typically based on a review of charts/documents. Primarily conducted in the USA and Canada, study findings suggested the presence of delirium in approximately 7–10% of older people admitted to the ED; this is lower in some studies137 and considerably higher among subgroups of older people, such as those who are older, in long-term care or living with dementia. 32 Furthermore, delirium is missed in between half and three-quarters of cases comparing rates of delirium as measured by researchers with detection rates based on chart review. 22,31,32 Although some of this variation may be attributed to differences in methodology and heterogeneity in the measures employed,135,136 it is also likely that contextual factors are implicated.

Assessment units: medical assessment unit; elderly/frailty assessment unit; surgical assessment unit

In contrast to the ED, there is a paucity of research on delirium prevalence and detection in assessment units. Generic assessment units are similar to EDs in that they manage demand and flow near the point of entry to the hospital within a relatively short time and have a diverse patient profile in terms of age and clinical presentation. However, dedicated assessment units for older people, although subject to the same time and pace pressures, are more likely to be staffed by those with interest and expertise in the care of older people.

Acute wards

The occurrence of delirium on acute wards varies with patients’ severity of illness and age profile. This pattern was evident in a study138 that uniquely examined the prevalence and incidence of delirium across acute wards in a large, tertiary hospital using a common suite of screening and diagnostic tools. Whereas an overall prevalence of 20.7% represented the burden of delirium in the hospital, this varied between wards, being highest among patients on medical, neurosurgical and orthopaedic wards and lowest in general surgical wards. Furthermore, those in advanced older age (i.e. ≥ 80 years) had nearly 35% delirium prevalence, compared with around 5% among patients aged < 50 years; and over half of patients with delirium had a pre-existing cognitive impairment. These findings underscore the significance of delirium detection and management being a major focus of clinical and care work on particular acute wards; the importance of detecting and assessing both delirium and cognitive impairment; and consideration of their reciprocal impact on treatment and recovery.

Multilevel knowledge of delirium

Knowledge and understanding of the significance of delirium, including its impact on patient distress and outcomes, were critical to the engagement of staff in delirium detection. Across all settings, delirium detection and the value attached to it were contingent on a particular type of knowledge held by professionals. This was more than ‘awareness’ of the syndrome, including its diagnostic features. Although important, such awareness was insufficient for staff to invest time and effort to systematically carry out the work of identification and action in relation to it. In addition, more in-depth knowledge of the impact that delirium had on patients, the distress that ensued as a consequence of the delirium and its immediate and longer-term effects on clinical and service outcomes was required. Acquisition of this multilevel knowledge meant that staff were able to recognise particular constellations of factors in the patient situation that made them think of delirium. This tacit knowledge, aligned with the clinical presentation, sensitised professionals to note and interpret observational cues to construct a clinical picture. This was built up collaboratively, employing a comprehensive investigatory approach combining observational cues, patient assessments, collateral history, medical notes and nursing observations, in a process referred to elsewhere as horizontal and not vertical use of expertise. 37

The significance of multilevel knowledge is reinforced from studies of the experiences of people with delirium and the staff caring for them. In a synthesis of qualitative studies of patients and nurses experiences of delirium,120 the authors concluded that patients who had suffered delirium experienced safety and comfort from being understood, supported, believed and responded to with reassurance and care and were able to distinguish the quality of care delivery from how they were acted on, spoken to and touched. Nurses who lacked delirium knowledge were more likely to view patients’ behaviour as strange, incomprehensible and a source of stress and irritation; and were unable to respond with support and empathy. Other studies focusing particularly on nursing staff120,121,123–128 have emphasised the importance of staff attitudes and responsiveness to the emotional needs of patients with delirium.

We found that the introduction of an easy-to-use screening tool had the potential to raise awareness of delirium when accompanied by learning and changes to, or modification of, existing systems for information gathering, for example including the tool as an integral component of the assessment process. However, insofar as delirium knowledge remained patchy, the outcomes poorly understood and the benefits of detection not visible to practitioners, the practice of delirium detection was inconsistent given the competing demands of the setting, findings that are echoed in other studies. 37,38,121,123–128,130 Thus, van den Steeg et al. 130 concluded that poor delirium knowledge and skills among nurses acted as barriers to delirium guidance adherence, including systematic use of a risk screening instrument. Problems of adherence in turn undermined belief in the benefits and goals of screening; the consequence was that nurses viewed screening as simply another form, the completion of which took time away from patient care delivery.

Health professionals (irrespective of role) with extensive clinical experience and/or exposure to patients at risk of delirium, and those with expertise in the care of people with a cognitive impairment, were more likely to understand the significance of delirium detection, be sensitive to observational indicators of ‘something not being quite right,’ and pursue lines of information gathering, investigation and interpretation to make sense of symptom patterns. These ‘delirium knowledgeable’ clinicians spanned professional disciplines. They were commonly found among those with extensive clinical experience and/or with specific skill and expertise in the care of older people, and had additionally made the connection between the diagnostic features of delirium and its significance for patient outcomes. ‘Delirium knowledgeable’ nurses, for example, would initiate diagnostic investigations to ascertain the source of delirium without deferring to medical staff. That ‘delirium knowledgeable’ practitioners were found across settings provides partial explanation of the variability in identifying and responding to delirium. The paradox was that for ‘delirium knowledgeable’ practitioners in some settings, a screening tool as an aide memoire for practice was seen as unnecessary, although among ‘delirium knowledgeable’ nurses in the 4AT ‘experienced’ site, the tool reinforced their expertise vis-à-vis medical staff.

Service delivery context, purpose and value attached to delirium screening

The second feature of the study findings relates to how delirium knowledge and practice played out at different points in the patient journey into and through the acute hospital.

The value and priority attached to delirium detection varied with the clinical purpose and organisational context. In an ED environment, the purpose of the setting – obtaining a differential diagnosis to inform decision-making on an immediate course of action within a very short time frame – did not make formal delirium detection salient at this point in the patient journey. The question posed by ‘delirium knowledgeable’ practitioners was whether or not the systematic detection of delirium in this context, with its attendant problems (difficulty of ascertaining a baseline picture; time constraints; and pace and flow pressures), would make a difference to patient outcomes. On the one hand, ‘delirium knowledgeable’ clinicians described their approach to practice as being sensitive to observational signs of ‘acute confusion, and then engaging in a comprehensive assessment process, drawing on multiple sources of information to guide investigations and action, suggesting that there was value in this work. For example, action on identifying and responding to infection as a cause of delirium meant that treatment could be speedily initiated to benefit patients. More problematic was introducing a detection tool for all patients meeting specific criteria (e.g. age). It was notable that although the 4AT had been introduced into the ED in Denbury some time prior to this study, the consensus (not just from ED interviewees but from staff in the MAU and wards) was that it was no longer completed. Knowledge and purpose, then, were key factors in variability of delirium recognition in the ED.

Although some contextual features of MAU settings were thought of as barriers to delirium detection, it was generally agreed that identifying delirium was consistent with their organisational and clinical purpose: to enable further observation, investigations and decision-making to develop a management plan for the patient. Here, too, knowledge and understanding of delirium were critical to engaging staff in the work of delirium detection, creating the conditions for staff to invest in it. Even so, organisational factors including systems for integrating detection in routine care delivery, attention on mechanisms for legitimising and valuing the role and contributions of different professionals, and clarity about how delirium detection informed treatment and care were also necessary to embed delirium detection in care routines. Critical also was that the outcomes of investigations and the determination of actions arising out of them were clearly communicated to those charged with the task of carrying them through at the next point in the patient journey.

The ward was the main site both for ongoing management of patients with delirium and for identifying new cases. In high delirium prevalence settings, such as care of older people wards, staff were more aware of and knowledgeable about delirium. Junior doctors, nursing and care staff reported greater awareness of delirium than colleagues in other medical and surgical wards; they also conveyed greater consistency in their understanding of delirium and ability to distinguish it from dementia, a commonly cited reason for underdetection.

Even so, practice in managing delirium was variable. From this study, as from other research,37,38 critical to the integration of delirium detection and management on acute wards, in addition to multilevel knowledge of delirium, were working practices that placed value on person-centred approaches with patients and acknowledged the expertise of different groups of staff, and systems in place for co-ordinating, evaluating, communicating and sharing progress on recovery.

Our findings do not offer grounds for complacency, as evidence from other studies attests. 122,123,128 Person-centred approaches to practice in respect of people on acute wards living with dementia (and delirium) are the exception and not the norm, as reported in national audits carried out in NHS hospitals in England. 142,143 A recent study144 of practice in respect of people with a cognitive impairment on 10 care of older people and orthogeriatric wards in hospitals in three English regions found that knowledge and understanding of delirium was variable and that consistent collection of data on delirium occurred on only a minority of wards where staff both had knowledge of delirium and had valued ongoing assessment and review as a means to effect and evaluate better management of it. It is perhaps surprising, given the co-occurrence of delirium and dementia, that policy emphasis on dementia in the UK,140–142 and reflected performance targets relating to it, has not also accorded more visibility to delirium.

Across all sites and settings, professional roles and expertise shaped knowledge and action, although the purpose of the setting and interdisciplinary practice were also key. Generally, medical staff assumed primary responsibility for diagnostics and overall treatment planning, but nursing and care staff played a pivotal role in noting and communicating change in patients and in pursuing work practices implicated in non-pharmacological delirium prevention and management interventions. In this respect, interdisciplinary working practices contributed to the establishment of a culture of care in which delirium detection and management were integrated into the broader work of supporting patients with a cognitive impairment in acute care.

Limitations

This was an exploratory study and its limitations should be acknowledged. It was undertaken in three sites only; the number of interviews conducted in each site was small, as was the number of interviewees drawn from different organisational settings on the patient journey from ED through to the ward. At ward level, the majority of those interviewed worked in care of older people wards. Thus, the findings are likely to overstate the engagement of staff in delirium screening compared with staff in wards in which knowledge and salience of delirium was low. At the assessment unit level, the study revealed different forms of service organisation from generic units to surgical units to designated units for older people and those with frailty. In the context of this study, although the findings are suggestive of how different organisational cultures shape the salience attached to delirium detection and practice, the full import of these different organisational forms on strategies to improve practice requires further systematic exploration.

Conclusions and implications

From this study, it is helpful to consider improvement in the identification and care of people with delirium as involving several interacting layers. At the individual level, multilevel knowledge of delirium creates the conditions in which delirium detection and action in relation to it is understood as meaningful and invested in, with regard to the time and skill to engage with it. The second layer refers to the existence of systems, mechanisms and processes to collectively engage and co-ordinate the work of different groups of staff to implement and sustain delirium detection and management in routine practice. This includes appropriate tools for identifying delirium. The third layer relates to the organisational purpose of the setting, namely to what extent can work practices and the division of labour through which these are carried out be modified or adapted to enhance delirium detection and achieve clinical and organisational outcomes. This poses a particular challenge for the ED. Considerable further research is needed to address these multiple challenges.

Ethics

This study was granted ethics approval prior to data collection in Scotland (Scotland A NHS Research Ethics Committee REC 15/SS/0071) and England (Yorkshire and The Humber – Bradford Leeds NHS Research Ethics Committee REC 15/YH/0317).

Study design: overview

Patients aged ≥ 70 years in EDs or acute general medical wards were prospectively recruited in three sites (Edinburgh, Bradford and Sheffield). The total target number of patients was 900 (all sites). Each patient underwent (1) a reference standard delirium assessment lasting up to 20 minutes, and (2) either the 4AT or the CAM (lasting up to 10 minutes). Randomisation was used to allocate participants to the 4AT or the CAM and to determine the ordering of the reference standard and 4AT or CAM assessments. A questionnaire on pre-admission cognitive function was administered to an appropriate informant (if one was available). At 12 weeks, data on clinical outcomes, including length of stay, institutionalisation and mortality, and resource utilisation were collected from medical and social care records, or from patients if the records did not provide sufficient information.

Inclusion and exclusion criteria

Sample size

It was planned that a total of 900 patients would be randomised: 450 to assessment by the 4AT and 450 to assessment by the CAM. Of the 450 patients within each group, 15% (n = 67) would be expected to have delirium. 3 The specificity of each triage tool would be estimated based on the 85% (n = 383) without delirium; sensitivity would be estimated from the 15% (n = 67) patients with delirium. This sample size would allow precise estimation of specificity and moderately precise estimation of sensitivity. Based on analysis using the normal approximation to the binomial distribution, the two-sided 95% confidence interval (CI) width for specificity would range from ± 0.030 to ± 0.050, and for true specificity would range from 0.90 to 0.50. The corresponding CI widths for sensitivity would range from ± 0.072 to ± 0.120.

For the secondary objective comparing the 4AT and the CAM, based on analysis by a continuity-corrected chi-squared test, we have 83% power to detect a difference in specificity of 0.10, assuming a null hypothesis of specificity = 0.70 for both tests and a 5% significance level. The difference detectable for sensitivity (null hypothesis sensitivity = 0.70 on both tests) would be 0.224 (80% power).

Recruitment

Patients were recruited between 08:00 and 22:00. Eligibility screening was carried out by a member of the clinical team, yielding sets of names of potentially eligible patients. Patients were screened in alphabetical order from study opening in October 2015 until July 2016.

This recruitment strategy was changed from July 2016 because analyses suggested that patients at a lower risk of delirium (i.e. those not requiring capacity assessment) were more likely to be recruited than those with impaired capacity. The new recruitment strategy lasted 5 months. The strategy involved using a pragmatic approach to facilitate some oversampling of patients at higher risk of delirium. From the lists of patients identified as per the original strategy, patients considered at higher risk of delirium on clinical grounds (e.g. older age, likelihood of admission, a greater number of comorbidities) were approached first, rather than this being done in alphabetical order. This change was approved by the funder and the Trial Steering Committee.

Potentially eligible participants (n = 4928) were identified as those without critical illness or not in coma, in the ED within 12 hours of attendance, or on acute medical wards within 96 hours of attendance, in three sites (Edinburgh, Bradford and Sheffield: see Appendix 10 for numbers screened, eligible and consented in each site). Study recruitment commenced on 19 October 2015 and was completed on 30 December 2016, with final follow-up data collection and database locking on 29 June 2017. In total, 843 patients were randomised.

Consent

Participants were eligible for this study whether or not they had capacity to consent. Informed consent was sought by trained researchers using a combined informal capacity assessment/consent process. Both verbal and written information was provided about the study, using a style and format suitable for the participant group (i.e. for varying levels of capacity). The researcher then asked the potential participant to recount the study information to check their understanding. This, together with the treating team views, was used to assess the person’s capacity to consent. When the potential participant lacked capacity to consent, recruitment proceeded under the provisions of the Mental Capacity Act 2005162 in England or the Adults with Incapacity (Scotland) Act 2000,163 using an appropriate personal or nominated consultee, guardian, welfare attorney or nearest relative (see the published protocol160 for additional details, including different approaches in Scotland and England).

Test methods

Ordering of reference standard delirium assessment, 4AT and CAM

The reference standard assessment for delirium was administered to all patients by the researcher who conducted the capacity assessment and consenting process. A different researcher administered either the 4AT or the CAM. The rationale for this is that the capacity and consenting process provides information to the researcher that exceeds what the 4AT or the CAM provides. This is not a concern for the reference standard assessment, which is aimed at providing a thorough assessment to optimise diagnostic accuracy. The order of the two assessments [(4AT or CAM assessment) and reference standard assessment] was randomised in the ratio 1 : 1. Participants were also randomised in the ratio 1 : 1 to be assessed using the 4AT or the CAM. Randomisation, using a secure online system, occurred immediately after consenting. Access to the secure system for each randomisation was via a unique username and password for each researcher; the full list of randomised allocations was held within the secure online system and concealed from researchers. The randomisation sequence was created using computer-generated pseudo-random numbers, and was stratified by study site, with block allocation. The two assessments took place within a maximum of 2 hours of each other, with a target interval of 15 minutes. Researchers were blinded to each other’s assessments. The result of the reference standard assessment was recorded in the case notes, and communicated to the clinical team after the assessments were complete. Figure 2 shows the study assessment flow chart.

FIGURE 2.

Diagnostic accuracy study: overview flow chart. a, Researcher A (i.e. the individual doing the consenting) will also always perform the reference standard assessment. Researcher A’s status will not be tied to a particular individual throughout the study.

When possible, the IQCODE was administered following the above tests (within 4 weeks of the patient joining the study).

Research staff and training

Researchers were nurses or trained graduate clinical research associates. All underwent a systematic training process comprising three initial modules with reading material and videos covering (1) the patient and carer experience of delirium, (2) basic knowledge of the features of delirium, clinical diagnosis and treatment approaches, and (3) specific coverage of delirium assessment, including cognitive testing, differentiating delirium from dementia, and the tools and tests used in the present study. These modules took 1 or 2 days each to complete. After these modules, researchers underwent face-to-face training with the chief investigator, and underwent role-play practice and supervised bedside practice with hospitalised patients for the reference standard assessment (guided by, and referring to, the DRS-R98 manual164) and the CAM (the latter training using instructions from the CAM training manual156). Researchers also underwent additional brief familiarisation sessions with the 4AT.

Analysis

The detailed statistical analysis plan was agreed prior to database lock,160 blinded to the randomised allocations. All analyses were performed by statisticians (JS, CW and VA).

Methods: measures of diagnostic accuracy

Descriptive statistics

A total of 4928 patients were eligible, of whom 843 (17.2%) were randomised across the three sites and two withdrew, leaving 841 with data for analysis (Figure 3).

FIGURE 3.

The STARD diagram of flow of participants through the study (total across all three). a, Two had moved ward, one had an issue with consultee. CI, chief investigator.

The initial DRS-R98-informed reference standard identified 96 (11.7%) patients as delirium present, and 723 (88.3%) as delirium absent [22 were missing because they withdrew from the study (n = 18) or had been discharged home (n = 3), or at patient request (n = 1)]. Final delirium ascertainment by the consensus group reclassified seven patients from delirium present to delirium absent, and 17 from delirium absent to delirium present. Thirty-two patients were classified as indeterminate and, therefore, were excluded from analyses (and two additional records were excluded because of an incomplete assessment that was interrupted for clinical reasons, and because of an error in which researchers completed the wrong part of the assessment resulting in an overall invalid assessment). Therefore, analyses were carried out for 785 patients (delirium present, n = 95, 12.1%; delirium absent, n = 690, 87.9%).

Descriptive data are presented in Table 5. Participants were tested in Edinburgh (n = 372: 4AT, n = 187; CAM, n = 185; could not assess CAM score, n = 3), Sheffield (n = 125: 4AT, n = 64; CAM, n = 61; missing 4AT, n = 2) and Bradford (n = 279: 4AT, n = 141; CAM, n = 138; could not assess, CAM n = 2; missing 4AT, n = 1). Only a minority of IQCODE questionnaires were completed (22%), mainly due to lack of availability of informants at the time of the assessments.

Primary objective

Secondary objectives

Resource use, costs and data collection

All significant resource consumption, giving special scrutiny to that which was thought to maybe differ in those diagnosed with delirium, was captured within the study, taking a NHS and Personal Social Services perspective. The main driver of the costs associated with patient care was expected to be the in-hospital time in the different wards. The time horizon of the base-case analysis for inclusion of costs was the 12-week period from randomisation.

The measurement of health-care resource consumption used only data collected as part of the trial. Patients’ use of hospital services was obtained from the Health Economics Questionnaire as entered into a section of the CRF, which had been completed via telephone call with the patient/carer and/or through scrutiny of the clinical notes 12 weeks after randomisation. The initial length of stay (length of stay for the initial hospital stay within the 12 weeks captured) was generated by using the day of randomisation and the discharge day directly from the clinical record.

Research nurses conducted a telephone interview, if this was thought of as necessary because health-care records were incomplete, with each of the patients or their legal proxy 12 weeks after randomisation. Thus, health-care activity and resource use were captured by research nurses based on a combination of scrutiny of the clinical notes and the telephone interview.

Methods for analysing and handling missing data

Observed means (95% CIs) of costs are presented for all patients. An analysis of the costs occurring during the 12 weeks is presented. Preparation of the data, assigning costs and all further calculations were performed in Microsoft Excel^® (Microsoft Corporation, Redmond, WA, USA).

To align with the population used for the main statistical analysis, patients with missing reference standard diagnostic information were omitted. Any other activity presumed to be missing was assumed equivalent to zero cost.

Unit cost assignment

All resource use was valued in monetary terms using appropriate standard UK unit costs assigned at the time of analysis. NHS Scotland Information Services Division (ISD) unit costs from 2015 to 2016 were used for inpatient/day-case, outpatient, A&E and other elective or non-elective hospital-based services in the 12 weeks after randomisation. 200 Unit costs for community health services were taken from Unit Costs of Health and Social Care 2016. 201

Cost calculation

Unit costs were assigned to inpatient stays specific to the admitting specialty. When details of the admitting specialty were missing, patients were manually sorted into the corresponding ward according to their medical diagnosis. In a first step, the inpatient, day-case, outpatient or rehabilitation costs were calculated so that all hospital-related costs could be summed. Subsequently, the costs occurring from the use of various community services (e.g. district nurse, GP, health visitors, psychological services, social workers, home carers, chiropodists, Meals on Wheels) were calculated and added as a separate cost considered in addition to the secondary care costs in order to generate the total cost per patient. Costs were calculated separately for:

• Scotland – national tariff costs (inpatient) were based on the ward days and whether the case was elective or emergency. As far as possible, all unit costs were taken from ISD200 and Personal Social Services Research Unit (PSSRU). 201

• England – in the absence of costs per hospital day for the different wards, the average day cost for inpatient stay, as well as day-cases, was used. For the outpatient costs, the NHS Reference Costs 2015–2016 were used. 202

Sensitivity analysis

In a second cost analysis, undertaken as a sensitivity analysis, the costs based solely on the initial length of stay were calculated based on the Unit Costs of Health and Social Care 2016201 for the initial stay, assuming a constant daily unit cost. Because the ward of the initial stay was unknown, the English and Scottish costs of the initial stay were calculated using the Unit Costs of Health and Social Care 2016 multiplied by the length of stay.

All health-care activity was costed irrespective of whether or not it was delirium related.

Scottish hospital costs

The Scottish costs taken from the CRFs, including every hospital stay and additional activity that was incurred during the 12 weeks after randomisation, were split into inpatient, day-case and outpatient costs (Table 11). The Scottish inpatient costs are costs per day in the different wards taken from the ISD Scottish Health Service Costs from 2015 to 2016. 200 In the details/assumptions column of Table 11 are the assumptions made for assigning the patients to different wards. This was necessary because the data set (from the CRFs) did not provide information on the exact nature of the wards, and because many of the wards are known to be general. In the case where two different wards were grouped into one (because there were unclear descriptions of where the patients had been), additional assumptions are described. For example, patients are often described as ‘renal’, which has been split into urology and nephrology according to NHS costing standard definitions. In these cases, the average cost of both ward types was calculated for inpatient per day, day cases and outpatient activities. In the frequent case of a missing allocation to a specific ward, the patients were allocated according to their medical diagnosis. This is an approximation that reflects not necessarily the geographical ward and associated costs, but the nature of the care that the patient is receiving according to their diagnosis. This care is often delivered in a general medical ward or a geriatric ward. The CRFs were completed by researchers on the study based on the follow-up telephone call and through the extraction of data from the electronic medical case notes. Given this combination, some details of derivations from the terms used are documented in the ‘details/assumptions’ column.

The costs of the initial hospital stay (unknown ward) were calculated separately using the same calculation for Scotland and England. A fixed cost per day was set201 and multiplied by the length of stay.

English hospital costs

The English hospital costs (Table 12) were handled in a similar manner to the Scottish costs, split as they were into inpatient, day-case and outpatient costs. However, one of the big differences is that the English inpatient costs were defined as costs per stay rather than per day, leading to very different values being displayed in the tables. The calculations incorporated from the initial stay, and including all stays from the CRFs, are based on the Unit Costs of Health and Social Care 2016. 201

Costs of tests in hospital

The cost of the different tests and imaging, which were included in the CRFs, are calculated separately using Scottish and English costs (see Appendix 16). The Scottish costs used the ISD Scottish Health Service Costs 2015–16,200 and the English costs used the NHS Reference Costs 2015–16. 202

Community health service costs

To calculate the full 12-week costs, the community NHS service costs were included. All community service costs were priced (see Appendix 17) according to the Unit Costs of Health and Social Care 2016. 201 When assumptions had to be made to estimate the cost per hour, these are stated in the column as ‘additional’. When feasible, the hourly costs for specific professions were assumed to be the same regardless of whether the patient was seen in a clinic or at home. If there were different values stated in the Unit Costs of Health and Social Care 2016, these values were taken accordingly. As the CRFs were often filled in precisely (stating 5, 10, 15 minutes), for the hourly calculation the price of an ‘hour of direct contact/per hour of face-to-face contact’ was chosen rather than an ‘hour worked’.

Costs of additional activities

To decide whether or not to cost all additional activities, the types and frequencies of additionally mentioned activities, such as ‘daily full package of care’, ‘housework help’ and ‘impact team’, were captured (see Appendix 18). The costs for these services were decided not to be formally calculated as such costs were very likely to be so small that they were insignificant. In addition, for some of those listed services, the free-text description recorded in the CRFs was insufficiently precise to be able to make proper assumptions about the service being referred to.

Cost analysis using original researcher delirium ascertainments

Cumulative costs are presented for the average per-patient costs for all patients with the original categorisation into delirium/no delirium as by the researchers. Because all patients were assumed to be treated according to the reference standard (because this was the information that had to be included in the case notes based on the protocol), it was not possible to assess different costs for true positives, true negatives, false positives and false negatives. This statement is not necessarily true for all patients, as some of them switched from the ‘delirium’ category to ‘no delirium’ and vice versa after the final central reference standard ascertainment process, by which point about 30 patients had been labelled as ‘indeterminate’. Therefore, the final costs are shown in two tables: first, the average cost of the patients as categorised by the researchers who carried out the initial assessment, and, second, the cost of the final delirium ascertainment, which was determined by analysis of the CRFs by the chief investigator and two colleagues (see Chapter 5).

Starting with the cumulative 12-week costs, as categorised by the local researchers, it can be seen that the cost of the initial inpatient stay (sensitivity analysis) for patients with delirium is more than double that for those without (Table 13). Taking into account the costs during the 12 weeks following randomisation, it can be seen that, using ISD (Scottish) costs, the costs of people with delirium are also more than one-third higher than the costs of those without delirium. 4 The same trend, although with lower magnitude, can be observed if the calculation is based on PSSRU costs (overall UK costs). 201

Cost analysis using final reference standard ascertainments

The costs, calculated after corrections, of the final reference standard ascertainments are slightly different, although they show the same trend as the costs based on the reference standard diagnosis as categorised by site (Table 14). The new indeterminate cost group (those patients for whom the diagnosis was considered uncertain based on the reference standard assessment) shows costs that are between those of the delirium and no delirium groups. This probably reflects the fact that patients in this group were a mix of those with delirium and those without delirium. The values from the initial stay (sensitivity analysis) again show a greater difference between patients with delirium and those without.

Model structure

In the basic structure, the model was a decision tree parameterised with the reference standard delirium and non-delirium probabilities, divided into branches dictated by the sensitivity and the specificity of the 4AT and the CAM (Figure 4).

FIGURE 4.

Decision tree. FN, false negative; FP, false positive; TN, true negative; TP, true positive.

In a next step, the model was parameterised with the costs and quality-adjusted life-years (QALYs) in order to calculate the incremental cost-effectiveness ratio (ICER). The utilisation and interpretation of the ICER outcomes is described hereafter. The ICER was calculated by dividing the difference in mean cost between the comparators by the difference in mean QALYs between those two groups:

in which C_i and E_i are the expected cost and effectiveness of the new intervention or the new diagnostic test (4AT), and C_c and E_c are the expected cost and effectiveness of the comparator strategy, which is usually the treatment or diagnostic test in place, in this case the CAM. Therefore, ΔC shows the incremental costs and ΔE represents the incremental effectiveness of the 4AT and the CAM.

If a new intervention is more effective and less costly, or if a new intervention is less effective and more costly, then the decision about whether or not to introduce the new intervention is straightforward, that is, the decision is clearly in favour of introducing it in the first scenario and against it in the second scenario. In these cases, the ICER calculation is playing a subordinate role. The ICER is important if we expect a trade-off between additional effect for additional cost and less effect for less cost. If a new intervention is more costly and more effective than the standard intervention, then the ICER is describing how much the intervention costs for an additional unit of benefit, in this study for a QALY. If it is cheap to gain an additional unit of benefit, it is still likely that the intervention will be introduced. In all cases in which the intervention is less costly and less effective, the value of the ICER represents how much money would be saved by renouncing a QALY. In this case it is the opposite way around, saying that if a small loss in benefits saves a considerate amount of resources it may be worth it. In a practical setting, this decision can be made based on the decision-maker’s willingness to pay, which varies between countries and interventions. In the UK, NICE adopts a willingness-to-pay threshold value of £20,000 per QALY. That means if a new intervention has an ICER value of < £20,000 per additional QALY or is worth > £20,000 saved per QALY lost, then it is likely to be considered a cost-effective use of NHS resources. Values outside these ranges are conventionally not considered cost-effective.

Model parameters

The costs of true-positive and true-negative patients, as well as the values for the sensitivity and specificity of the different tests, were taken from the within-trial analysis. The data for the costs can be found in Tables 11–14. All other values were gathered through a new survey of experts conducted for this study. This step was necessary, as a systematic review of the literature in order to identify parameters did not provide sufficient information. These values included mortality estimates, quality-of-life estimates and costs associated with delirium, particularly with respect to false-negative and false-positive diagnoses. Nevertheless, the literature search provided information that was used to construct an expert interview and the subsequent survey.

Reporting

The base-case results are reported using the Scottish cost estimates and the ‘best estimates’ from the expert elicitation. Alternative cost estimates were used in a sensitivity analysis, as were high and low estimates of parameters obtained from expert elicitation.

Methods for the sensitivity analysis

Three sets of sensitivity analyses were conducted to test the robustness of the results to alternative assumptions and to understand the uncertainty present in the estimates. First, a scenario analysis (an alternative to the base-case analysis with different assumptions or sets of parameter values) was conducted using English costs estimates instead of Scottish estimates. In addition, for both Scottish and English estimates, other parameter estimates were taken at the ‘highest estimates’ or ‘lowest estimates’ (from the expert elicitation), respectively. Second, a one-way sensitivity analysis was conducted in which the values of individual model parameters were varied over a range of ±25% of the base-case values. The differences in costs, and the differences in QALYs, between alternatives in each one-way sensitivity analysis are reported using tornado plots. Finally, a probabilistic sensitivity analysis was conducted to investigate uncertainty in the cost-effectiveness estimates due to sampling uncertainty of the parameter estimates. The probabilistic sensitivity analysis includes a gamma distribution for costs and beta distributions for probability and utility values and 1000 iterations. The results of the probabilistic sensitivity analysis are reported using a scatterplot on the cost-effectiveness plane and a cost-effectiveness acceptability curve.

Results from the health economic model

All of the results have the common problem seen with the economic evaluation of diagnostic tests, namely that the incremental costs and QALYs are very small and cluster close to zero (see Appendix 21). This makes the results of the cost-effectiveness analysis difficult to interpret, with wildly fluctuating ICERs. In general, the CAM produced slightly higher health-care costs than the 4AT, with only very small differences in QALYs.

The results of the base-case analysis show that the difference in costs between the 4AT and the CAM is £90.35, indicating that the 4AT resulted in lower total health-care costs over 12 weeks. The difference in QALYs was –0.00053, indicating slightly worse health outcomes using the 4AT than using the CAM over 12 weeks. The ICER of £170,533 means that, for each QALY lost, £170,533 would be gained in resources. Using conventional thresholds, the 4AT would be considered cost-effective relative to the CAM.

In the scenario analysis with English costs at the best estimate of parameters from expert elicitation, the difference in costs was –£61.52. The difference in QALYs remains the same as in the base case, yielding an ICER of £116,133. As in the base case, this would be considered cost-effective at conventional thresholds.

Scenario analysis using the lowest and highest estimates from the expert elicitation exercise produced different results. At the lowest estimates, the ICER was £24,289, which is in a range that may be considered cost-effective, depending on other factors. At the highest estimates, the 4AT dominates the CAM, being more effective and less expensive.

One-way sensitivity analysis

The results of the one-way sensitivity analysis indicate that in relation to differences between the 4AT and the CAM in total cost the model was most sensitive to the costs of false-negative classified cases, the specificity of both the 4AT and the CAM and the costs of true-negative classified cases. The model was less sensitive to the sensitivity of the 4AT and the CAM, the true proportions of positive and negative cases and other cost estimates.

In relation to differences between the 4AT and the CAM in QALY outcomes, the model was most sensitive to 4AT and CAM specificity and the QALY estimates for each of the four diagnostic classifications. The model was less sensitive to the sensitivity of the 4AT and the CAM and the true proportions of positive and negative cases (see Appendices 20 and 21).

Probabilistic sensitivity analysis

The results of the probabilistic sensitivity analysis indicate that there is considerable uncertainty in the estimated cost-effectiveness of the 4AT compared with the CAM. In Figure 11 (see Appendix 22), many points are spread across all four quadrants of the plane, indicating that there is the non-negligible probability that each alternative may be more or less costly and/or more or less effective. The cost-effectiveness acceptability curve (see Appendix 23, Figure 12) displays the probability that the 4AT would be considered a cost-effective alternative to the CAM at various willingness-to-pay thresholds for additional QALYs. The results indicate that the probability of cost-effectiveness is not greatly influenced by the threshold. This is because increasing the threshold makes scenarios in which the 4AT is more expensive but more effective more acceptable; however, it makes scenarios in which the 4AT is less effective but less expensive less acceptable. Scenarios in which one test dominates (more effective and less expensive) the other are unaffected by the threshold value.

Discussion of the cost analysis

Our main finding from the cost analysis was that delirium is associated with higher costs, taking into account the 12-week costs as well as those of the initial hospital stay. This is true if it is calculated using English costs as well as if it is calculated using Scottish costs. The latter showed a bigger effect; this could be because the average cost per stay was taken into account in the English costs, whereas the Scottish calculation was more detailed, being based on the single days that were estimated to be spent in the different specialties or receiving care for specific conditions.

Previous studies have also shown that a diagnosis of delirium confers a higher health-care cost than is observed in patients without delirium, for example as shown by Leslie et al. 21 in 2008, who looked at the 1-year financial costs of care related to delirium. The costs of delirium arise from multiple possible sources, including increased risk of falls, other medical complications such as pressure sores, increased staff time, new institutionalisation and increased length of stay. 4 There may also be substantial indirect costs, for example the effects on relatives and staff in addition to those on patients. 13 Additionally, delirium is associated with a higher risk of long-term cognitive impairment10,11,66 and other psychiatric comorbidities. 204,205 The prevention of delirium has been shown to be cost-effective. 4,206 However, the treatment of delirium is under-researched, so there remains a lack of unequivocal trial evidence that it is effective. Likewise, it is unknown if the prompt detection of delirium, leading to earlier treatment, is cost saving.

Because all patients were treated according to the reference standard diagnosis, it was not possible to split the cost analysis further than separating costs for people with delirium and costs for those without. Therefore, it was also not possible to define the actual costs for 4AT and CAM as well as split those into true positives, true negatives, false positives and false negatives. Another limitation of this study was that the local researcher reference standard diagnosis did not fully agree with the final reference standard diagnosis, but the original reference standard was the one recorded in the clinical case notes. A further issue to consider is that the reference standard process probably had a tendency to produce an earlier and more accurate diagnosis than is normal in standard care. In fact, owing to the underdetection of delirium observed in many studies,7 the study process possibly led to increased rates of delirium diagnosis in the recruited population.

Discussion of the cost-effectiveness analysis

To calculate the cost-effectiveness of the 4AT and the CAM, and to address some of the limitations from the cost analysis, a health economics model was conducted. From the cost analysis, it could have been hypothesised that the cost for false positives is likely to be lower than the cost for false negatives and that the latter is considerably higher than the cost for true positives. This assumption was confirmed by taking into account expert opinion in our model, with results showing the cost for a false-positive case to be £4653.00, whereas the cost for a false negative case is £8955.70. The additional costs resulting from a missed diagnosis of delirium (cost false negative – cost true positive) were estimated at £2066.70 in the base-case analysis.

In terms of QALYs, there is very little difference between the 4AT and the CAM. The ICER of > £170,000 is stating that, for each QALY lost, £170,000 could be saved, which could be spent elsewhere. Equivalently, it would not be cost-effective to switch from the 4AT to the CAM because the cost per additional QALY gained would be > £170,000, which is clearly above the NICE value of £20,000 per QALY gained. 207

This study had several limitations from an economic point of view, starting with relatively limited numbers that meant that there was insufficient power to analyse the economic outcomes. Additionally, there were no data available within the trial about quality of life and, therefore, most parameters rely on expert opinion. These expert parameters can, on the other hand, be seen as a strength of the study, as they included estimates of a diverse sample of experts, including specialists from geriatrics, critical care, general medicine, general surgery, psychiatry, cardiology, A&E and palliative care.

This cost-effectiveness analysis demonstrates the potential of the 4AT as a screening test for delirium detection. It also highlights the need for a better understanding of the clinical pathway of delirium, as a considerable part of this analysis had to rely on expert opinion. Improved detection of delirium may have the potential to produce large cost savings within the health-care system, but substantial research will be required to realise this potential.

As is common in the economic evaluation of diagnostic testing strategies, the clustering of incremental costs and incremental QALYs either side of zero leads to an unstable ICER that fluctuates wildly with small changes in the underlying model assumptions. In addition, there is clear potential for screening tools, such as the 4AT, to be cost-effective; large incremental changes resulting in a clearly cost-effective strategy may rely on a more consistently demonstrated impact on the clinical pathway. Linkage of a 4AT result to a clearly defined treatment protocol that contains a clearly demonstrated effect on longer-term clinical and cost outcomes may be a necessary prerequisite to very high-value care.

Part 1

When respondents were asked how much knowledge of delirium is necessary for health-care practitioners to use the 4AT effectively, 7.14% (6/84) said ‘none/very little’, 58.33% (49/84) said ‘some’ or ‘a moderate amount’ and 34.52% (29/84) said ‘quite a bit’ or ‘an extensive amount’.

Part 2

When respondents were asked how much training in the use of the 4AT is necessary for health-care practitioners to use the tool effectively, 16.67% (14/84) stated ‘none/very little’, 59.52% (50/84) stated ‘some’ or ‘a moderate amount’ and 23.81% (20/84) stated ‘quite a bit’ or ‘an extensive amount’. Although the 4AT was designed to require minimal knowledge of delirium and training, these results demonstrate that many 4AT users believe that at least moderate levels of both are necessary for it to be used effectively.

Alertness

I’m not trying to be overly pedantic but altered level of alertness in normal waking hours, yes it could well be delirium; but altered alertness for someone coming in to A&E in the middle of the night having fallen . . . we need to be careful about overinterpretation.

Junior doctor, Care of older people ward

Just by knowing the patient, you would know if someone were more drowsy; you’d notice it naturally.

Staff nurse, Care of older people ward

Attention

Date of birth might not always be the best one. There are cultural issues. Elderly Asian men and women may not have a formal record of their date of birth. Isn’t there a similar sort of question to use instead?

Senior nurse, Care of older people ward

Acute change or fluctuating course

That’s not difficult when you know their baseline – these are the essential ones. Easy when you know the patient.

Junior doctor, Surgical ward

That would be really difficult if you’ve got someone who is confused in front of you and there is no one who knows the person with them. It might take time to find out. It’s not the question as such; it’s being able to find the answer. If you don’t have that information to hand how would you score it. What would you do? That’s the critical question for delirium.

Junior doctor, ED

We do drum it into staff that they are allowed to ring GPs and they are allowed to ring relatives and the care homes to find out what the patient is normally like, and get old notes. So there’s lots of different ways you can try and find out if this is a patient’s normal state or whether this is a significant change.

Consultant, ED