Care home residents' quality of life and its association with CQC ratings and workforce issues: the MiCareHQ mixed-methods study

Adult Social Care Outcomes Toolkit scoring

Three scores can be derived from the ASCOT tools: current SCRQoL, expected SCRQoL and SCRQoL gain. All ASCOT tools measure current SCRQoL, which asks about the person’s situation now. The interview and mixed-methods tools also ask what the situation would be if services were not in place to support the person and nobody else stepped in. We call this expected SCRQoL. By taking the expected SCRQoL scores away from the current SCRQoL scores, we can estimate the impact that the service(s) are having on the person’s quality of life, which we call SCRQoL gain:

As ASCOT is a preference-weighted measure, responses to each domain for current and expected SCRQoL are weighted to reflect English population preferences and then entered into an algorithm (Equation 2) to calculate a score ranging from − 0.17 to 1:13

Scores of 1 represent optimum or ‘ideal’ SCRQoL and scores of 0 indicate a state that, according to the preferences exhibited by the general population, is equivalent to being dead. Negative scores indicate a state worse than being dead. 13

Psychometric properties of the Adult Social Care Outcomes Toolkit

Psychometric testing has consistently revealed that ASCOT has acceptable internal reliability. 13,15 Research with a wide range of social care user populations, including older adults, younger adults and adults with physical and sensory impairments, has established its validity, feasibility and reliability. 13,15,16

Expected SCRQoL, which is still relatively uncommon in quality-of-life research, has been shown to be a reliable indicator of social care need, as it is highly correlated with functional ability. 13,17 Recent work exploring the feasibility and validity of the expected SCRQoL questions found the approach both feasible and valid with a sample of social care users, with many people saying that they found the questions easy or very easy to answer. 17 Nonetheless, it is not without its limitations. In particular, the expected SCRQoL questions are not suitable for people with limited or low cognition. 17 Given that up to 80% of care home residents18 are believed to be living with dementia, these questions are entirely unsuitable for use in interviews with most care home residents.

ASCOT-CH4 and the social care-related quality of life of care home residents

To measure the impact of services in this population, we needed to employ a mixed-methods approach. The mixed-methods version for use in care homes is called ASCOT-CH4 (Adult Social Care Outcomes Toolkit Care Homes-four levels; herein referred to as CH4) and was designed for residents who cannot self-report. Methodologically, CH4 involves structured observations of residents, alongside a staff proxy interview (for each resident; see Report Supplementary Material 12), a resident interview (to the extent that they are able; see Report Supplementary Material 11, 13 and 14) and a family proxy interview (where possible; see Report Supplementary Material 12). (See Appendix 1 for a detailed account of this method.) The mixed-methods approach has been found to have excellent face validity and inter-rater reliability. 19

Previous studies taking this approach19–22 have found that residents’ needs are mostly met in the ‘basic’ domains (safety, accommodation, food and drink and personal cleanliness). However, there is evidence of unmet needs in the higher-order domains (Control over daily life, Social participation and engagement in meaningful Occupation), with some residents spending long periods of time disengaged. Staff recognise the importance of being occupied and staying socially active, but note that many residents decline to take part in organised activities or appear too tired and withdrawn. 21 The impact of skilled care and support to help and encourage people to remain active and engaged should not be underestimated. 23,24 However, at the same time, we should not ignore the possible contribution of unmet needs to other areas, including aspects of residents’ health-related quality of life (HRQoL). For example, unrecognised depression, anxiety or pain may lead to unmet needs in these domains.

Pain

Studies estimating the prevalence of pain among the care home population suggest that it is common. Indeed, this is the case for all groups of older people because they experience higher levels of chronic disease than younger age groups. 25 The actual pain prevalence estimates for resident populations vary considerably, although the usual estimate is between 30% and 60%. 26–39 A recent large-scale study using validated pain scales in England suggested that the prevalence of pain in this population is around 35%. 33

Although at least one-third of residents live with and experience pain, the consensus is that pain is often under-recognised and undertreated in care homes. 40–43 This is likely to be attributable to a variety of causal factors, including resource considerations and the attitudes of health-care professionals. 44 However, difficulty in detecting and assessing pain is one of the most widely cited reasons for this under-treatment. 44–46 Pain is a subjective experience47,48 and cannot be measured adequately using an objective diagnostic test, which is why mainstream approaches rely on self-reporting. As McCaffery48 notes:

Pain is whatever the experiencing person says it is, existing whenever he says it does.

McCaffery48

There are, however, a number of challenges to enabling care home residents to self-report and share their subjective experiences of pain. Although impairments more common in later life, such as hearing and sight loss, do present challenges,49 the key challenge to measuring these experiences is that a high proportion of residents live with dementia. 18 The characteristics of the group of conditions classified as dementia mean that the ability to communicate experience may be limited, particularly for those living with more severe dementia, and, thus, either research or diagnostic methods that rely on this may not be particularly appropriate for this group. 46,50–54 It is not, however, just the ability to communicate clearly that is required for accurate self-report. The resident must understand the request, be able to recall pain events – often in a given time frame – and interpret that internal experience with reference to an external framework. 55 All of these can be challenging for people living with dementia.

There is evidence that pain is under-recognised among residents living with dementia in studies comparing the numbers of older adults in care home settings who receive pain medication. Morrison and Sui56 found that, following hip fractures, cognitively intact older people received three times the level of medication received by older people living with dementia. This finding was echoed in studies conducted in Canada57 and the USA,58 both of which concluded that residents living with dementia in nursing homes were less likely to receive pain medication than residents who did not have dementia. This has also been found in other contexts, such as in the treatment of scabies. 59

Pain is associated with decreased functional ability, lower levels of activity and socialising, higher levels of falls, reduced appetite, poorer sleep, greater levels of irritability, aggression, anxiety and depression, and resistance to care. 25,60–66 Unsurprisingly, untreated pain has been found to be associated with poorer overall quality of life. 30,33,64,67,68 Specifically, for older adults living with dementia, untreated pain has been shown to exacerbate the symptoms of dementia, such as impaired cognition. 30,64,69,70

Depression and anxiety

Estimates in studies looking at the prevalence of mental health conditions suggest that large numbers of residents live with a mental health condition. The most common of these conditions is depression. 71–73 However, as with pain, estimates vary, complicated by differences in how studies define and measure depression. At the lower end is an international prevalence estimate of just under 10%; at the upper end this is just over 50%. 74–81 Considering specifically residents with a diagnosis of dementia, recent work in the UK has estimated that the prevalence of depression among this group is at the middle and lower end of this range. One study has estimated that 26.5% of older people living in residential care and 29.6% of older people residing in nursing homes experience depression,82 and another study places this figure at around 10% of the care home population. 77

Anxiety in later life has historically received less attention than depression, despite being quite common. 83–85 A systematic review in 201075 found only three studies that estimated anxiety among older adult care home residents. A review in 2016 found 18 studies,86 a number that, although still very small, suggests that studies focusing on anxiety in care homes are becoming more commonplace. The reviews reported the prevalence of anxiety disorders among residents as somewhere between 3.2% and 20%. However, clinically significant anxiety symptoms were found to be more widespread, with prevalence across studies ranging from 6.5% to 58.4%.

Living with anxiety and/or depression can have a range of negative impacts on an older person’s life, including lower quality of life,68,87–89 lower sociability and greater loneliness,90 and poorer health, including increased pain. 91–95 Anxiety and depression have also been linked to sleep problems and smoking,84,96,97 loss of independence and functional ability,87,98–103 cognitive decline,104–107 behavioural problems,94,108 suicide109 and higher mortality levels. 110–112

Even though the impacts of living with depression and anxiety in later life are, as outlined above, well documented, and the conditions themselves are often treatable,113,114 there is a large body of evidence suggesting that depression, in particular, is both under-recognised and undertreated or poorly managed, particularly in residents who live with dementia. 107,115–122 Less evidence exists around the under-recognition and undertreatment of anxiety in care home residents, but, given the lack of attention that anxiety has received in this setting,86 it is likely that both of these are very common.

As with pain, the assessment of depression and anxiety in care homes is made more challenging by the characteristics and symptoms of dementia. 114,123 However, there are other challenges, including a lack of suitable tools124 (as many of the tools available have been developed for younger populations), lack of training and knowledge on depression and anxiety among care home staff115,125 and the difficulty in isolating symptoms of depression and anxiety from other conditions and somatic complaints. 126

A new approach to measuring pain, anxiety and depression in care homes

Although measures of pain, anxiety or depression have been used with older people and those living in care homes, unlike the ASCOT for SCRQoL, none of them aims to measure the impact of social care services. Existing measures focus on the person’s current situation or diagnosing a condition without any reference to input from services or support and without a mechanism for attributing improvements or change in outcomes to those services. There is also no evidence to suggest that tools have attempted to use innovative methods or adapted forms of communication to support the inclusion of people with cognitive or communication difficulties. This is important in care homes, where many residents live with dementia and have physical and sensory impairments that make self-completion or self-report challenging or impossible19,127 and places care home residents at risk of living with unmet needs in both their health and social care-related quality of life. 87,128–130

Work package 1: measuring health and social care-related quality of life

Work package 2: psychometric testing using the mixed-methods ASCOT with new health-related measures

Work package 3: care home quality, resident outcomes and care workforce

Public involvement and engagement strategy

Public involvement and engagement was an integral part of the study from the design stage, when the application was reviewed by five members of the public from the Personal Social Services Research Unit’s (PSSRU) Research Advisors Panel. During the study, patient and public involvement (PPI) was delivered in the following ways:

1. Study Steering Committee – two members of the public with experience of social care were recruited as lay representatives to the Study Steering Committee. One was a partly retired care home manager and the other was an informal carer whose relative had lived in a care home. Their role on the Study Steering Committee was to attend meetings, give a public/patient perspective on issues that arose and comment on emerging findings.

2. Co-researchers in WP1 – we recruited three lay co-researchers from the PSSRU’s Research Advisors Panel to assist with the focus groups with staff and contribute to the development of the new measures of pain, anxiety and depression. This is described in detail in Chapter 3.

3. Contributions to study outputs – lay advisors from the Study Steering Group contributed to the drafting of the Plain English summary of this report and reviewed it for written quality and clarity. The co-researchers from WP1 co-authored Chapter 7.

Ethics approvals

Ethics approval was required for the primary data collection undertaken in WP1 and WP2. WP3 drew on both the primary data collection undertaken in WP2 and publicly available data sets. The analysis of publicly available data received a favourable ethics approval by the University of Kent (SRCEA 201).

The primary data collection received a favourable opinion for ethics approval from the Health Research Authority, which governs research ethics in the UK (reference 18/LO/0657). As this was a social care study, approval was sought and received from the Association of Directors of Social Services (ADASS) and by each local authority involved in the study.

The main ethical issues were around participant consent and the participant observations being conducted in communal areas of the care home.

Participant consent

The study consent procedures complied with the Mental Capacity Act155 and Code of Practice. 156 Where residents lacked the capacity to consent, personal consultees were asked for their advice about including the person in the research. Thereafter, consent was treated as a continuous process, with researchers being sensitive to both verbal and non-verbal indications of residents’ unwillingness to participate in the research.

Participant observations

Observations were carried out as unobtrusively as possible in communal areas of the homes (e.g. lounge, dining area, activity areas, gardens and corridors). Informed written consent was sought only from study participants who were the subject of the observations and interviews. As care homes are communal environments, non-participatory residents, staff and visitors were present during the observations. To ensure that observations were conducted with the full knowledge of those living and working in the home, and to confirm that people were happy with our presence, we asked homes to display posters raising awareness of the research, with the names and photographs of the research team, 2–3 weeks before data collection. On the day of fieldwork we were also given a tour of the home and introduced to those present. At the end of the observational period, we said goodbye to those present and notified the staff and manager that we had stopped observing.

Search strategy

We searched five databases {MEDLINE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), the Cochrane Library, PsycInfo^® [American Psychological Association, Washington, DC, USA; via EBSCOhost (EBSCO Information Services, Ipswich, MA, USA)] and Abstracts in Social Gerontology}. The searches for the review focusing on pain tools were conducted between mid-November 2017 and mid-December 2017. The searches for the review looking at anxiety and depression tools were carried out in January 2018. We used a combination of search terms related to our population and constructs of interest (see Appendix 2 for the full search strategy). Owing to the large number of papers on pain, anxiety and depression, we focused on original and review articles published in the previous 10 years (from 2007), which tested or reviewed more than one tool with older adults. Only studies published in English were included.

In both reviews, following the literature search and tool identification outlined below, data were extracted on each tool and used to populate a spreadsheet. The data extracted were organised around the following headings, which related to characteristics of the tool:

• review article identifying the tool

• name of tool

• original article outlining tool

• availability of the tool

• main population designed/tested with

• other populations used with

• aspect(s)/dimensions of pain measured

• method(s) of data collection

• details of measurement scale/rating/scoring

• details of development and testing

• other (methodological issues, reflections on the tool).

The data were then summarised in chart form (see Appendix 2, Tables 21 and 23).

Pain tools rapid review

The search strategy (see Appendix 2) yielded a total of 2167 papers. Following a review of the titles, 1865 of these papers were rejected. When duplicates were removed, there were 196 unique papers.

The title and abstract of each of the 196 papers were reviewed for relevance and adherence to the inclusion criteria (see Appendix 2). Twenty-six papers were included in the final rapid review, the majority of papers being rejected as they focused on a single tool (Figure 1). A total of 22 tools for measuring pain in older people, people living in care homes and people living with dementia were identified from these final 26 papers. The identified pain tools are presented in Table 3. Further details of the key characteristics of these tools can be found in Appendix 2, Table 21.

FIGURE 1.

The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)160 flow diagram of pain tools search.

The tools identified in the review can be categorised by mode of data collection. Three different modes were identified among the tools: self-report, proxy report and observation. Most tools tend to use just one mode of data collection, although the interRAI Long-Term Care Facilities tool161 uses all three. However, it should be noted that many of the observational tools include verbal manifestations of pain, such as saying ‘it hurts’, in their evidence for rating the presence or intensity of pain, for example negative vocalisations in the Pain Assessment in Advanced Dementia (PAINAD)162 or ‘vocal complaints: verbal’ in the Checklist of Nonverbal Pain Indicators (CNPI). 163

Ten tools identified in the review use self-report methods and have been tested or validated for use with older people, people living in care homes or people living with dementia. These tools were usually not developed specifically for older people or people living with dementia and other cognitive impairments. Instead they tend to be tools developed for use either with the general population or with children, such as the Faces Pain Scale (FPS). 164 The self-report tools usually, although not exclusively, focus on the measurement of pain intensity and use a range of approaches to facilitate self-report, including words, numbers and visual representations of pain. Often these different methods are combined in a single tool. For example, the 21-Point Box Scale uses a number scale anchored by words at either end165 and the Iowa Pain Thermometer166 uses a visual representation for pain alongside words to add further clarity to its representation of pain intensity.

The pain tools use a variety of words to describe different pain states. For example, the McGill Pain Questionnaire167 begins with ‘no pain’ and follows with words that represent increased intensity of pain (mild, discomforting, distressing, horrible . . . excruciating). Similar approaches are used in other scales, such as the various versions of the Verbal Descriptive Scale, including the Verbal Descriptive Scale (pain thermometer),166 on which the six-point scale ranges from ‘no pain’ through increasing severity (mild, moderate, severe) up to ‘as bad as it could be’. Words are also used in tools and sections of tools that focus on aspects of pain beyond intensity, such as the experience of pain (e.g. McGill Pain Questionnaire167). Numbers are used by some tools to represent pain intensity, with different scales using sets of numbers. Different versions of the Numeric Pain Rating Scale168 use different scales. The most common was 0–10 or 0–100 points, but other numerical scales are also used (e.g. the 21-Point Box Scale165). On all of the scales, greater intensity of pain is represented by a higher number.

As noted above, visual methods are also employed to represent and capture the intensity of a person’s pain. For example, many of the numerical scales are presented visually on a scale resembling a ruler. Other visual methods include the Visual Analogue Scale and the aforementioned pain thermometer or the Colour Analog Scale,169 on which visually greater levels of pain are represented by the top of the thermometer, where the colouring is more intense. Another form of visual representation used in pain tools is faces. The various versions of the FPS164,170,171 use faces with different expressions to represent different pain states.

Because of the subjective nature of pain, self-report is, quite understandably, viewed as the ‘gold standard’ of pain measurement and diagnosis. 65,172 However, high levels of cognitive impairment in residents of older adult care homes,18 and the associated challenges of recognising and reporting pain, have often demonstrated that self-report pain scales, and, indeed, self-report scales more generally, are inappropriate for use in long-term care settings. 55,173–178 Thus, the focus of pain measurement tools specifically for older people and especially for older people living with dementia has moved to observational methods.

Most reviews of observational-based pain tools agree that, although self-report is preferable, where people are not able to reliably self-report, observational methods are the most appropriate55 approach to the measurement or diagnosis of pain. 46,55,70,179 Moreover, specific observational pain tools are recommended by professional bodies. For example, the Abbey Pain Scale (APS)180 is recommended in the UK by the Royal College of Physicians, the British Pain Society and the British Geriatrics Society. 45 We found 11 pain tools that use observation to investigate, measure or diagnose pain among older adults, people living in older adult care homes and people living with dementia.

Unlike the majority of the self-report tools, these tools were almost exclusively developed for people living with dementia. The most common approach employed by the observational pain tools in this review is the identification of pain behaviours, which in turn populate a pain intensity score, such as in the PAINAD,162 where pain is scored on a range of 0–10. In this tool, 1–3 is a reflection of mild pain, 4–6 is a reflection of moderate pain and 7–10 is a reflection of severe pain. Some tools measure other aspects of pain. For example, the CNPI163 focuses on the presence of pain, and one of the earliest observational tools, the Discomfort Scale in Dementia of the Alzheimer’s Type (DS-DAT),181 measures the frequency and duration of pain. The Mobilisation–Observation–Behaviour–Intensity–Dementia-2 Pain Scale (MOBID-2)182 looks at the location of pain in addition to its intensity. In some tools, such as the APS180 and Doloplus-2,183 the severity behaviours themselves are also rated. The pain behaviour items in the observational tools found in this review are presented in Appendix 2, Table 22.

Many of the observational-based tools in the review were designed to be administered by care workers, nursing assistants, nurses and health professionals rather than by researchers. This is often reflected in the way the tools were designed to be used. Nearly all are short tools – with the exception of the 60-item Pain Assessment Checklist for Seniors with Limited Ability to Communicate (PACSLAC)184 – and developers claim that they can be administered in under 5 minutes, and, in the case of the APS, about 1 minute. 180 However, many of these tools were not designed to provide a one-off snapshot of the pain experienced by a resident. For example, the APS and the Doloplus-2 were designed to be used on multiple occasions to understand a person’s pain. Other tools were also clearly designed for use in practice, as they stipulate that the gathering and rating of pain behaviours should occur during activities. The MOBID-2, for example, sets out five different movements that should be carried out by the person being assessed. These include turning to both sides in bed and opening both hands, one at a time. A different pain assessment is made for each movement. Similarly, the Non-Communicative Patient’s Pain Assessment Instrument (NOPPAIN)185 was designed to be used to assess pain during a range of activities, including dressing and bathing.

A number of the tools listed above have been widely used and are well tested and validated. In particular, the APS, the Doloplus-2, the PACSLAC and the PAINAD tend to perform best in validity, reliability and psychometric testing. 70,172,178

There are, however, a number of caveats and concerns around the use of observational pain measures. One of the most salient concerns about using observational means to assess pain is that they focus inherently on behaviours. Although the behaviours used as indicators of pain are clearly actions and behaviours that people might engage in when in pain, it has been pointed out that many can also be indicative of other states, such as hunger, thirst, delirium, depression or anxiety. 46,55,186 There are also concerns that an observational-based pain tool cannot assess the intensity of a person’s pain, only its presence. 185,187 A number of studies have suggested, however, that this concern has been overstated and note the high level of agreement on pain intensity between self-report and observer-rated pain scores. 70,179,188 In the light of these concerns, many reviews of observational-based pain tools have concluded that best practice should involve attempts to elicit self-report, supplemented, where needed, by observation and other methods to assess pain in older people with cognitive impairments, such as physical examination, medical history and reports from family members. 46,65

Stage 1: desk-based drafting of tools for each item

Specifically, the aim of this stage was to develop the following draft tools for each item:

• resident self-report questions and response options focused on the resident’s current frequency of pain, anxiety and depression

• proxy questions and response options asking staff and family members about the resident’s ‘current’ and ‘expected’ (see Chapter 1) frequency of pain, anxiety and depression

• initial observational guidance for each domain, drawing on the observational markers outlined in the tools.

Stage 2: focus groups with care home staff

The aims of the focus groups were to:

• understand how care home staff recognise when residents are in pain or are feeling anxious or depressed

• find out which words care home staff use to describe residents’ anxiety, pain and depression.

In addition, the groups were used to provide an initial review of the proxy questions.

Stage 3: cognitive interviews with staff and family members

The aims of the cognitive interviews were to improve, revise and finalise the item questions and response options for the interviews to be used in the data collection outlined in Chapter 4.

This chapter presents each stage of work separately, with its own methods and results.

Methods

Informed by the reviews, the research team created draft items for each of the three domains. For the observational guidance, the observable indicators of pain, anxiety and depression identified in the two reviews were collated. The initial draft was just a list of these markers (see Appendix 2, Tables 22 and 24).

For the self-report and proxy questions, a number of different question options were developed. The process involved thinking about frequency statements that reflected the four outcome states of the ASCOT: ideal state, no (unmet) needs, some (unmet) needs and high (unmet) needs (see Chapter 1). A number of different options were created (see Appendix 3, Box 1, for examples). In this stage of the development, the focus was on formulating questions to understand what the resident’s experience was now, while they were living in the care home (with help and support in place). Using ASCOT terminology, these would be the questions relating to ‘current’ frequency of pain, anxiety or depression (see Chapter 1). These were reviewed at a research team meeting, and then revised and reviewed by the team again.

Results

The first team review considered a number of possible question options (see Appendix 3, Box 1, for examples). This led to further refinement of question options and revision of those that remained. Decisions following the team review are outlined below.

Methods

Focus groups were conducted with care home staff to elicit a range of views and uncover collective views and meanings. They have been used previously in the development of ASCOT and other research tools and questionnaires. 219–221 Four older adult care homes were recruited to the study, the aim being to hold one focus group in each home. Located in South East England, all of the homes were run by a large national chain and were registered for older adult residential and nursing care. Recruitment to the individual groups was administered by the care home managers, who received guidance from the research team. Managers were asked to recruit between five and eight people to each group222 and to choose a range of staff who had interacted directly with residents, such as care, nursing and activity staff. Some homes invited other employees, such as maintenance staff and receptionists, to the groups as they felt that all staff in the home helped, supported and knew the residents.

The focus groups themselves were dual-moderator groups,223 with the administration shared between an academic researcher and a PPI co-researcher. As well as co-moderating the focus groups, the three project PPI co-researchers were involved in planning the focus groups and analysing the data from the focus group sessions. More information about the PPI co-research can be found in Chapter 7.

The groups consisted of an introduction and three substantive sections reflecting the aims of the group (the focus group topic guide can be found in Appendix 3, Box 3):

• how staff recognise when residents are in pain, or are feeling anxious or depressed

• the words staff use to describe residents’ anxiety, pain and depression

• a review of two draft proxy questions on pain, anxiety and depression.

The academic researcher led the introduction and the final section testing the draft questions, and the PPI co-researcher led the first two substantive sections of the groups. In the findings below, a PPI co-researcher is identified as mod1 and the academic researcher is identified as mod2. With the agreement of those present, all groups were audio-recorded and fully transcribed. The focus groups’ data were organised and analysed using NVivo 12 (QSR International, Warrington, UK). Coding was carried out by a single researcher using a primarily deductive thematic analysis224 focused on the need to develop research tools from this work. However, all PPI co-researchers and another member of the research team familiarised themselves with the focus group transcripts and met to discuss the findings and how these might feed into the development of the pain, anxiety and depression domains. They also helped with generating and reviewing themes and introduced ideas that had come more inductively from the data. The whole research team met to further review the themes and to discuss the findings of this section of the study and how these fed into the development of new ASCOT domains.

The sample

During November and December 2018, focus groups were carried out in three of the four care homes. The fourth home withdrew from the study owing to issues beyond the control of the project. Across the three groups, 22 staff from the three homes participated. Table 5 outlines some basic demographic and role information about the focus groups participants. Although the majority of participants were those who provided direct social or health care, the groups also included some home managers and others who supported the residents of the home.

Findings

Methods

Cognitive interviewing, drawing on experimental psychology, is increasingly seen as a way of improving the quality and accuracy of survey tools. 225–227 It has also been used to help inform the development of other ASCOT tools. 219,228–231 Cognitive interviews are an attempt to use qualitative interview methods and evidence to critically evaluate whether or not a survey question ‘fulfils its intended purpose’. 226 This is achieved by conducting an interview where the focus is on the thought processes involved in answering a survey question rather than, as is usual, on the answer provided. More specifically, the cognitive interview may explore people’s comprehension, recall, judgement and estimation, and response processes for a given question. 232 The cognitive interviewee is presented with a survey question and, through the use of two techniques, probing and thinking aloud,226,227,233 the aim of the interview is to establish the thought processes that occur during the answer of the survey question. Thinking aloud is a technique whereby the person is asked to verbalise their thoughts while attempting to answer a question. The other technique, probing, involves the interviewer actively asking specific questions about the thought processes behind answering a question. Examples of probing in a cognitive interview could include asking the participant why they chose a specific answer, or how they interpreted certain phrases or words in a question. Both techniques can be employed in the same interview; for example, thinking aloud can be followed up by probing to explore aspects of the thought process that the participant did not verbalise.

During January and February 2019, 37 cognitive interviews were conducted with both care home staff and relatives/friends of care home residents. They were organised into three phases, with reflection on the previous phase informing revisions to the draft questions subsequently tested in the next phase. Although the final tool also included structured questions that could be used with care home residents themselves, we did not include care home residents in this phase of testing. It was felt that the cognitive burden associated with participating in a cognitive interview, particularly thinking aloud,226 would be too great for many older adult care home residents. Staff and relatives were recruited from four care homes in South East England and given a gift voucher to thank them for their time. One-to-one interviews were carried out in a quiet and private room by one of three researchers.

There were three iterations, or phases, of cognitive testing. The first two focused on testing the ‘current’ questions (see Appendix 3, Boxes 2 and 4) and the final phase included the ‘expected’ equivalent for each item (see Appendix 3, Box 5). Each cognitive interview tested six questions. Phase 1 interviews tested two versions of the four-point response option scale for each item developed from the desk-based review. In phase 2, one of the four-point scales was replaced with a three-point scale, described in detail below. In the third and final phase, we tested one four-point ‘current’ scale and the equivalent ‘expected’ scale for each item and finalised the wording for piloting.

Question ordering was varied to reduce bias in feedback due to ordering effects. Staff were asked to think of a person they supported when answering the test questions, whereas relatives were asked to think about their family member while answering the questions. An example of the phase 2 staff interview schedule can be found in Appendix 3, Box 6.

Each of the interviews was audio-recorded and transcribed verbatim. NVivo 12 was used to organise and analyse the data. Although cognitive testing is now one of the most common ways to pre-test survey questions, there is no agreement on how data created in the cognitive interview should be analysed. 227,234,235 There are some formal methods that aim to systemise the analysis of cognitive interviews, but analysis in these modes often requires a great deal of time and resource. 235 They also often take a very theoretical approach, drawing directly on the thought processes that lie behind answering a question. Willis’227 review of these more formal methods suggests that adopting them is not justified given the time and resources they require. Our analysis also rejected these formal methods, not only because of resource and time implications, but because our interest in the cognitive interviewing data was in how they could inform the development of our new HRQoL questions, rather than to explore the cognitive processes in depth. The analysis and subsequent inductive coding took a practical approach and focused on identifying issues and problems with the survey questions and how interview participants understood and interpreted the questions. This was carried out by a single researcher, but analysis and findings were shared with the project team in a series of meetings.

The sample

In total, 21 staff and 16 relatives of residents participated in the cognitive interviews. All but one of the staff members were white, and most were female (with one male staff member and four male family members). Participants were spread across the adult age range, but, overall, relatives tended to be > 55 years old, whereas staff were often younger.

Findings

To understand both how participants responded to and interpreted questions and how this informed the development of the questions across the three stages of interviews, this section will focus on two issues: problems with and changes to question stems, and problems with and changes to answer response scales.

Problems with and changes to the question stems

Although the interviews in phase 1 tested two different questions for each HRQoL domain, the two questions shared the same question stem:

• Which of the following statements best describes how often you think [the resident/your relative] is in pain?

• Which of the following statements best describes how often you think [the resident/your relative] feels worried or anxious?

• Which of the following statements best describes how often you think [the resident/your relative] feels down or depressed?

The main finding was that, rather like in the focus groups, most participants interpreted the questions in a way that was consistent with our intention. This was demonstrated in two different ways: first, when participants explained, by thinking aloud as they answered a question or when probed by the interviewer, why they had chosen an answer option, and, second, by responding to probes from the interviewer about their understanding of the question and the concepts it contained.

An example of the first way in which participants demonstrated an understanding of the question is a staff member thinking aloud while attempting to answer one of the anxiety questions. In the following quotation, they think through some anxiety behaviours they have seen in the person they are answering the question about. This demonstrates their understanding of our anxiety question and how it matches what we intended when developing it:

So, if you can tell me what’s going through your head right now.

I’m trying to think, would that be anxious? Yes, it would be anxious ‘cause they’re constantly–, they’re worried about the time of breakfast, and lunch, and if they’ve got to go out, and what time bed is. Is it time for them to get up? So, they’re constantly asking the time because they’re worried in case they might miss breakfast.

Care home staff 02

The following example reflects the second way of asking questions in a cognitive interview: responding to interviewer probes about the meaning of the question or terms used in the question. This example relates to a pain question, and the interviewer asks the participant how they interpreted the term ‘pain’ in the question:

What does the term pain mean to you?

A physical pain. When you’ve hurt yourself or a headache or a stomach pain, something like that.

Relative 09

For pain in particular, testing revealed the importance of helping respondents focus on rating ‘current’ frequency of symptoms with symptom ‘management’ in place (e.g. staff giving medication). Participants' responses suggested that their understanding of the term ‘well managed’ reflected our intended meaning:

Well managed, erm, whether the correct amount of medication has been given. That if there’s exercises that’s going to help it that they’re being dealt with, you know. Well, that’s what I would call being well managed.

Relative 05

Moreover, some participants demonstrated an awareness of the interaction between management of and experience of pain:

Yes. I’m kind of assuming well managed, if it was well managed she wouldn’t be in a lot of pain most of the time. So, because she’s sometimes in pain it’s not terribly well managed. Sorry, that doesn’t make sense does it?

That’s OK.

If it’s well managed, she shouldn’t be in pain very much at all. If it’s not well managed at all then she’d be in a lot of pain, but because it’s kind of well managed sometimes she’s sometimes in pain. So, I think that’s quite a difficult one. But it also depends on the intervention of somebody else.

Relative 02

Whereas participants’ understanding of the anxiety and pain questions broadly reflected our intentions when developing them, participants’ understanding of the depression question did not always align with our intended meaning. As outlined in Chapter 2, the new items developed in this study were designed to sit alongside the ASCOT-CH4, a quality-of-life measure, rather than a clinical or diagnostic tool. The intention was to develop an item that reflected generally feeling down, feeling depressed or having a low mood, as opposed to diagnosing or screening for clinical depression. The question used in the focus groups and the first phase of the cognitive testing avoided directly mentioning depression and instead talked about feeling down or depressed (see Appendix 3, Box 2). Although all participants were able to provide an answer and understood the meaning of individual terms used in the question, a number struggled with the use of both ‘feeling down’ and ‘feeling depressed’ in the same question. They found the use of the two terms together contradictory, as these were perceived as different points on a spectrum of feelings:

And what do you think the question means by the terms feeling down or depressed?

Well down, just having a low day, got out of the wrong side of the bed, don’t know why you feel down, you just, you know, it might be the weather, could be a host of different things that make you feel down.

Uh-huh. What about depressed?

Depressed? That’s a different kettle of fish, isn’t it, depression?

Relative 07

Where this was the case, participants usually gave an answer based on ‘feeling down’, as they felt that this was most appropriate to the experience of the resident on whose behalf they were answering. They tended to take a little more time to work through the question, and reported feeling less confident in their answers. In one case, ‘feeling depressed’ caused the participant to struggle to provide an answer:

For me to judge somebody, even my mother, she’s depressed.

OK.

I don’t think it’s, you know it’s down to me to judge that really.

And if it was just about depression how would you answer?

I couldn’t answer it really.

OK.

To be honest with you, I don’t think I’m . . . capable of judging that you know.

Relative 03

This response to the question mirrors our findings in the staff focus groups, where at least one person felt that they could not answer the question because they felt that specific medical knowledge was required to be able to judge if a person had depression.

Consequently, our review of the first stage of the cognitive testing concluded that the wording of this question needed to be revised to ensure that it reflected a non-medical, non-diagnostic approach to understanding how care home residents felt. To reflect quality of life, the term ‘feeling depressed’ was replaced by ‘low mood’ and the domain was renamed low mood. The new question, which was tested in phases 2 and 3 of cognitive testing, was ‘Which of the following statements best describes how often you think [the resident/your relative] feels down or has a low mood?’.

The use of the new question stem in the second and third phases of the cognitive testing seemed to address the problem. Participants viewed ‘feeling down’ and ‘low mood’ as related states, reflecting broadly the same thing:

So, thinking about the terms feeling down or low mood, what do you think we mean by those terms, feeling down, shall we do that one first.

Yeah, feeling down is we – we can’t feel their feelings, but you can – we can see their expression, like face and like frustration, and sometimes they pull hairs, like and put their hands on their head and yeah, and we can see there is some sort of low mood.

OK, and what do you think we mean by low mood?

Low mood means sad probably, yeah, sad, and missing something, kind

Care home staff 20

A similar point was made by a relative of a care home resident. They also talked about how the terms used in our question had not made her think that it was about whether or not her relative had a diagnosis of depression:

OK, what I’m going to ask you is what do you think the question meant by feeling down or has a low mood? What kind of things do you think–, well what kind of things did you think about when you saw those?

I thought about feeling a little bit depressed about your surroundings and about where you are and feeling down in terms of being in the care home, her situation. Not whether she had clinical depression or anything like that, just whether does she feel–, feeling down to me is a bit–, it’s very different than feeling depressed but I–, I focused on the feeling down bit. And I don’t think she was ever depressed but I do think she felt very down that her life had come to that conclusion that she needed to be in a care home and she was very aware that she was a single lady all her life and that now she was reliant on being in a care home and reliant on other people, so I saw that as did she feel down in her situation and her surroundings?

And what about the bit that says and–, or has a low mood? Did you–, did you think about that while answering the question?

Yeah. I feel down and low mood are the same thing to me.

Relative 12

In the above example, the term ‘depressed’ is used, but the respondent makes clear that this is not the same as a diagnosis of depression, which here is referred to as ‘clinical depression’. This term came up in other responses during the second and third phases of cognitive testing, but again it was not used in the medical sense to denote a diagnosis:

I’m going to ask you to kind of explain to me why you went for that one [answer option] as opposed to any of the others?

Occasionally because if she doesn’t see her niece and that, she will feel a bit low and a bit depressed, a bit looking depressed because she hasn’t seen her niece.

And this question uses the term and asks you to think about feels down or has a low mood, and what did those terms mean to you, when you were looking at this question?

Just feeling a bit depressed and lonely because she hasn’t seen her relative and that and she feels that she’s been left here and forgotten about.

Care home staff 13

The explanations of the (now) low mood questions given by participants in the second and third stages of cognitive testing clearly suggested that understanding of the question and terms used matched our intending meaning. This is not to say that every participant found it easy to answer the low mood question. Indeed, a couple of participants noted that it could be hard to know what another person, be it a relative or a person they support, was thinking and feeling, which made answering with confidence challenging. Moreover, some found the scale used for the response options difficult, as we discuss in the next section.

Recruitment and sampling

In February 2018, an initial scoping questionnaire was sent to all independent sector care homes on the CQC register in six local authorities in South East England that offered care to those aged > 65 years and/or living with dementia (n = 1242). Within the questionnaire, care homes could express an interest in being contacted about participating in future research.

Of the 112 homes that expressed an interest in future research, we removed those that had been involved in a previous study (n = 10), those with 10 or fewer beds (n = 7) and those that offered care primarily for people with physical disabilities (n = 1). A further six homes from two local authorities were later removed for efficiency reasons (i.e. the time taken to obtain additional research governance approvals).

Owing to high staff turnover within the sector, we also checked the contact details of the managers of these homes and removed any homes that had since closed or changed both ownership and registered manager (n = 3). Three homes subsequently expressed interest in the study by word of mouth. This resulted in a sample of 88 homes. We sent letters to each care home (including an information sheet for the care home manager; see Report Supplementary Material 1), and followed up by telephone and/or face-to-face meetings to explain in detail what participation involved. We also used the meeting to provide all documentation for recruitment, including a research flow chart of the process (see Report Supplementary Material 5). Of the 88 homes invited to take part, 20 agreed to participate in the study (10 residential care homes and 10 nursing care homes).

Care home managers oversaw resident recruitment. Managers of homes with ≤ 40 residents were asked to invite all residents to participate, including those lacking capacity (see Data collection for full details of the resident consent process). We asked homes with > 40 residents to randomly select 20. The research team gave the following instructions: ‘Please list the permanent residents (long-stay) in your home in alphabetical order. Now, find the nth person on the list and select them as person number one. From that person, please select every x person, and continue this while you move down the list until you have reached 20 residents. These are the residents who have been randomly selected to take part’. The nth starting number was randomly assigned by the research team. The researchers calculated the x number in proportion to the size of the home. The team repeated this process until a minimum of five residents had consented in each home. Posters were provided that managers could place in their care home to make residents and visitors aware of the research (see Report Supplementary Material 6).

Data collection

The study received a favourable opinion for ethics approval by the Health Research Authority (reference 18/LO/0657) and approval from ADASS. Research governance was sought from and approved by the four local authorities involved in the study.

We sought advice from staff and managers regarding each resident’s capacity to consent. Those who could consent for themselves received the information to do so (see Report Supplementary Material 3 and 7). In accordance with the Mental Capacity Act,155 residents lacking the capacity to consent were included in the study only if a personal consultee (e.g. unpaid carer, friend or relative) advised us that they thought the resident would like to participate (see Report Supplementary Material 10). Family carer information sheets were provided to inform residents’ family members of the research (see Report Supplementary Material 4), along with consent forms for those taking part in an interview about the resident (see Report Supplementary Material 8). During data collection in each home, consent was viewed as a continuous process, and researchers monitored this, taking into consideration the advice from staff and consultees. Before beginning observations or interviews, researchers explained the study and checked the resident’s willingness to participate. Information sheets were provided for staff (see Report Supplementary Material 2), as were consent forms for staff who took part in interviews about a resident (see Report Supplementary Material 9).

Between June and December 2019, researchers spent between 1 and 3 days collecting data in each care home using the mixed-methods ASCOT-CH4 approach described briefly in Chapter 1 and in detail in Appendix 1.

The number of residents recruited determined the exact period of time spent in each home. For example, with five residents, we completed 2 hours of observation, with the remaining time used to conduct interviews and allow staff to complete questionnaires on residents’ demographics, dependency and health.

Questionnaires

Methods: statistical analysis

The Rasch analysis was conducted in WINSTEPS^® version 3.92.1 (www.winsteps.com). All other statistical analyses were conducted in Stata version 16.

Feasibility

There were no missing data for the overall rating for each item. These final scores are a combined rating made by researchers based on a range of evidence sources.

Observations by researchers were completed for all residents. The evidence gathered by observation was supplemented by that from the qualitative and/or quantitative interviews with the residents, their family and care staff (Table 10). In the majority of cases (≥ 87%), data were missing for family interviews. There were also missing data for resident interviews in 39–57% of cases; the majority of complete cases were represented by qualitative interview only, without a rating of quality-of-life score for each domain. The missing values for staff interviews were low (≤ 1%) for all items.

For resident, family and staff interviews, the proportion of missing values for the three new items (anxiety, low mood and pain) were within the same range as for the ASCOT-CH4 items.

The absence of missing data for the overall ratings indicates that the ASCOT-CH4 method is feasible in terms of the researcher judging and rating an overall score for the eight ASCOT items and also in terms of the three new items. However, it is important to note that the majority of ratings were based on quantitative evidence from staff interview and observations only. This is because of the challenges and barriers associated with conducting interviews with resident or family, for example if the resident does not have close family.

Table 11 shows the association between staff and overall ratings (quantitative interviews only). There were moderate to strong associations between staff and overall ratings for all ASCOT-CH4 items (Cramér’s V of 0.25–0.5). As hypothesised, the associations were stronger for the three new items (Cramér’s V of ≥ 0.5).

There was exact correspondence of staff and overall rating of quality of life for 44–76% of the ratings. Where there was not exact correspondence, the majority of discrepant responses resulted from researchers rating the quality-of-life state as one lower than the staff rating. Owing to researchers’ reliance on staff report to judge the appropriate rating for items related to internal health states, the highest correspondence was for the rating of pain (75.8%), with a high degree of correspondence also observed for low mood and anxiety. The level of correspondence was, however, also high for the ASCOT domains of personal comfort and cleanliness and personal safety. These are quality-of-life domains for which care staff would be expected to make care-related judgements on a routine basis, and is evidence of reliance on staff ratings for these items and/or that they used similar external indicators to observe and form the same judgement. The level of correspondence of staff to overall ratings was lowest (44–47%) for the higher-order ASCOT domains of social participation, occupation and control over daily life.

Construct validity

Table 12 shows the results of the construct validity analysis by hypothesis testing. There is good overall evidence of the construct validity of the new items, which performed as hypothesised in terms of relationship to measures of similar or overlapping constructs (see Table 12).

All three items’ expected ratings of quality of life (i.e. the underlying need, without care), but not current quality of life (i.e. with the compensatory action of care), are associated with ADL dependency. The hypothesised significant positive relationship was found between the new pain item and another item designed to capture pain frequency, as well as with two items designed to capture the related construct of pain intensity (i.e. interRAI pain intensity, EQ-5D-5L pain item). As hypothesised, the new anxiety and low mood items, which capture the frequency of anxiety and low mood, were associated with measures designed to capture the similar constructs of clinical diagnosis of generalised anxiety disorder, clinical depression and/or the intensity of anxiety and depression.

Exploratory factor analysis

The EFA of the ASCOT-CH4 items and the three new items is shown in Table 13. The KMO measure of sampling adequacy (KMO = 0.79) and Bartlett’s test of sphericity (p < 0.001) indicated that the variables were adequately related for factor analysis. The items were analysed with principal factors. The eigenvalues for the first four factors extracted were 3.46, 0.98, 0.37 and 0.20, respectively, which indicates a two-factor solution if Jolliffe’s criterion is applied (≥ 0.70). This was verified by visual inspection of the scree plot. Oblique oblimin rotation of the factors was applied to facilitate interpretation of the two-factor model. 13

All the factors loaded above the threshold of ≥ 0.40 for reliability of interpretation,260 except food and drink and personal safety for factor 1. The first factor included the eight ASCOT-CH4 items, although personal safety loaded onto factor 2 with a slightly higher value than factor 1. This is consistent with other studies of the structural characteristics of ASCOT. 13,16 The second factor included the three new items, as well as, potentially, personal safety. The two factors accounted for 79.8% and 22.6% of the variance, respectively.

The uniqueness is the proportion of variance not explained by the common factor. Six of the 11 items had a uniqueness of ≥ 0.60, which indicates that these variables may not be well explained by the two factors and that the variables are also only weakly related to each other. This is consistent with the evidence for the eight items in the ASCOT. 13,15,16

Rasch analysis

Table 14 shows the Infit statistics for the two Rasch analyses with the (1) ASCOT items and (2) the three new items. They were in the acceptable range of 0.7–1.3 for the eight ASCOT items and also the three new items, as well as overall for each scale. The z-standardised probability statistics were within the acceptable range of ± 2.0 for all items, except the new item for pain. This indicates that the data for this item do not fit the Rasch model.

The category response curves shown in Figures 7 and 8 indicate that there were no disordered category thresholds for the new items of pain, anxiety and low mood. There were no ratings in the study sample of high-level needs for three of the eight ASCOT items (personal comfort and cleanliness, accommodation comfort and cleanliness and dignity), so it was not possible to evaluate the full range of category thresholds using the sample data. There was, however, evidence of a disordered category threshold for personal safety, which could be addressed by collapsing together the ratings for some needs and high-level needs into a single category to improve model fit.

FIGURE 7.

Category probability curves: ASCOT items. (a) Food and drink; (b) accommodation; (c) personal comfort and cleanliness; (d) social participation; (e) occupation; (f) control over daily life; (g) personal safety; and (h) dignity.

FIGURE 8.

Category probability curves: new items. (a) Anxiety; (b) low mood; and (c) pain.

In principal components analysis of the residuals, the Rasch models explained 53.7% (ASCOT-CH4) and 49.9% (new items) of the variance. In the Rasch model with the ASCOT-CH4 items, unexplained residual variance in the second, third and fourth contrasts was 8.1%, 7.0% and 6.2%, respectively. In the Rasch model with the new items, the second contrast had 20.6% unexplained residual variance, with < 0.5% for other contrasts. Although the criteria of ≤ 5% unexplained residual variance of additional factors was not met for either of the Rasch models, the eigenvalues for the subsequent contrasts were ≤ 2.0. This indicates that there were likely to be no secondary dimensions beyond random noise.

The Mantel–Haenszel statistic for all comparisons did not reach significance when adjusted for multiple comparisons. Therefore, there was no evidence of uniform DIF by gender, age or dementia diagnosis.

Internal consistency

As the structural validity analysis indicated that the eight ASCOT-CH4 items and the three new items load onto two separate factors, Cronbach’s alpha was calculated separately for each.

Cronbach’s alpha was 0.76 (eight items) and 0.61 (three items), respectively. Unlike the eight ASCOT-CH4 items, the three new items do not meet the criterion value of ≥ 0.70 for acceptable internal consistency.

Feasibility of using the CH4 method

Based on the combined rating made by the researchers, there were no missing scores (current and expected) for SCRQoL or the new items. However, as expected, there were high levels of missing information from resident self-report interviews. Of the 197 residents included in the analysis, around 60% could tell us something about their SCRQoL in a semistructured, conversational way, but only 15% (dignity) to 24% (food and drink) could give an answer to the conventional structured interview, in which a response option must be chosen. The easiest SCRQoL domains for residents to rate were food and drink, accommodation and personal safety, which are all very tangible concepts for residents to grasp. The most difficult SCRQoL domain was dignity (nearly 85% of resident interview data were missing), which is more cognitively challenging, asking residents to reflect on the impact of the way they are treated by staff. The pattern was similar for the new items, with around 58–60% being able to tell us something about their pain, anxiety and low mood in a conversational or semistructured way but fewer than one-quarter responding to the structured interviews.

Thus, the majority of residents in our sample were not able to self-report and, of those who could, many were able to give only partial responses and not consistently in all domains. This is despite under half having a diagnosis of dementia and around one-third lacking the capacity to consent. Thus, inability to self-report is likely to be due to the multiple and complex needs in this population, including cognitive impairment, physical frailty and sensory impairments.

Staff interviews were conducted for all residents and mostly staff felt able to respond to every domain. No data were missing for the new items, indicating that it is certainly feasible to gather this information from care workers. Family interviews were much harder to obtain, which is why such large numbers of missing data were reported. These figures represent low response rates. When family members were interviewed, they actually answered most questions. It is currently recommended that family interviews be completed face to face, which is one of the main barriers to including family members in interviews. Future research may want to explore the feasibility of conducting telephone or video call interviews to increase response rates.

Therefore, with the exception of the family interviews, which had low recruitment rates, the ASCOT-CH4 was a feasible method of data collection. Self-report alone, however, was not. The majority of residents were unable to complete a full, structured interview. Although staff proxy interviews were helpful in informing final ratings, agreement between staff and final ratings ranged from 44% (occupation) to 75% (pain). Where ratings diverged, researchers’ ratings were usually one outcome state lower than staff’s (e.g. ‘ideal’ to ‘no needs’). Thus, the mixed-methods approach added value to a purely proxy approach by giving residents a voice in the final ratings of their own SCRQoL, even if only partially or through the evidence collected in observations. Without this, residents’ outcomes in these domains would sometimes have been overestimated by staff.

Psychometric properties of the new items

There is good evidence of the construct validity of the three new items, both current and expected ratings. EFA indicates that the three new items do not load onto a single factor with the eight ASCOT-CH4 items. This is expected, as the concepts captured by the new domains (i.e. pain, low mood and anxiety) relate to aspects of HRQoL, which is distinct from the ASCOT construct of SCRQoL (i.e. aspects of quality of life that may be improved by social care support).

The structural validity analysis using classical test theory (EFA) provides very tentative support for using the three new items as a subscale alongside ASCOT-CH4, which employs the same methodology and approach, to allow consistency for comparison. The item response theory findings, combined with the high uniqueness of this item in the EFA, indicate that pain may not fit into a measurement scale alongside low mood and anxiety. This is also supported by the inadequate internal consistency between these three items using Cronbach’s alpha. As such, the evidence that the three items form a separate measurement scale is tentative. It may be better to conceptualise these three items as separate ‘modules’ that relate to the concepts of psychological health and pain that may be added flexibly alongside ASCOT-CH4 (as a separate scale), with low mood and anxiety combined together and pain standing alone.

Conclusions

The ASCOT-CH4 offers a robust methodology for measuring the social care outcomes of care home residents who cannot self-report. Previous work had already established excellent inter-rater reliability of the domains of SCRQoL. 19,20 We applied this methodology to the new domains of pain, anxiety and low mood and found that the mixed-methods approach could be applied to these domains. The three additional domains could be used alongside the ASCOT in the future to capture the impact of care homes on these important health-related constructs.

Dependent variable

For the purpose of this analysis, care home residents’ outcomes are measured with the ASCOT SCRQoL using the mixed-methods approach (CH4). We assessed the impact of CQC quality rating on both the current (overall) SCRQoL and the eight ASCOT domains (see Table 1).

Independent variables: Care Quality Commission quality ratings

Our main variable of interest was CQC care home quality ratings (see Chapter 1), either overall or on one of the five KLOEs. The ratings considered were from the inspection report closest to the date of fieldwork; see Chapter 1 for a summary of frequency of inspections by CQC.

Independent variables: confounding factors

People’s quality of life is not affected only by the quality of care services; it is also highly correlated with individual and other characteristics. 261 Probably the most important confounders, as shown by previous studies, are the levels of physical and cognitive impairment, which are negatively correlated with quality of life. 22,262,263 If we did not control for impairment in the analysis, and if those with higher care need were more likely to reside in care homes of lower quality, then the association between CQC quality ratings and SCRQoL would be upwards biased. We included the following measures of impairment in the analysis: the count of ADLs a resident manages independently and level of cognitive impairment as measured by the MDSCPS. Previous research has shown that the expected SCRQoL score (i.e. the resident’s hypothetical SCRQoL if services were not in place to support the person and nobody else stepped in; see Chapter 1) is sometimes better at capturing impairment and social care needs than ADLs. 261 We therefore included the expected SCRQoL score instead of ADLs in different specifications. Other individual characteristics used as controls were gender, ethnicity and age group. We also included the source of funding (i.e. private or public) as a control in one specification to capture whether or not care homes are providing better care to their private-paying residents.

At the care home level, we included only contextual factors (i.e. factors not related to care quality improvement through the care homes) as independent variables: type of care home registration (i.e. residential or nursing), sector (i.e. private or voluntary) and care home capacity (i.e. number of beds). The quality improving aspects (e.g. staffing, training, pay, management style) should be captured by the quality ratings.

At the local area level, the main factors that could affect both care home quality and residents’ outcomes are the local authority social care policy and commissioning practices (which can affect care home revenue) and the income and wealth of the local population with care needs (which may determine the proportion of self-funding residents and, ultimately, care home revenue). Further local area factors could be the level of competition between social care providers and the availability and quality of the local workforce. To capture the potential effects of all these local area-level factors, we included dummy variables of the local authority district where the care home was located.

To ensure consistency of fieldworker ratings, anyone using the ASCOT mixed-methods toolkit is required to attend training by the ASCOT team. A hierarchy of evidence is used to resolve any conflicts in evidence and establish inter-rater reliability (see Appendix 1). For this study, the fieldworkers collecting the data were the researchers involved in the original ASCOT tool development, with excellent inter-rater reliability having already been established between them in previous studies. 19,20 Nonetheless, controls for the study (i.e. MOOCH or MiCareHQ) and fieldworker were included to capture potential subjective differences in which the residents were rated with respect to SCRQoL.

Statistical methods

The model used to assess the relationship between the SCRQoL of resident i in care home j (Q_ij) and the CQC quality rating of care home j (R_j) is:

where X_ij are individual, care home and local area characteristics of resident i in care home j, and ε_1ij is the error term. Everything else being equal, the coefficient β₁ captures the effect on SCRQoL from a change from a lower to a higher care home rating for a resident with characteristics at the mean of the sample.

Residents with different (initial) care needs might have a different capacity to benefit from higher-quality care services. To capture the differential benefit in SCRQoL by level of care need from higher- compared with lower-rated homes, we also estimated a model in which we interacted the resident’s care need level (N_ij) with the quality rating of the care home (R_j):

where N_ij is a dummy variable equal to ‘0’ if the resident has a high care need (i.e. has an expected SCRQoL score in the bottom half of the sample) and equal to ‘1’ if the resident has a low care need (i.e. has an expected SCRQoL score in the top half of the sample). The coefficient β₂₂ captures the effect on SCRQoL for a high-needs resident from an increase from a lower to a higher care home rating, and β₂₃ captures the difference in the effect on SCRQoL between low-need and high-need residents from an increase from a lower to a higher care home rating. The effect on the SCRQoL of a low-needs resident from a change from a lower to a higher care home rating is given by β₂₂ + β₂₃.

Multilevel estimators can be advantageous in studies where the observed individuals are ‘nested’ into groups. However, for this analysis there was a rather small number of residents per care home, with an average of 8.8 and a minimum of 4. Moreover, a likelihood ratio test comparing the multilevel estimation results with ordinary least squares (OLS) results for Equation 4 was insignificant (p = 0.2701), showing that the OLS estimation results are not statistically different from those of the multilevel estimation. We therefore used OLS for the regression analysis.

Descriptive statistics

The distribution of the current SCRQoL score for the 475 care home residents in the sample is negatively skewed (mean 0.75, standard deviation 0.17, range 0.24–1.00, skewness –0.53 and kurtosis 2.53), but it is similar to that of other ASCOT measures in samples of older and younger adult social care service users16,264 and is not greatly different from the normal distribution (Figure 9). The current (and individual domain) SCRQoL scores were therefore used as dependent variables in their untransformed form.

FIGURE 9.

Kernel density plot of ASCOT-CH4 current SCRQoL score.

Of the 54 care homes included in the sample, 13 (24%) had a CQC rating of ‘requires improvement’, 36 (67%) were rated ‘good’ and five (9%) were rated ‘outstanding’. No care homes were rated ‘inadequate’ in our sample and only a small number of homes had an ‘outstanding’ rating (Figure 10). We therefore grouped ‘good’ and ‘outstanding’ homes together in the analysis.

FIGURE 10.

Distribution of CQC quality ratings of sampled care homes overall and for each CQC KLOE.

The national distribution of CQC ratings among care homes for older people as of September 2018 (i.e. half way between the two data collection points) was 2% ‘inadequate’, 21.5% ‘requires improvement’, 74% ‘good’ and 2.5% ‘outstanding’ (proportions were calculated from the CQC care directory data). Therefore, the proportion of ‘good/outstanding’ homes in our sample (74%) was very close to the national proportion (74.5%). For the quality ratings distribution in our sample of care homes to be even closer to the national distribution of care home quality ratings, we would have needed one home in our sample to be rated ‘inadequate’. However, homes rated ‘inadequate’ are very difficult to recruit as (1) they are very few and (2) they are usually in this category for only a short period (i.e. a care home rated ‘inadequate’ for one of the five key questions has to improve quickly and is reinspected within 6 months). 132 A home rated ‘inadequate’ was, in fact, recruited in the MOOCH study. However, by the time of data collection, that home had improved its rating to ‘requires improvement’ and was reported accordingly.

Figure 11 illustrates the equally weighted expected SCRQoL score, SCRQoL score gain (i.e. difference between the current SCRQoL score and the expected SCRQoL score) and unmet need (i.e. difference between the ideal state and the current SCRQoL score) in each ASCOT domain by the level of residents’ care needs and overall CQC care home rating. By definition, care residents with high care need (i.e. those in the bottom half of the expected SCRQoL scores) have a relatively lower expected SCRQoL score. The dignity domain captures ‘the negative and positive psychological impact of support and care on the service user’s personal sense of significance’13 (see Table 1) and, therefore, it does not make sense to ask the expected question for this domain (i.e. hypothetical ratings in absence of support). Instead, ASCOT uses a dummy code for the expected SCRQoL score for dignity, which assumes that all residents would have no (unmet) needs (i.e. a score of 0.67).

FIGURE 11.

Equally weighted expected SCRQoL score, SCRQoL score gain and unmet need in each ASCOT-CH4 domain by level of residents’ care needs and overall CQC care home rating.

We observed that high-need residents generally have a bigger gain on all ASCOT quality-of-life domains than low-need residents, confirming findings from previous studies that service users with higher-level impairment have a greater capacity to benefit from social care services. 13,265 We also observe, mainly for high-need residents, a higher SCRQoL score gain (and a lower unmet need) in ‘good/outstanding’ than in ‘requires improvement’ homes. With the exception of personal cleanliness, for all other domains, high-need residents have a higher SCRQoL score gain in ‘good/outstanding’ homes of about 0.07–0.10 (or 10–24%).

For low-need residents, we also find a relatively lower unmet need with respect to most ASCOT quality-of-life domains in ‘good/outstanding’ homes.

Table 15 presents a comparison of resident and care home characteristics by CQC care home quality rating. Most of the individual characteristics (e.g. age, gender and ethnicity, as well as impairment as measured by the number of ADLs that the resident manages independently) were not statistically different between homes with different CQC quality ratings. However, residents in ‘good/outstanding’ homes had a significantly higher mean current SCRQoL score, a higher proportion were self-funders (i.e. they funded their care wholly with their own funds), a higher proportion had significantly lower care needs initially (as measured using the mean expected SCRQoL score) and a higher proportion were cognitively intact. In terms of care home characteristics, compared with ‘requires improvement’ homes, a significantly smaller proportion of ‘good/outstanding’ homes offered residential care services with nursing, and ‘good/outstanding’ homes had, on average, a significantly smaller capacity.

Multivariate regression analysis

The statistical analysis was conducted using Stata^® version 16 (StataCorp LLC, College Station, TX, USA). To account for potentially correlated errors due to unobserved care home characteristics, we estimated fully robust standard errors by clustering at the care home level.

Table 16 includes OLS estimation results for five models of current SCRQoL score. Model (a) includes the number of ADLs a resident manages independently as a measure of impairment, as in a previous analysis using the MOOCH data. 20 Similar to this previous study, we found a positive relationship between residents’ quality of life and a ‘good/outstanding’ versus ‘requires improvement’ CQC rating; as well as positive relationships with being female and being able to perform more ADLs independently, and a negative relationship with cognitive impairment.

Previous research has shown that the expected SCRQoL score is sometimes better at capturing impairment and social care needs. 261 Our estimation results confirm this. When ADL count was replaced with expected SCRQoL [models (b) and (c)], the coefficients for the cognitive impairment measure (i.e. MDSCPS) became smaller (and some of them became statistically insignificant). This shows that the expected SCRQoL is relatively better at capturing aspects of cognitive impairment. Moreover, we also noticed that the coefficient of the overall CQC rating became smaller (and significant only at the 10% level), which may be a result of expected SCRQoL covering aspects of care need (e.g. lack of occupation or social participation) that are not captured by either ADL count or MDSCPS.

In a further model we tested whether funding source for social care services has an effect on the resident’s SCRQoL; see model (d). In other words, we tested whether self-funders (through higher fees) are buying better services and, therefore, have better care outcomes. We did not find any evidence of a significant relationship between a resident’s funding source and SCRQoL. This suggests that residents receive similar levels of care and attention regardless of the funding source and fees paid.

Finally, in model (e) we tested whether residents with different (initial) care needs might have a different capacity to benefit from higher-quality care services (see also Equation 4) and found that high-need residents had more capacity to benefit. A high-need resident with characteristics at the sample average would have a current SCRQoL score 0.091 (p = 0.028) higher (equivalent to 12% of the average quality of life in the sample) if their care home is rated ‘good’ or ‘outstanding’ than if it is rated ‘requires improvement’. On the other hand, a low-need resident would have only a 0.034 (p = 0.362, i.e. not statistically different from zero) higher current SCRQoL score if her or his care home is rated ‘good/outstanding’ compared with if it is rated ‘requires improvement’.

Table 17 presents the summary of marginal effects of CQC quality ratings (both overall and for the five KLOEs) on current SCRQoL and (weighted) ASCOT domain scores by level of residents’ care needs for models based on Equation 4; the coefficient in row 1, column 1, is from Table 16, model (e).

We found no statistically significant relationship between the overall CQC rating and ASCOT basic quality-of-life domains (i.e. personal cleanliness and comfort, food and drink, accommodation cleanliness and comfort, and personal safety), indicating that all care homes managed to meet residents’ basic quality-of-life needs well. However, the differences between ‘good/outstanding’ and ‘requires improvement’ homes were evident with respect to outcomes on ASCOT higher-order quality-of-life domains, in particular control over daily life and social participation and involvement. Both high-and low-need residents had a higher quality-of-life score with respect to control over daily life and social participation and involvement in better-rated homes, but high-need residents benefited more once again.

In terms of individual CQC KLOEs, ‘caring’ and ‘well led’ related more to ASCOT quality-of-life scores. Homes rated ‘good/outstanding’ with respect to ‘caring’ (i.e. care homes in which staff involved and treated residents with extra compassion, kindness, dignity and respect) were most strongly related to ASCOT higher-order domain scores. Compared with high-needs residents, the relationship was relatively stronger for low-need residents on all four higher domains (i.e. control over daily life, social participation and involvement, occupation, and dignity). This is likely to be because residents with lower needs are more physically and cognitively able to benefit from increased choice (and to have those choices respected), meaningful occupation and social activities. Conversely, high-need residents benefited significantly more from care homes with better leadership, management and governance (i.e. ‘good/outstanding’ rating with respect to ‘well led’). Good management is important for generating better care outcomes in those with the highest care needs. This may be because well-managed homes have more effective working environments and better skill development opportunities for care staff, which in turn is associated with better-quality care (see Chapter 6). ‘Well led’ is the only KLOE that was related not only to higher ASCOT domains, but also to basic domains, in particular food and drink and personal safety.

Sensitivity analysis

As there were no ‘inadequate’ rated care homes in the sample, an important question is how this might have affected the research findings. As mentioned, according to national distribution of care home quality ratings, we should have had one of the 54 sample care homes that was rated as ‘inadequate’ rather than ‘requires improvement’. Therefore, the potential selection bias is rather low. Moreover, we performed a sensitivity analysis that found that the size of positive association between CQC quality ratings and residents’ quality of life found in this study might have been underestimated. Specifically, we excluded care homes with the highest quality rating (i.e. ‘outstanding’) from the estimation of model (e) in Table 16. When doing this, the coefficient for CQC quality rating became smaller (i.e. 0.081; p = 0.066). This means that residents of the ‘outstanding’ homes in the sample have on average (and everything else being equal) relatively higher SCRQoL than residents of ‘good’ homes. Assuming that the opposite is true for residents of ‘inadequate’ homes (i.e. everything else being equal, they have lower SCRQoL than residents of ‘requires improvement’ homes), the true relationship between CQC quality rating and residents’ SCRQoL would be stronger (i.e. greater coefficient).

Sample

We created an unbalanced panel of 5555 independent sector care homes observed over the 3 years, with a total of 12,052 observations. In total, 2540 of the homes were observed across all 3 years. For each year there are approximately 4000 care home observations, representing more than one-third of the more than 11,000 care homes for older people registered with the CQC at the time. Representativeness of the sample was assessed by comparing to the national database of care home providers across the 3 years. Nursing homes represented 38.3% of the sample and the voluntary sector represented 11.8%, and the average size of a care home was about 41 beds. These were in line with national figures and we proceeded on the basis that the sample was representative of all care homes in England.

Dependent variable: quality

We measured quality using the CQC’s quality rating system, which assesses people’s experiences of care and is centred on an inspection of the care home (see Chapter 1). Because of the small number of homes that had either the lowest (i.e. ‘inadequate’) or the highest rating (i.e. ‘outstanding’), we used a binary variable indicating high or low quality (i.e. 0 if a home was rated ‘inadequate’ or ‘requires improvement’ and 1 if a home was rated ‘good’ or ‘outstanding’).

Independent variables: staffing measures

Independent variables: covariates

We included in the model a number of confounding variables at care home and local area level (postcode district, i.e. the first half of a UK postcode) that are likely to influence quality and wages. We list these measures and their data sources in Table 18. We further included training incidence, job vacancies and staff turnover as covariates in the first-stage wage equation.

Statistical methods: multiple imputation

The staffing data contain missing information (Table 19). Given the richness of the ASC-WDS, we employed multiple imputation techniques, assuming that the data are missing at random (i.e. independent of unobservable parameters). We generated 50 imputations using chained imputation methods with ordered logit (overall quality rating) and predictive mean matching specifications (staffing measures). We assessed the size of the Monte Carlo error generated from the multiple imputation process and found this to be small enough to mean that the effect of staff measures on quality ratings would be unlikely to change if the multiple imputation process were repeated. 271

Statistical methods: instrument for wage

It is likely that the staffing measures are endogenous in a model of care home quality. 150 To assess the true effect of an endogenous variable, we required an instrument that is correlated with the endogenous variable (staffing measures) but is not directly related to the dependent variable (quality). Following previous studies, we used exogenous increases to the minimum wage floor as an instrument for wage, specifically the proportion of workers employed by provider j being paid less than the future National Living Wage rate. 272

We assessed for the endogeneity of wage and exogeneity of the instrument using relevant variable addition tests. 273 We also assessed the strength of the instrument using weak and overidentification tests, the latter when including a second instrument, a spatial lag of Pension Credit uptake, in a complete-case regression of quality ratings.

Statistical methods: model

Given the panel nature of the data and the use of instrumental variables, our analysis used a linear probability model of quality ratings:273

where the observed level of quality (i.e. the rating), R, for care home j in time t depends on the vector of staffing measures of interest, S, other exogenous care home characteristics, local demand and supply factors, X, and a random error, ε. We first estimated this model using pooled observations over time and assuming that the staffing measures (i.e. training incidence, turnover, vacancies and wages) are exogenous. We then used the panel nature of the data to estimate linear probability models of quality using both random-effects and fixed-effects specifications, assuming that the random-error term ε_jt is composed of both a home-specific error term, h_j, that does not vary over time, and a time-variant error term, u_jt. Stata version 16 was used for the analysis, specifically using the reg, ivreg2 and xtivreg2 commands, and we clustered standard errors by care home to account for heteroscedasticity.

Statistical methods: robustness checks

We further assessed the results by estimating random-effect probit specifications, given the binary nature of the dependent variable, and by running the regression models on quality ratings for the five KLOEs.

Descriptive statistics

Descriptive statistics for the complete cases are presented in Table 19, along with information on numbers of missing data. About three-quarters of care homes in our sample had a quality rating of ‘good’ or ‘outstanding’. Voluntary sector care homes represented 12% of care home observations, whereas nursing homes represented 38%. Seventy per cent of the observations were for homes providing services to those living with dementia, and average care home size was around 41 beds.

Figure 12 illustrates that voluntary (i.e. not-for-profit) care homes have better quality ratings than private (i.e. for-profit) ones and that care homes without nursing were also slightly better rated than care homes with nursing.

FIGURE 12.

Care home quality ratings by sector and type of service.

In terms of staffing measures, the average hourly wage for care workers in the sample of care homes was £7.85; 28.1% of staff had received dementia training; 12.7% had received training for person-centred care/dignity; and the average annual staff turnover and current job vacancy rates were 28.9% and 4.1%, respectively (see Table 19).

Figure 13 shows that care worker wages are slightly higher in care homes rated ‘good’ or ‘outstanding’ than in homes rated ‘inadequate’ or ‘requires improvement’, and that they are also higher in voluntary-sector care homes than in private-sector care homes. Figure 14 shows that care worker average hourly wage (left side) had a positive skew and was leptokurtic. The transformed natural logarithm of wage reduced both skewness and outliers in the distribution.

FIGURE 13.

Mean care worker hourly wage by sector, service type and quality rating (2018 £).

FIGURE 14.

Kernel density plots of (a) care worker average hourly wage and (b) natural logarithm.

Multivariate regression analysis

Table 20 presents the results from estimating the random-effects linear probability models of care home quality ratings based on Equation 5. A Mundlak test, whereby the mean of controls that vary over time are included in a random-effects model, showed that time-invariant unobservables (h_i) are not related to the regressors (joint F-test statistic of mean of time-varying controls of 15.84; ρ = 0.359) and that the random-effects model with instrumental variables (REIV) estimation is most appropriate. 274

Phase 1

A day-long planning workshop was arranged in central London, this being a more convenient location for the co-researchers than the University of Kent. The aim of the workshop was for the academic and co-researchers to get to know one another and for the PPI co-researchers to familiarise themselves with the project. During this workshop, the academic researchers shared their initial ideas about how the focus groups might work. Working together, the co-researchers and the academic researchers turned this into a final guide for facilitating the groups (see Appendix 3). We also discussed how to facilitate a focus group and the role each of the co-researchers felt comfortable adopting. It was agreed that each co-researcher would co-facilitate one focus group with an academic researcher.

Phase 2

The focus groups are described in detail in Chapter 3. Here we focus on the role of the PPI co-researchers.

Three focus groups were carried out by the team of PPI co-researchers and academic researchers. Each group consisted of three sections. The PPI co-researcher facilitated the first two sections, looking at, respectively, how care home staff recognise pain, anxiety and depression, and the words that care home staff use to describe residents’ pain, anxiety and depression. The final section, testing draft questions, was facilitated by an academic researcher. The co-researchers were asked to express a preference for which focus group they attended. Immediately prior to each group, the academic researcher and co-researcher met to run through the plan for the group.

Phase 3

Following the focus groups, the PPI co-researchers were sent copies of verbatim transcripts of the audio-recordings of the three groups. They were asked to review them and think about what we could learn from the data collected and, in particular, how this might feed into the development of our new ASCOT domains. This was then discussed at length in a face-to-face team meeting at the University of Kent Canterbury campus.

Phase 4

The PPI co-researchers were asked to reflect on their experiences of working as a co-researcher. They were sent five questions as a guide to help them write a short reflection:

1. What was it like to be a PPI co-researcher on this project?

2. What impact did being a co-researcher have on you?

3. What aspects of your involvement went well and which did not?

4. What impact do you think the co-researchers had on the project as a whole?

5. What can others learn from your experience on the project?

Two of the three co-researchers submitted a reflection on their experiences, both of whom used the questions to structure their reflections. Their reflections were used as the key part of this chapter.

The same questions were answered by the academic researcher who worked most closely with the PPI co-researchers.

Phase 5

As co-authors, the co-researchers were invited to comment on both drafts and final versions of this report and other publications.

Patient and public involvement co-researcher 1

Patient and public involvement co-researcher 2

Academic researcher

Objective 1: measuring the health and social care-related quality of life of care home residents

As outlined in Chapter 1, many older care home residents are living with frailty and dementia and have difficulty reporting their own needs, outcomes and characteristics using conventional methods such as questionnaires and structured interviews. The ASCOT is a suite of tools designed to measure the aspects of people’s quality of life most affected by social care. As well as conventional interview and self-completion tools, there is a mixed-methods version, ASCOT-CH4, for use in care homes with those who cannot self-report. Although this tool has been previously used in research and shown to have excellent inter-rater reliability and face-validity, the psychometric properties of the care homes tool had not yet been established. This study sought to address this evidence gap, and in Chapter 4 and Chapter 5 we described how the measure was used in two studies to collect information about care home residents’ current and expected SCRQoL.

In line with previous research using the self-completion measures, the results indicated that the ASCOT-CH4 had acceptable internal reliability (α = 0.76, eight items) and structural validity. The feasibility of the mixed-methods approach was measured by examining missing data. There were no missing data for final ratings. Every resident had a final ‘current’, ‘expected’ and ‘gain’ score for social care-related quality of life (see Chapter 1), based on researcher ratings. Conversely, had we relied on resident self-report only, using the structured interviews, we would have had missing data for 75–85% of residents in each domain. Staff proxy ratings helped inform final ratings and had very little missing information, but family members were harder to access. Had we relied on proxy report by staff only, residents’ outcomes would have been overestimated, as, where they diverged, researcher ratings for ‘current’ SCRQoL were often one outcome state lower than the perspective given by staff. Thus, the mixed-methods approach was both necessary and feasible.

However, a rapid review (see Chapter 2) of existing measures of pain, anxiety and depression found that there was no equivalent way of measuring the impact of care homes on these important aspects of HRQoL. Despite a significant number of care home residents experiencing symptoms of pain, anxiety and depression, evidence from the research literature indicates that symptoms are often under-recognised and undertreated in care homes (see Chapter 1). Existing tools tend to focus on severity of symptoms, with a view to diagnosing clinical conditions (see Chapter 2). However, the frequency, not just the severity, of symptoms can have a significant, negative impact on a person’s quality of life. Therefore, informed by the review, we drafted new tools measuring these concepts, using the ASCOT mixed-methods approach. We tested and refined the concepts and draft wording through focus groups with care home staff, and then cognitively tested these with staff and family members using a combination of verbal probing techniques and thinking aloud (see Chapter 3). Lay co-researchers were involved in the focus groups and the subsequent revisions of the new measures (see Chapter 7).

Staff in three focus groups were able to describe the verbal and non-verbal (behavioural) signs of pain, depression and anxiety in residents, which were helpful when conceptualising the observable indicators relevant to our tools. Although staff in the focus groups, and participants in the subsequent cognitive testing, were able to answer our draft questions for the new items, some changes were required. First, we changed the depression item to low mood, to avoid clinical, diagnostic or screening connotations. Second, the use of a four-point scale anchored to ‘hardly ever’ at the ‘ideal state’, instead of ‘never’, improved both ease of answer and clarity of meaning.

The final domains were called pain, anxiety and low mood. The question wording and response options for ‘current’ and ‘expected’ interviews and ratings were finalised in line with the ASCOT approach. In addition, as these measures were designed for use with people who cannot always self-report, observational guidance with non-verbal and observable indicators of outcome states in each domain was developed ready for piloting (see Chapter 3 and Appendix 3, Figures 3, 4, 16 and 17). A primary data collection was undertaken, and 182 residents from 20 care homes for older adults (10 nursing and 10 residential) were recruited to the study from four local authorities in South East England (see Chapter 4). To explore the construct validity of the new items, questionnaires also contained staff-rated, validated scales relating to these constructs so that we could explore hypothesised relationships with the attributes in the analysis.

As expected, psychometric testing of the three new items found similar results to the ASCOT-CH4 in terms of feasibility (see Chapter 4), indicating that self-report alone would not be feasible for this population but that a mixed-methods approach adds another perspective to one purely gathered by staff proxy interviews. The new domains did not form a unidimensional scale with the ASCOT-CH4 but could be used alongside it to measure these important aspects of HRQoL. The structural validity analysis indicated that, rather than being considered a three-item scale representing aspects of ‘health-related quality of life’ (pain, anxiety and low mood), it may be better to conceptualise these three items as separate ‘modules’ that relate to the concepts of psychological health and pain. These may be added flexibly alongside ASCOT-CH4 (as a separate scale), with low mood and anxiety combined and pain standing alone.

Taken together, the results indicate that the mixed-methods approach to measuring health (pain, anxiety and low mood) and social care-related quality of life (ASCOT-CH4) presented in this report offers a robust methodology for measuring the outcomes of care home residents who cannot self-report.

Objective 2: assessing how far Care Quality Commission quality ratings are consistent with indicators of residents’ care-related quality of life

We used the data from the study described in Chapter 4 and also from a previous study, MOOCH, described in Chapter 5, to examine the relationship between CQC quality ratings and residents’ SCRQoL, controlling for confounding variables. Multivariate regression analyses described in Chapter 5 replicated the findings of previous research and found a significant, positive association between residents’ SCRQoL and regulator quality ratings. The analysis also extended our understanding by finding that the impact of quality was greatest for high-need residents, with the equivalent of a 12% improvement in mean current SCRQoL for residents in ‘good/outstanding’ homes compared with homes rated ‘requires improvement’. Of the five KLOEs, being rated ‘good/outstanding’ for ‘caring’ and ‘well led’ were significantly associated with SCRQoL, with ‘well led’ the most important for high-need residents. Good management is positively associated with better care outcomes in those with the highest care needs. It is likely that well-led services have a better and more effective working environment and ensure adequate skill development for care staff (see Chapter 6 and the results of objective 3).

Objective 3: assessing how much the skill mix and employment conditions of the care workforce matter for quality

We conducted an analysis of secondary data of English care homes (n = 12,052 observations of 5555 care homes) for the years 2016 to 2018 to model the relationship between CQC quality ratings and workforce characteristics (see Chapter 6). We focused on the effect that training provision to staff and staff terms and conditions (wages and turnover/vacancy rates) had on the quality of care homes. We expected all of these variables, which can be affected by policy, to be important determinants of quality. To assess the relationship between care quality and workforce characteristics we used longitudinal panel data models, multiple imputation to address missing data, and an instrumental variable approach to control for the potential endogeneity between quality and workforce characteristics, in particular staff wages.

We found a significant positive association between quality and both wages and training (in dementia and dignity-/person-centred care). Staff turnover and job vacancy rates had a significant negative association. Specifically for wages, we found that a 10% increase in care worker average hourly wage increased the likelihood of a ‘good’ or ‘outstanding’ rating by 7%. Improving employment conditions, including pay and training, may have an important positive association with quality in care homes.

Search terms

(Pain) AND (Scale OR assessment OR measure) AND (Elderly OR residents Or geriatric OR nursing homes OR care homes OR cognitive impairment OR dementia OR Alzheimer OR older people)

(Pain) AND (Scale OR assessment OR measure) AND observation* or behaviour*

Databases

MEDLINE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), Cochrane Library, PsycInfo (via EBSCOhost) and Abstracts in Social Gerontology.

Search terms

(anxiety OR depression) AND (Scale OR assessment OR measure) AND (Elderly OR residents Or geriatric OR nursing homes OR care homes OR cognitive impairment OR dementia OR Alzheimer OR older people)

(anxiety OR depression) AND (Scale OR assessment OR measure) AND observation* or behaviour*

Databases

MEDLINE, CINAHL, Cochrane Library, PsycInfo (via EBSCOhost) and Abstracts in Social Gerontology.