Family-Reported Experiences Evaluation (FREE) study: a mixed-methods study to evaluate families' satisfaction with adult critical care services in the NHS

Research governance

The FREE study was sponsored by the Intensive Care National Audit & Research Centre (ICNARC) and phase 1 was co-ordinated by the Newcastle Clinical Trials Unit (CTU). An ethics application was made to the National Research Ethics Service Yorkshire and the Humber Research Ethics Committee on 14 August 2012 and received a favourable opinion on 17 October 2012.

The National Institute for Health Research (NIHR) Clinical Research Network (CRN) Portfolio details high-quality clinical research studies that are eligible for support from the NIHR CRN in England. The FREE study was adopted onto the NIHR CRN Portfolio on 14 September 2012.

Global NHS permissions were obtained from the lead comprehensive local research network (CLRN) – Northumberland, Tyne and Wear – and local NHS permissions were obtained for the two NHS hospital trusts that participated in phase 1. A study site agreement, based on the model agreement for non-commercial research in the health service, was signed by each participating NHS hospital trust and the sponsor (ICNARC).

Study management

Phase 1 of the study was led by EM (coinvestigator) and SW (lead clinical investigator) with support from the Study Management Group (SMG), which comprised the chief investigator (KR) and other coinvestigators (DAH, SH, DKH, LH, AR, MR). An experienced research associate was employed to conduct, transcribe and analyse the focus group discussions and cognitive interviews.

Recruitment of NHS hospital trusts

Two NHS hospital trusts were recruited to take part in the study: one in Newcastle upon Tyne and one in London.

Focus groups

Four focus group discussions were conducted with:

1. health-care professionals involved in the delivery of intensive care (to inform whether or not the FS-ICU-24 covered all dimensions relevant to the quality of intensive care in the NHS and on which a family member might be expected to have a view and the relevance and redundancy of items)

2. representatives from ICUsteps (for a further perspective on the face and content validity of the FS-ICU).

Cognitive interviews

Up to three rounds of cognitive interviews were planned, each involving four to eight participants. 19 At the end of each round of interviews, the findings were reviewed by the research associate with EM and SW. The wording of items was modified, if necessary, and then tested in subsequent rounds of interviews. Interviews continued until no fresh insights emerged.

Participating NHS hospital trusts

Local research and development (R&D) approval was obtained at the NHS hospital trust in London for the London-based focus group discussion (with health-care professionals) and at the NHS hospital trust in Newcastle for the Newcastle-based focus group discussion (with health-care professionals) and for recruitment of family members for the cognitive interviews.

Participants

Thematic analysis

Themes that emerged from the focus group discussions and cognitive interviews have been combined and are presented for each of the FS-ICU-24 items below.

Summary of changes to the 24-item Family Satisfaction in the Intensive Care Unit questionnaire

The changes, agreed by the SMG, to the original FS-ICU-24 are summarised below. KR and EW led on implementing the agreed changes and on the reformatting/redesign of the questionnaire to enhance response.

The UK 24-item Family Satisfaction in the Intensive Care Unit questionnaire

The final version of the UK FS-ICU-24 questionnaire (see Appendix 2), adapted from the original Canadian FS-ICU-24 questionnaire, is outlined below. The questionnaire comprised three parts: demographic information about the family member completing the questionnaire (under the heading About you), satisfaction with care and satisfaction with decision-making.

The first part of the questionnaire asked for information about the family member completing the questionnaire as follows:

Research governance

The FREE study was sponsored by ICNARC and phase 2 was co-ordinated by the ICNARC CTU. An ethics application was made to the National Research Ethics Service Committee South Central – Berkshire B on 21 December 2012 and a favourable opinion was received on 20 February 2013.

The CMP has approval under Regulation 5 of the Health Service (Control of Patient Information) Regulations 200224 to process patient identifiable information without consent [approval: PIAG 2–10(f)/2005]. Patient identifiable information processed includes NHS number, date of birth, postcode and sex. A satisfactory Information Governance Toolkit score is required for this approval. A legal agreement is made between ICNARC and the participating units ensuring that the identity of the source of all data (of the hospital, of the unit, of the staff and of the patient) shall remain confidential.

The NIHR CRN Portfolio details high-quality clinical research studies that are eligible for support from the NIHR CRN in England. The study was adopted onto the NIHR CRN Portfolio on 28 January 2013.

Following central R&D approval, site-specific information forms were submitted for each NHS hospital trust. A study site agreement, based on the model agreement for non-commercial research in the health service, was signed by each participating NHS hospital trust and the sponsor (ICNARC).

To ensure transparency, the study was registered for an ISRCTN number. Registration was confirmed on 28 May 2013 (ISRCTN47363549).

Following guidelines from the NIHR, a Study Steering Committee (SSC), with a majority of independent members, was convened to oversee the study on behalf of the funder (the NIHR Health Services and Delivery Research programme) and the sponsor (ICNARC). The SSC met annually during the study. It was chaired by an independent member and comprised lay members (representing patient and family member perspectives), clinicians (critical care nurses and doctors) and methodologists [the chief investigator (KR) and lead clinical investigator (SW) representing the SMG].

Management of phase 2

EW was responsible for the day-to-day management of phase 2 of the FREE study with support from the study administrators, the study statistician and the SMG.

Study management group

The SMG was responsible for overseeing day-to-day management of the study and comprised the chief investigator (KR), lead clinical investigator (SW) and coinvestigators (DAH, SH, DKH, LH, EM, AR, MR). The SMG met regularly throughout the study to ensure adherence to the study protocol and to monitor the conduct and progress of the study.

NHS support costs

Resources equivalent to 0.39 whole-time equivalent of a Band 6 research nurse for each participating unit were agreed with the lead CLRN on 26 February 2013. This was based on an average of 385 patient admissions per year, staying in critical care for 24 hours or more, with an estimated average of 2.5 family members per patient. Participating units, assisted by the ICNARC CTU, negotiated resources required locally for the study with their respective R&D departments and CLRNs.

Changes to the protocol

Following favourable opinion of the study protocol from the research ethics committee on 20 February 2013, four non-substantial amendments were submitted and given favourable opinions. In summary these were:

• amendment 1 (April 2013) – minor semantic changes and formatting to the consent form, UK FS-ICU-24 (see Appendix 2) and QODD (see Appendix 3) questionnaire

• amendment 2 (July 2013) – further minor semantic changes to the consent form

• amendment 3 (October 2013) – further minor semantic changes made to the UK FS-ICU-24

• amendment 4 (February 2014) – revision to the estimated sample size (see Sample size) and a change to the chair of the SSC.

Patient and public involvement

Engagement with patients and their family members was vital to ensuring the success of the cohort study. A former critical care patient (LH) and a family member of a former critical care patient (MR) were coinvestigators on the FREE study proposal and as such contributed to the design, conduct, analysis and interpretation of the study. Both were members of the SMG and were closely involved in the conduct and monitoring of the study and are coauthors of this report. The charity ICUsteps provided assistance in identifying family members to be independent members of the SSC.

Intensive care unit recruitment

The study aimed to recruit a representative sample of 20 adult general ICUs from England, Wales and Northern Ireland. Adult general ICUs were defined as ICUs or combined intensive care/high-dependency units. Stand-alone high-dependency units and specialist ICUs (e.g. neurosciences, cardiothoracic) were not eligible for participation in the study. The criteria for inclusion were:

• active participation in the CMP, defined as submission of data no later than 6 weeks after the end of each quarter and returning corrected data validation reports no later than 6 weeks after receipt

• agreement from the local principal investigator (PI) to recruit up to four eligible family members of consecutive patients staying in the unit for 24 hours or more

• provision of timely data on recruited family members entered onto a secure, dedicated, web-based data entry system (web portal)

• commitment to recruit participants for a minimum of 12 months.

All units actively participating in the CMP were invited to express interest in taking part in the study. A stratified random sampling process was chosen to select 20 representative adult general ICUs and to minimise selection bias.

Intensive care units that expressed an interest in taking part in the study were divided by geographical location: England, Wales and Northern Ireland. England, as the largest country, was subdivided into four regions: north-east, north-west, south-east and south-west. Within each country/region, ICUs were divided into those located in university and non-university hospitals and then into large and small ICUs (based on the median number of beds of all ICUs in the CMP). One ICU and one reserve ICU were randomly selected from each group (Table 2). Then the 20 randomly selected ICUs were formally invited to take part in the study and asked to sign a confirmation of participation agreement before submission of the site-specific information form to the local R&D. If an ICU declined the invitation to take part in the study, then the reserve ICU was invited.

Given that intensive care nurses interact daily with patients and their family members, participating ICUs were encouraged, where possible, to appoint a senior critical care nurse as the local PI to lead and oversee the conduct of the study.

Intensive care unit management

Family member recruitment

The study procedures for recruitment and follow-up of family members are summarised in Figure 1.

FIGURE 1.

Summary of study procedures for recruitment and follow-up of family members.

Follow-up

Patients for whom at least one family member had been recruited were followed up until discharge from the ICU. Approximately 3 weeks after the patient had been discharged from, or had died in, the ICU, a questionnaire pack (see Appendix 6) was sent from the ICNARC CTU, by post, to the family member(s). Following evidence-based practice25 for maximising responses to postal surveys, the questionnaire pack included a cover letter, the participant information sheet (see Appendix 5), the UK FS-ICU-24 (see Appendix 2), a ‘do not wish to participate’ form, a stamped addressed return envelope and a pen. During the first month of recruitment, family members of non-surviving patients were additionally sent the QODD questionnaire23 (see Appendix 3).

If no response was received within 4 weeks after the first questionnaire was posted, a reminder letter was sent with another questionnaire pack. In consultation with the research ethics committee, there was no further follow-up of family members thereafter. For questionnaire packs returned indicating that the recipient was not known at the address, the postal address was checked with the study team at the recruiting ICU.

Follow-up of family members ended on receipt of a completed (or blank) questionnaire, a completed ‘do not wish to participate’ form or notification to the ICNARC CTU by telephone or e-mail that the family member wished to withdraw from the study, or once the second questionnaire pack had been sent.

Data collection

A secure web portal, hosted by ICNARC, was set up to enable staff at participating ICUs to enter patient and family member data. Data for the FREE study were collected to:

• enable identification of patients staying in the ICU for 24 hours or more during the study period

• record the name and postal address of family members who had consented to take part in the study

• enable a questionnaire pack to be sent to family members 3 weeks following the patient’s discharge from, or death in, the ICU

• describe the sociodemographic profile of family members who consented to take part in the study

• enable linkage to the CMP database to monitor recruitment to the study and adherence to the protocol at participating ICUs.

Monitoring recruitment of family members

Recruitment of family members was closely monitored. Data entered onto both the secure web portal and the ICU screening logs were checked weekly against the estimated number of patient admissions to the ICU based on CMP data. The number of family members recruited in the previous 7 days was compared with the weekly average recruited. Any discrepancies or other concerns relating to recruitment were discussed with the local PI and ICU study team.

Every month, an ICU recruitment report, based on data entered onto the secure web portal and the ICU screening log, was produced and reviewed with the local PI. The report included the total number of patient admissions (who stayed for 24 hours or longer) and the total number of eligible family members who were and were not recruited during the previous month and how these compared with the monthly averages. The percentage of patients for whom no family member was recruited, particularly where family members had been missed (i.e. not approached about the study), was highlighted and discussed with the local PI.

Data management

Data management was an ongoing process. Data collected for the study were monitored and checked throughout the recruitment period to ensure that data were as complete and accurate as possible. The secure web portal was monitored daily by the study team at the ICNARC CTU to ensure that data entry at participating ICUs was timely, both to enable questionnaire packs to be sent out on time to family members and to allow time for translation of the UK FS-ICU-24 and accompanying documents, where required. Details of family members (name and postal address) recruited were checked for completeness to avoid unnecessary delays in sending out questionnaires.

Every quarter, data entered on the secure web portal and the ICU screening logs were linked to data in the CMP database. Any queries relating to discrepancies in the number of patient admissions (who stayed for 24 hours or more) during the previous quarter or any issues with data linkage were investigated and resolved in consultation with the ICU study team.

Data from completed questionnaires were entered centrally into a secure database at the ICNARC CTU following a standard operating procedure. All identifiable information, such as names (e.g. of patients, family members or critical care staff members), were removed. If a family member provided a comment rather than ticking one of the response options, the comment was recorded and reviewed by two members of the SMG. All queries relating to data entry were reviewed by two members of the SMG (SH/EW) and any disagreement was reviewed and discussed with a third (KR).

Quality checks of data entered were conducted every 2 months. A 20% random sample of questionnaires entered was checked for accuracy of data entry and any issues with data quality were addressed with the ICNARC CTU team. All fields in the database with missing data were checked against the paper UK FS-ICU-24.

Network support

To maintain the profile of the study, regular updates on study progress were provided at quarterly meetings of the NIHR CRN Critical Care Specialty Group and at local CLRN meetings. In addition, updates were provided at national meetings, such as the Annual Meeting of the CMP and the UK Critical Care Research Forum.

Sample size

The duration of recruitment was 1 year, chosen to avoid bias from seasonal variation. Data from the CMP database indicated that an average of 520 patients are admitted per ICU per year. Of these, 74% (385 patients per ICU per year) stayed at least 24 hours, corresponding to approximately 7700 patients across 20 ICUs. No data existed on the average number of family members per patient who visit the ICU. It was assumed that, with an average of 2.5 family members per patient, approximately 19,250 family members would be recruited over a 12-month recruitment period. Assuming a 66% response rate to the postal questionnaire,18 it was estimated that a total sample of approximately 12,700 responses associated with 6700 patients would be received. Using available FS-ICU-24 data, published and unpublished, mean baseline satisfaction domain scores of 80 with standard deviation (SD) 20 were expected. This sample size would therefore give > 90% power to detect (p < 0.01) a binary patient factor present in 10% of the patient population associated with an increase or decrease in domain score of 4 points.

The estimated sample size was revised following completion of the first full quarter of recruitment. Revised targets were calculated for each site based on recruitment of an average of two family members for 80% of patients staying at least 36 hours in the ICU, allowing that on average 10% of patients may have no family members visit. For the purpose of the revised estimate, 36 hours was chosen, instead of 24 hours, to allow for family members of short-term patients who might not have an opportunity to visit during the period of 24–36 hours. Based on the revised ICU targets, it was expected that 14,200 family members (of 7100 patients) would be recruited. Assuming a 60% response rate, it was estimated that a total sample of approximately 8500 responses associated with 5500 patients would be achieved. This sample size would retain > 90% power to detect (p < 0.01) a binary patient factor present in 10% of the patient population associated with an increase or decrease in domain score of 4 points.

Statistical analysis

A descriptive analysis was conducted to describe the:

• representativeness of ICUs that participated in the study (ICU characteristics)

• duration of participation of ICUs in the study

• recruitment and follow-up of family members, overall and by participating ICU

• patient characteristics and outcomes associated with recruitment and non-recruitment of family members, overall and by participating ICU

• patient characteristics and outcomes associated with the number of family members per patients recruited, overall and by participating ICU

• characteristics of family members recruited

• time taken from first mailing of the questionnaire pack to receipt of a completed questionnaire

• response rate from family members to the questionnaire, overall and by ICU

• patient characteristics and outcomes associated with the response, or non-response, of family members to the questionnaire

• family member characteristics associated with response, or non-response, to the questionnaire.

Discrete variables were summarised as numbers and percentages, which were calculated according to the number of ICUs for which, or individuals for whom, data were available; where values were missing, the denominator is stated in the table. Continuous variables were summarised by standard measures of central tendency and dispersion: mean and SD and/or median and interquartile range (IQR).

Adult general intensive care units

Of the 210 adult general ICUs participating in the CMP in November 2012, 142 (67.6%) expressed an interest in taking part in the study. Twenty ICUs, plus 20 reserve ICUs, were randomly selected to ensure a representative sample of ICUs with respect to geographical location, hospital type (university/non-university) and size of ICU. The 20 ICUs were formally invited to take part in the study and 19 of them accepted. One ICU declined to take part and the corresponding reserve ICU was contacted and accepted the invitation.

Of the 20 ICUs, 19 obtained local R&D approval and the study opened to recruitment on 28 May 2013. One ICU withdrew from the study on 5 July 2013, as it was unable to secure adequate resources (NHS support costs) from its CLRN for the study. The corresponding reserve ICU accepted the invitation to take part and opened to recruitment on 4 September 2013. The remaining 19 ICUs opened to recruitment between 28 May and 8 July 2013 (Figure 2). Delays to opening included issues relating to confirmation of NHS support costs from the CLRN, delays in obtaining R&D approval and delays in local set-up, such as staff training.

FIGURE 2.

Duration of participation of critical care ICUs.

One ICU closed to recruitment on 10 December 2013 because local resource issues meant that the ICU had to withdraw temporarily from participation in the CMP. As a result, no CMP data from this ICU were available for the study analysis. The remaining 19 ICUs continued recruiting family members of patients admitted to the ICU up to 30 June 2014 (Figure 2). The last family member was recruited on 29 July 2014. Complete CMP data for the period 28 May 2013 to 30 June 2014 were received from the 19 ICUs.

Patients

Between 28 May 2013 and 30 June 2014 there were 18,757 admissions to the 20 participating ICUs, of which 1677 were readmissions to the ICU during the same hospital stay. Of the 17,080 patients admitted to critical care during the study period, 4341 (25.4%) stayed for less than 24 hours. Of the remaining 12,739 patients (74.6%), who stayed in the ICU for 24 hours or more, nine were excluded from the study because they were discharged from the ICU after 1 August 2014. This was because there was insufficient time before the database was closed (on 12 September 2014) for a questionnaire to be posted to and completed by their family members. In total, data from 12,730 patients were included in the analysis.

Of these 12,730 patients, there were 5788 (45.5%) for whom no family member was recruited because there were no eligible family members or no eligible family members visited the patient during their stay in the ICU (n = 2200 patients); or the family members were not approached about the study or were approached but not recruited because, for example, they did not visit the patient again in the ICU (n = 3588). Family members of 6942 (54.5%) patients were invited to take part in the study. Of these, the family members of 547 patients refused the invitation. For 15 patients, family members were recruited but subsequently withdrew because they no longer wished to take part or were withdrawn because they were found to be ineligible for the study (Figure 3).

FIGURE 3.

Flow of patient admissions.

Overall, at least one family member was recruited for 60.6% (n = 6380) of the 10,530 patients who stayed in the ICU for 24 hours or more and who were visited in the ICU by one or more eligible family members. Recruitment varied across ICUs, ranging from 41.2% to 79.4% of patients with one or more eligible family members recruited (Table 6).

No family member(s) were recruited for 39.4% of patients because eligible family members were not approached (n = 3588, 34.1%), the family member(s) refused the invitation to take part (n = 547, 5.2%) or the family member(s) withdrew or were withdrawn from the study (n = 15, 0.1%). There was considerable variation across ICUs in the reasons reported for non-recruitment of family members (Table 7). For example, the proportion of patients whose family members were not approached ranged from 15.7% to 56.9% of patients.

The characteristics and outcomes of patients for whom at least one family was recruited, compared with those of patients for whom no family member was recruited, are presented in Table 8. Patients with no family members recruited were more likely to have been admitted to ICU following elective surgery (27.7%) than patients for whom at least one family member was recruited (18.5%) and were less likely to have been mechanically ventilated during their first 24 hours in the ICU (36.9% vs. 48.1%, respectively). In addition, their median (IQR) ICNARC predicted risk of death was 0.08 (0.03–0.28) compared with 0.16 (0.05–0.37) for patients for whom at least one family member was recruited. Even so, the ICU mortality rate for patients for whom no family members were recruited was higher than patients for whom at least one family member was recruited (14.7% vs. 10.9%). This is probably because the group of patients for whom no family members were recruited included a mix of low-risk patients (e.g. admitted following elective surgery), who tend to have a short stay (24 hours or less) in the ICU, and very sick (high-risk) patients, whose family members were not approached about the study because of the severity of their illness. During the study, ICU staff reported that family members of very sick patients, particularly those who were clearly distressed, were often not approached about taking part.

Overall, an average of two family members per patient were recruited. The first family member was recruited within a median (IQR) of 2 (1–3) days of the patient’s admission to the ICU. However, there was considerable variation across ICUs in the number of family members per patient recruited (Table 9).

Family members

Response rate

Of the 12,346 family members recruited, 12,303 (99.7%) were sent a questionnaire pack between 21 and 50 days following the patient’s discharge or death (allowing for delays in this information being entered onto the secure web portal).

The remaining 43 (0.3%) family members were not sent a questionnaire pack because either information (e.g. postal address) provided was incomplete or the date the patient was discharged from the ICU was not provided by the ICU staff or was provided more than 50 days after the patient had been discharged from, or had died in, the ICU.

Of the 12,303 family members who were sent a questionnaire pack, no response was received from 4634 (37.7%) family members, the pack was returned as ‘not known at this address’ (and no further contact information was available from the ICU) for 55 (0.4%) family members and 441 (3.6%) family members refused the invitation to complete the questionnaire. A total of 7173 (58.3%) family members completed and returned the questionnaire (Figure 4).

FIGURE 4.

Family members’ response to postal questionnaire.

A total of 55 family members requested that the questionnaire and accompanying documents be translated (Table 13). Of these 55 family members, 14 (25.5%) completed the questionnaire, 1 (1.8%) refused, 40 (72.7%) did not respond and for 3 (5.5%) the questionnaire was returned as ‘not known at this address’.

TABLE 13 - Languages requested

Language	Number of requests
Urdu	14
Welsh	8
Punjabi	5
Polish	4
Tamil	4
Gujarati	3
Romanian	3
Bengali	2
Chinese (Cantonese)	2
Lithuanian	2
Portuguese	2
Arabic	1
Greek	1
Slovakian	1
Spanish	1
Thai	1
Turkish	1

The time, in days, from the questionnaire pack being sent to receipt of a completed questionnaire is presented in Figure 5. Overall, questionnaires were returned within a median of 10 (IQR 6–31) days. Most were returned after the first questionnaire pack was sent, although there was a modest increase in the response rate after the second questionnaire pack was posted; a second questionnaire pack was sent to just over half (54.9%) of family members 4 weeks after the first questionnaire pack.

FIGURE 5.

Time taken (in days) from first mailing to receipt of a completed questionnaire.

There was variation across ICUs in the proportion of family members who completed the questionnaire, ranging from 48.9% to 73.8% of the family members recruited. Although the overall rate of refusal was low (3.6%), there was some variation across ICUs, ranging from 1.2% to 5.2% of family members refusing to complete the questionnaire (Table 14).

TABLE 14 - Response rate of family members, by ICU

[N], number of patients for whom the information was available.

ICU	Questionnaire pack 1 sent (%)	Questionnaire pack 2 sent (%)	Completed UK FS-ICU-24 received (%)	Reason for no UK FS-ICU-24 received			Time taken (days) for receipt of UK FS-ICU-24, mean (SD), median (IQR) [N]
				No response (%)	Return to sender (not delivered) (%)	Refusal (%)
A	682 (99.1)	375 (54.5)	386 (56.6)	265 (38.9)	4 (0.6)	27 (4.0)	18.1 (18.0), 10.0 (6.0–29.0) [386]
B	531 (95.3)	281 (50.4)	327 (61.6)	184 (34.7)	2 (0.4)	18 (3.4)	18.2 (19.9), 10.0 (6.0–30.0) [327]
C	449 (100.0)	239 (53.2)	258 (57.5)	172 (38.3)	3 (0.7)	16 (3.6)	18.2 (17.5), 11.0 (7.0–25.0) [258]
D	646 (100.0)	376 (58.2)	354 (54.8)	261 (40.4)	4 (0.6)	27 (4.2)	19.2 (20.0), 10.5 (7.0–30.0) [354]
E	481 (100.0)	224 (46.6)	321 (66.7)	138 (28.7)	2 (0.4)	20 (4.2)	17.3 (16.7), 10.0 (6.0–24.0) [321]
F	873 (100.0)	469 (53.7)	510 (58.4)	315 (36.1)	3 (0.3)	45 (5.2)	17.4 (17.2), 9.0 (6.0–29.0) [508]
G	1051 (99.9)	639 (60.7)	554 (52.7)	464 (44.1)	5 (0.5)	28 (2.7)	18.9 (18.6), 10.0 (6.0–33.0) [554]
H	711 (100.0)	437 (61.5)	369 (51.9)	314 (44.2)	4 (0.6)	24 (3.4)	19.8 (20.2), 12.0 (7.0–33.0) [369]
I	310 (100.0)	135 (43.5)	200 (64.5)	94 (30.3)	–	16 (5.2)	15.5 (15.2), 9.0 (6.0–18.0) [200]
J	474 (100.0)	229 (48.3)	303 (63.9)	153 (32.3)	3 (0.6)	15 (3.2)	17.7 (17.5), 10.0 (6.0–26.0) [302]
K	392 (98.5)	189 (47.5)	265 (67.6)	116 (29.6)	1 (0.3)	10 (2.6)	20.1 (24.6), 10.0 (6.0–28.0) [265]
L	370 (100.0)	169 (45.7)	273 (73.8)	88 (23.8)	–	9 (2.4)	19.2 (17.5), 11.0 (7.0–34.0) [272]
M	789 (100.0)	434 (55.0)	474 (60.1)	281 (35.6)	3 (0.4)	31 (3.9)	19.6 (19.3), 11.0 (7.0–33.0) [473]
N	1008 (100.0)	585 (58.0)	540 (53.6)	436 (43.3)	9 (0.9)	23 (2.3)	18.3 (18.9), 10.0 (6.0–30.0) [539]
O	325 (100.0)	180 (55.4)	202 (62.2)	112 (34.5)	–	11 (3.4)	19.9 (18.5), 11.5 (6.0–33.0) [202]
P	1034 (99.8)	641 (61.9)	506 (48.9)	471 (45.6)	5 (0.5)	52 (5.0)	20.6 (19.4), 13.0 (6.0–34.0) [504]
Q	408 (100.0)	223 (54.7)	251 (61.5)	148 (36.3)	3 (0.7)	6 (1.5)	19.6 (19.9), 10.0 (7.0–33.0) [250]
R	750 (99.7)	410 (54.5)	442 (58.9)	268 (35.7)	–	40 (5.3)	19.2 (22.1), 9.0 (7.0–31.0) [442]
S	255 (100.0)	150 (58.8)	154 (60.4)	96 (37.6)	2 (0.8)	3 (1.2)	21.9 (21.9), 13.0 (6.0–34.0) [154]
T	764 (100.0)	397 (52.0)	484 (63.4)	258 (33.8)	2 (0.3)	20 (2.6)	19.4 (22.8), 10.0 (6.0–30.0) [483]
Total	12,303 (99.7)	6782 (54.9)	7173 (58.3)	4634 (37.7)	55 (0.4)	441 (3.6)	18.9 (19.5), 10.0 (6.0–31.0) [7163]

Initial psychometric assessment

The initial psychometric assessment was conducted using responses from family members of patients who were discharged from, or died in, the ICU between the start of the study and 12 July 2013 (i.e. to whom initial questionnaires were sent out up to 2 August 2013) and with a returned questionnaire received by the ICNARC CTU by 9 September 2013.

Only one family member response per patient was used in the initial psychometric assessment. Where multiple family member responses associated with the same patient had been received, the first family member (according to the order entered on the FREE study secure, web-based data entry system, corresponding to the order in which they visited the patient after 24 hours) was used in the analysis.

Item responses were rescaled and, where relevant, reversed, according to the developer’s rules, so that each response was on a scale from 0 (least satisfied) to 100 (most satisfied). The overall family satisfaction score and domain scores (satisfaction with care and satisfaction with decision-making) were calculated by averaging the item responses for the items included overall/within each domain, provided at least 70% of items had been completed (i.e. 17 out of 24 for the overall family satisfaction score, 10 out of 14 for satisfaction with care and 7 out of 10 for satisfaction with decision-making). Code for processing of FS-ICU-24 responses was translated from SPSS (Statistical Package for the Social Sciences, version not known, IBM Corporation, Armonk, NY, USA) syntax files available from www.thecarenet.ca.

Patients with a family member included in the initial psychometric assessment were described by their age in years (mean with SD, and number and percentage by decade of age), sex, surgical status (elective/scheduled surgery, emergency/urgent surgery or non-surgical), ICNARC physiology score (mean and SD),27 APACHE II score (mean and SD),26 ICU outcome (alive or dead) and ICU length of stay (median and quartiles for survivors and non-survivors). Family members included in the initial psychometric assessment were described by their age in years (mean with SD, and number and percentage by decade of age), sex and relationship to the patient (partner, parent, child, sibling, other relative or other non-relative).

The following descriptive statistics were reported for each individual item on the UK FS-ICU-24:

• number and percentage of missing responses

• number and percentage of responses in each category

• median and quartiles of item response.

The following descriptive statistics were reported for the overall family satisfaction score and for each domain score (satisfaction with care and satisfaction with decision-making):

• number and percentage of missing scores (response to fewer than 70% of items)

• number and percentage with floor (0) and ceiling (100) scores

• median, quartiles, minimum and maximum of score.

The following criteria were used to flag items for possible removal:

• items with high rates of non-response or ‘not applicable’ responses (> 10%), suggesting irrelevance or lack of comprehensibility or acceptability

• items with poorer discrimination (> 70% selecting the lowest or highest category)

• redundant items (item to own scale correlation > 0.8, corrected for overlap)

• items measuring a construct other than that intended (component loading < 0.4 in confirmatory principal components analysis).

Principal components analysis was performed using expectation-maximisation estimation of the covariance matrix to account for missing data28 with columnwise average nominal sample size. 29

Items meeting the above criteria were discussed among the SMG (including DKH, as representative of the original FS-ICU authors) before a final decision on removal.

Internal consistency (reliability) was evaluated through calculation of the item to own scale correlations and Cronbach’s alpha for each domain. 30 Item to own scale correlations < 0.4 or Cronbach’s alpha < 0.8 were taken as indicative of lack of internal consistency.

Confirmatory factor analysis techniques were used to test the goodness of fit of the two-factor solution for the FS-ICU-24 – that is, the division into the domains of satisfaction with care and satisfaction with decision-making – developed in the North American validation study. 11 Structural equation models were used to estimate the factor loadings for the two-factor solution, using full information maximum likelihood assuming data were missing at random. 31 Goodness of fit was assessed by the likelihood ratio chi-squared statistic versus the saturated model, root mean-squared-error of approximation (RMSEA), comparative fit index (CFI), Tucker–Lewis index (TLI) and coefficient of determination (CD). Values of RMSEA < 0.08, CFI > 0.9 and TLI > 0.9 were taken to indicate a good fit. 32

The criterion validity of the UK FS-ICU-24 among family members of non-survivors was assessed by comparison with the QODD questionnaire. The hypotheses tested were that higher values for the overall family satisfaction score and the two domain scores of the UK FS-ICU-24 would be highly correlated (Pearson correlation coefficients > 0.6) with the overall QODD score, and that scores on four specific items in the QODD (pain control, breathing comfort, care by doctors, care by all providers) would be moderately correlated (Spearman’s rank correlation coefficients 0.4–0.6) with the satisfaction with care domain score. Finally, responses to specific items on the QODD were compared with specific items on the UK FS-ICU-24 (assessed by Spearman’s rank correlation coefficients), as detailed in Table 17.

Full psychometric assessment

The full psychometric assessment was conducted on the final FREE study data set. As with the initial psychometric assessment, only one family member response per patient was used in the full psychometric assessment; however, the approach to selecting the family member was revised to prioritise those most likely to give complete and informative responses. Where multiple family member responses associated with the same patient had been received, priority was given to the family member(s) identified by the site as the patient’s next of kin, followed by the patient’s spouse or partner, then parents or children and finally other family members. Where more than one family member was given equal highest priority, the first family member to visit (according to the order entered on the FREE study secure, web-based data entry system, corresponding to the order in which they visited the patient after 24 hours) was used in the analysis.

Completed questionnaires were coded and scored as for the initial psychometric assessment. However, following the results of the initial psychometric assessment, an alternative, lower, threshold of 60% of items having been completed (i.e. 15 out of 24 for the overall family satisfaction score, 9 out of 14 for satisfaction with care and 6 out of 10 for satisfaction with decision-making) was considered for calculation of the scores. This lower threshold was considered to permit family members who did not answer the four questions relating to involvement in decision-making to be included in the analysis.

The patients, family members, item responses and scores were described as for the initial psychometric assessment. Items with high item non-response rates (> 10%) were taken as indicative of possible irrelevance or lack of comprehensibility or acceptability. Items with poorer discrimination (> 70% selecting the lowest or highest category) were taken as indicative of possible floor/ceiling effects. Item to own scale correlation > 0.8, corrected for overlap, was taken as indicative of possible redundancy. Component loadings < 0.4 in a confirmatory principal components analysis, conducted using the same methods as for the initial psychometric assessment, were taken as indicative of an item measuring a construct other than intended.

Internal consistency was evaluated through calculation of the item to own scale correlations and Cronbach’s alpha for each domain. 30 Item to own scale correlations < 0.4 or Cronbach’s alpha < 0.8 were taken as indicative of lack of internal consistency.

Confirmatory factor analysis techniques were used to test the goodness of fit of the two-factor solution for the FS-ICU-24 developed in the North American validation study11 using the same methods as for the initial psychometric assessment.

Exploratory factor analysis was undertaken using polychoric correlations, which assume that ordinal data are representative of an underlying continuous distribution,33 considering models with two or three principal factors. Initially orthogonal (varimax) rotations34 were applied and subsequently oblique (promax, power 3) rotations were considered to allow for correlation between dimensions. 35 Multiple imputation using fully conditional specification36 was used to account for item-level non-response.

Performance-importance plots were used to identify items that may form likely candidates for quality improvement. 37 Each point in the performance-importance plot corresponds to one item on the UK FS-ICU-24. The Spearman’s rank correlation between each item and the overall family satisfaction score or domain score is plotted on the y-axis against the percentage of respondents rating the item as ‘excellent’ on the x-axis. The plot is divided into quadrants at the mean values on the x- and y-axes. Points in the upper left quadrant correspond to items that are highly correlated with overall satisfaction but have a lower than average proportion of ‘excellent’ ratings and therefore show potential for improvement.

Initial psychometric assessment

At the point of data lock for the initial psychometric assessment, 992 family members had been consented for 481 patients who were discharged from, or died in, 19 participating ICUs between 28 May 2013 and 12 July 2013. Of these, 565 (57%) family members had returned questionnaires, representing 357 (74%) patients from 18 units (the one unit to have no questionnaires returned started the study only on 8 July 2013 and had only two consented family members). The first family member for each patient (based on the order in which sites identified them) was included in the analysis (n = 357).

Characteristics of the patients and family members included in the initial psychometric assessment are reported in Table 18.

Responses to individual items on the UK FS-ICU-24 are reported in Table 19 (satisfaction with care), Table 20 (satisfaction with decision-making) and Table 21 (satisfaction with decision-making, if your family member died in the ICU). Characteristics of the UK FS-ICU-24 overall family satisfaction score and domain scores are summarised in Table 22, and the distributions of the scores are shown in Figure 6. Among the 284 (80%) family members for whom both domain scores could be calculated, correlation between the two domains was 0.790 (Figure 7).

FIGURE 6.

Distribution of UK FS-ICU-24 overall family satisfaction score and domain scores in the initial psychometric assessment.

FIGURE 7.

Correlation between UK FS-ICU-24 domain scores in the initial psychometric assessment (n = 284). The size of circle is proportional to the number of respondents.

A total of seven items were flagged for discussion because of high levels of ‘not applicable’ responses or non-response:

• three items from satisfaction with care:

  • symptom management – breathlessness (12.9% not applicable, 2.0% no response)

  • symptom management – agitation (12.0% not applicable, 2.0% no response)

  • the atmosphere (mood) in the ICU waiting room (9.8% not applicable, 1.4% no response)

• and four items from satisfaction with decision-making:

  • Did you feel included in the decision-making process? (25.5% not applicable, 1.7% no response)

  • Did you feel supported during the decision-making process? (28.0% not applicable, 1.4% no response)

  • Did you feel you had control over the care of your family member? (21.6% not applicable, 1.1% no response)

  • When making decisions, did you have adequate time to have your concerns addressed and questions answered? (42.0% not applicable, 1.7% no response).

Following discussion among the SMG, it was agreed that there were good reasons to expect ‘not applicable’ responses for these items – some patients would not have experienced particular symptoms, some family members would not have used the waiting room and some family members would not have been involved in the decision-making process – and consequently that these did not represent a lack of comprehensibility or acceptability and that the items were relevant to a substantial proportion of family members. All these items were therefore retained in the questionnaire.

Only one item, ‘when making decisions, did you have adequate time to have your concerns addresses and questions answered?’, had > 70% of respondents selecting either the highest or lowest option. However, this was the only binary item on the questionnaire and therefore it was not considered to represent a lack of discrimination.

Eleven items were flagged for discussion for potential redundancy:

• six items from satisfaction with care:

  • concern and caring by ICU staff [towards the patient] (item to own scale correlation 0.805)

  • consideration of the family member’s needs (0.844)

  • emotional support [for the family member] (0.826)

  • concern and caring by ICU staff [towards the family member] (0.837)

  • co-ordination of care (0.820)

  • skill and competence of ICU nurses (0.814)

• and five items from satisfaction with decision-making:

  • ease of getting information (0.821)

  • understanding of information (0.811)

  • honesty of information (0.804)

  • completeness of information (0.857)

  • consistency of information (0.836).

All of these items were only just over the threshold for discussion (0.8) and it was felt that all items were capturing sufficiently different aspects of satisfaction for all to be retained.

Only one item, ‘the atmosphere (mood) in the ICU waiting room’ had a component loading of < 0.4 in the confirmatory principal components analysis (Table 23). This was only just below the threshold and was considered an important area of potential lower satisfaction.

No concerns were identified over the reliability (internal consistency) of the UK FS-ICU-24. All item to own scale correlations were greater than 0.4 (see Tables 19 and 20) and Cronbach’s alpha exceeded 0.8 for the overall family satisfaction score (0.96) and for both domain scores (0.95 and 0.94 for satisfaction with care and satisfaction with decision-making, respectively; see Table 22).

The confirmatory factor analysis (Table 24 and Figure 8) indicated that the North American two-factor solution for the FS-ICU-24 (corresponding to the two underlying constructs of satisfaction with care and satisfaction with decision-making) was not a good fit to the FREE study data (RMSEA > 0.08 and CFI and TLI < 0.9).

FIGURE 8.

Confirmatory factor analysis: structural model and factor loadings from the initial psychometric assessment.

Of 44 included family members of a patient who died in the ICU, 36 (82%) returned a completed or partially completed QODD questionnaire and were included in the validity analysis. No concerns were identified over the validity of the UK FS-ICU-24 among family members of patients who died in the ICU. The majority of hypothesised relationships between the scores and individual items of the two instruments were supported by the data (Table 25). In particular, the overall family satisfaction score and both domain scores of the UK FS-ICU-24 correlated with the QODD total score (correlation coefficients 0.614, 0.622 and 0.641 for the overall family satisfaction score, satisfaction with care domain score and satisfaction with decision-making domain score, respectively; Figure 9).

FIGURE 9.

Correlation between UK FS-ICU-24 scores and QODD total score.

Full psychometric assessment

From the final FREE study data set, selecting one family member per patient following the priority order as set out above resulted in a cohort of 4615 family members for the full psychometric assessment. Characteristics of the included patients and family members were similar to those from the initial psychometric assessment, although the revised approach to selecting the family member (to allow for patients with more than one responding family member) resulted in a higher proportion of partners and parents and lower proportions of other relatives and non-relatives (Table 26).

Responses to individual items on the UK FS-ICU-24 are reported in Table 27 (satisfaction with care), Table 28 (satisfaction with decision-making) and Table 29 (satisfaction with decision-making, if your family member died in the ICU). Table 30 reports the results of a question added to the original FS-ICU questionnaire, ‘how satisfied were you with the amount of control you had over the care of your family member?’. There was moderate correlation with the original FS-ICU question, ‘did you feel you had control over the care of your family member?’ (Spearman’s rank correlation 0.529, p < 0.001). However, there was some notable discordance. For example, almost one-third of family members who indicated that they ‘felt really out of control’ indicated that they were either ‘completely satisfied’ or ‘very satisfied’ with the amount of control they had.

Using a 60% response threshold for inclusion in the overall family satisfaction score and domain scores made minimal difference to the characteristics of the scores but permitted an additional 508 respondents (11%) to be included in the domain score for satisfaction with decision-making and small increases in the numbers of respondents included in the other scores. Results are therefore presented using the 60% threshold. Characteristics of the UK FS-ICU-24 overall family satisfaction score and domain scores are summarised in Table 31, and the distributions of the scores are shown in Figure 10. Among the 4319 (94%) family members for whom both domain scores could be calculated, correlation between the two domains was 0.770 (Figure 11).

FIGURE 10.

Distribution of UK FS-ICU-24 overall family satisfaction score and domain scores in the full psychometric assessment.

FIGURE 11.

Correlation between UK FS-ICU-24 domain scores in the full psychometric assessment (n = 4319). Size of circle proportional to number of respondents.

Levels of non-response or ‘not applicable’ responses were slightly lower in the full psychometric assessment than in the initial psychometric assessment, because the selection process was revised to prioritise family members who were more likely to provide complete responses. However, the same items were identified as having higher levels of non-response, although ‘the atmosphere (mood) in the ICU waiting room’ no longer reached the 10% threshold. The item ‘concern and caring by ICU staff’ was identified as having a possible ceiling effect, with 72% of respondents rating this as ‘excellent’. The same items as in the initial psychometric assessment were identified as having possible redundancy (item to own scale correlation > 0.8).

All component loadings were ≥ 0.4 in the confirmatory principal components analysis (Table 32). However, a number of items were loading on both components, suggesting either correlation between the components and/or the presence of additional important components.

No concerns were identified over the reliability (internal consistency) of the UK FS-ICU-24. All item to own scale correlations were > 0.4 (see Tables 27 and 28) and Cronbach’s alpha exceeded 0.8 for the overall family satisfaction score (0.957) and for both domain scores (0.938 and 0.925 for satisfaction with care and satisfaction with decision-making, respectively; see Table 31), similarly to the initial psychometric assessment.

Substituting the alternative question on satisfaction with the amount of control increased Cronbach’s alpha for the overall family satisfaction score from 0.957 to 0.959 and for the satisfaction with decision-making domain score from 0.925 to 0.930.

The confirmatory factor analysis (Table 33 and Figure 12) confirmed the findings from the initial psychometric assessment that the North American two-factor solution for the FS-ICU-24 was not a good fit to the FREE study data (RMSEA > 0.08 and CFI and TLI < 0.9).

FIGURE 12.

Confirmatory factor analysis: structural model and factor loadings from the full psychometric assessment.

The exploratory factor analysis (Table 34) identified a three-factor solution, which was a better fit to the FREE study data when validated in a structural equation model (Table 35 and Figure 13) than the North American two-factor solution (RMSEA < 0.08; CFI and TLI > 0.9). This comprised the same domain of satisfaction with care, but the domain of satisfaction with decision-making was split into two separate factors: satisfaction with information (corresponding to the first six items) and satisfaction with the decision-making process (final four items). There was considerable correlation between the dimensions; consequently the best-fitting model used an oblique rotation and correlated error terms were permitted in the structural equation model. Correlations between the domain scores for the three domains are shown in Figure 14.

FIGURE 13.

Structural equation model: structure and factor loadings for the FREE study three-factor solution.

FIGURE 14.

Correlation between domain scores for the FREE study three-factor solution.

Performance-importance plots for the overall family satisfaction score and the two original domain scores are shown in Figures 15–17. The performance-importance plot for the overall family satisfaction score suggests that the highest-priority items for quality improvement relate to the provision of information (items 1–6 from the satisfaction with decision-making domain) and how well the ICU staff provide emotional support to the family member (item 4 from the satisfaction with care domain). The item relating to the atmosphere (mood) in the ICU waiting room (item 11 from the satisfaction with care domain) had the lowest proportion of ‘excellent’ ratings, but was ranked a lower priority for quality improvement because of its lower correlation with the overall family satisfaction score and the satisfaction with care domain score.

FIGURE 15.

Performance-importance plot for the UK FS-ICU-24 overall family satisfaction score. Cx denotes item x from the satisfaction with care domain; Dx denotes item x from the satisfaction with decision-making domain.

FIGURE 16.

Performance-importance plot for the UK FS-ICU-24 satisfaction with care domain score. Cx denotes item x from the satisfaction with care domain.

FIGURE 17.

Performance-importance plot for the UK FS-ICU-24 satisfaction with decision-making domain score. Dx denotes item x from the satisfaction with decision-making domain.

Overall family satisfaction score

Across the five populations (Table 37), the median overall family satisfaction score varied from 81.3 for incomplete questionnaires to 87.5 for complete questionnaires. All scores showed skewed distributions towards higher scores (Figure 18).

FIGURE 18.

Distribution of overall family satisfaction scores across populations of questionnaires: (a) all returned questionnaires; (b) complete returned questionnaires; (c) incomplete returned questionnaires; (d) ≥ 70% complete returned questionnaires (traditional approach); and (e) ≥ 60% complete returned questionnaires (modified traditional approach).

Satisfaction with care domain score

Across the five populations (Table 38), the median satisfaction with care domain score varied from 83.3 for incomplete questionnaires to 89.3 for complete questionnaires. All scores showed skewed distributions towards higher scores (Figure 19).

FIGURE 19.

Distribution of satisfaction with care domain scores across populations of questionnaires: (a) all returned questionnaires; (b) complete returned questionnaires; (c) incomplete returned questionnaires; (d) ≥ 70% complete returned questionnaires (traditional approach); and (e) ≥ 60% complete returned questionnaires (modified traditional approach).

Satisfaction with decision-making domain score

Across the five populations (Table 39), the median satisfaction with decision-making domain score varied from 75.0 for incomplete questionnaires to 85.0 for complete questionnaires. Scores showed skewed distributions towards higher scores, although there was more variability in the score for incomplete questionnaires (Figure 20).

FIGURE 20.

Distribution of satisfaction with decision-making domain scores across populations of questionnaires: (a) all returned questionnaires; (b) complete returned questionnaires; (c) incomplete returned questionnaires; (d) ≥ 70% complete returned questionnaires (traditional approach); and (e) ≥ 60% complete returned questionnaires (modified traditional approach).

Satisfaction with information domain score

Across the five populations (Table 40), the median satisfaction with information domain score was 79.2 for all populations of questionnaires except for the incomplete questionnaires, which had a median score of 75.0. Scores generally showed skewed distributions towards higher scores but there was variability in the scores, particularly for the incomplete questionnaires (Figure 21).

FIGURE 21.

Distribution of satisfaction with information domain scores across populations of questionnaires: (a) all returned questionnaires; (b) complete returned questionnaires; (c) incomplete returned questionnaires; (d) ≥ 70% complete returned questionnaires (traditional approach); and (e) ≥ 60% complete returned questionnaires (modified traditional approach).

Satisfaction with the decision-making process domain score

Across the five populations (Table 41), the median satisfaction with the decision-making process domain score varied from 58.3 for incomplete questionnaires to 87.5 for complete questionnaires. There was variability in the distribution of scores for incomplete questionnaires and, although the distribution of scores was generally skewed towards higher scores for the other populations of questionnaires, this was most pronounced for the complete questionnaires (Figure 22).

FIGURE 22.

Distribution of satisfaction with the decision-making process domain scores across population of questionnaires: (a) all returned questionnaires; (b) complete returned questionnaires; (c) incomplete returned questionnaires; (d) ≥ 70% complete returned questionnaires (traditional approach); and (e) ≥ 60% complete returned questionnaires (modified traditional approach).

Selection of data

Completed questionnaires were received from 7173 family members. Of these, 154 family members of patients admitted to one ICU were excluded from further analyses because CMP data were not available for the patients admitted to this ICU. This resulted in a data set of 7019 family member responses for analysis.

Family members who did not return a questionnaire are referred to as ‘non-responders’, those who returned a questionnaire but with missing or ‘not applicable’ responses to one or more of the 24 items as ‘partial responders’ and those who returned a questionnaire with all 24 items completed as ‘complete responders’, used as the reference category.

Statistical analysis

Responses to each individual item in the UK FS-ICU-24 were summarised, with particular attention to ‘not applicable’ and missing responses. The numbers and patterns of missing items were also explored within each domain (satisfaction with care and satisfaction with decision-making) and overall.

Logistic regression models including family member, patient and ICU/hospital characteristics were constructed to identify the determinants that may be associated with (mutually exclusive) patterns of missing items as well as the pattern of any missing item. The outcome for each model was a binary indicator of family members having the particular pattern of missing items.

Methods of imputation were used to complete non- and partial responses. This allows observations with missing items to be included, to address the potential bias and loss of precision that could result from complete case analysis. The imputation of family member characteristics and outcomes is complicated by several issues: the variables are hierarchical in nature, with a small number of family members (level 1) for each patient (level 2) and a large number of patients in each ICU/hospital (level 3). This could complicate the application of traditional approaches to dealing with missing data, taking into account the multilevel structure in the imputation model. In addition, several variables of various types (categorical, continuous and ordinal) at level 1 needed to be imputed. All variables to be included in the analysis were 100% complete at the ICU/hospital and patient level.

When applying multiple imputation to incomplete questionnaire data, one can either impute the incomplete items prior to computing scores (item level) or impute the scores directly from the available scores (scale level). The latter approach imputes satisfaction scores for both partial and non-responders, including the average response to the completed questions as an auxiliary variable for partial responders in an attempt to recapture much of the item-level information that a scale-level approach ignores. Previous work addressing this question44 found that item-level imputation is preferable to scale-level imputation in that it provides eventual estimates that are consistently more reliable. In addition, one of the problems when imputing the scores at the scale level was the asymmetric distribution of the scores and, although transforming is recommended45 to get better imputed values, transformation can yield substantial bias if the transformed variable is not close to normal. 46

We therefore considered two alternative approaches to how the outcome of family satisfaction was imputed:

1. scale-level, following standard rules to average item responses for partial responders with at least 60% of items completed and using multiple imputation to impute satisfaction scores for non-responders (including partial responders completing fewer than 60% of items)

2. item-level, using multiple imputation to impute individual missing items for both partial and non-responders and using the imputed items to calculate satisfaction scores.

Multilevel multiple imputation was used to complete non- and partial responses for outcomes and family member characteristics. Given the natural structure of the data and the planned analysis (a multilevel approach), it was necessary to account for the multilevel structure in the imputation phase to avoid biasing parameter estimates. 47 The imputation model was an extension of the joint modelling approach to mixed numerical and categorical data with a multilevel structure. We used Stata/SE 13.0 (StataCorp LP, College Station, TX, USA) with REALCOM-Impute, a MLwiN 2.15 (MLwiN, Centre for Multilevel Modelling, Bristol, UK) macro supported by Stata that generates imputations for hierarchical data, to generate the multiply imputed datasets. 48

Currently, only two-level hierarchical structures can be handled in REALCOM-Impute, and the FREE study data have a three-level structure. We therefore substituted fixed for random effects at the ICU/hospital level,49 collapsing the data to a two-level structure with a dummy variable for each ICU. Ten completed data sets were imputed and the model of interest was fitted to each of these data sets. The results from the 10 imputed data sets were combined using Rubin’s rule. 50

Satisfaction with care

Of the 14 items that constitute the original satisfaction with care domain score, 13 all had the same response options, ‘excellent/very good/good/fair/poor’, with a further response option of ‘not applicable’. The response options for the final item (satisfaction with the level or amount of health care) were ‘very dissatisfied/slightly dissatisfied/mostly satisfied/very satisfied/completely satisfied’, with no response option for ‘not applicable’. The distributions of responses to each of the 14 items are presented in Figure 23.

FIGURE 23.

Responses to the 14 items constituting the satisfaction with care domain score. N/A, not applicable. (a) How did we treat your family member (the patient)? Q1. Concern and caring by ICU staff? [The courtesy, respect and compassion your family member (the patient) was given.] (b) Q2. Symptom management? (How well the ICU staff assessed and treated your family member’s symptoms) Pain. (c) Breathlessness. (d) Agitation. (e) How did we treat you? Q3. Consideration of your needs? (How well the ICU staff showed an interest in your needs.) (f) Q4. Emotional support? (How well the ICU staff provided emotional support.) (g) Q5. Concern and caring by ICU staff? (The courtesy, respect and compassion you were given.) (h) Teamwork. Q6. Co-ordination of care? (The teamwork of all the ICU staff who took care of your family member.) (i) Nurses. Q7. Skill and competence of ICU nurses? (How well the nurses cared for your family member.) (j) Q8. Frequency of communication with ICU nurses? (How often nurses communicated to you about your family member’s condition.) (k) Doctors. Q9. Skill and competence of ICU doctors? (How well doctors cared for your family member.) (l) The ICU. Q10. The atmosphere (mood) of the ICU was? (m) The waiting room. Q11. The atmosphere (mood) in the ICU waiting room was? (n) Level/amount of health care. Q12. Some people want everything done for their health problems while others do not want a lot done. How satisfied were you with the LEVEL or amount of health care your family member received in the ICU?

Of the 7019 questionnaires received, 5008 (71.4%) had complete responses to all 14 items constituting the satisfaction with care domain. The proportion of missing data (no response) for all items was low but was highest for the items relating to symptom management of the patient (agitation, 1.9%; breathlessness, 1.8%; pain, 0.9%). When combined with a response of ‘not applicable’ – considered as missing when generating the score – the proportion of missing data was higher – again, highest for two of the items about symptom management of the patient (breathless, 12.9%; agitation, 10.7%), followed by the item on atmosphere/mood of the waiting room (8.7%) and the item about emotional support of the family member (8.1%).

When we examined patterns of missing data (Figure 24), we found that most frequently a response was missing for item C2b only (symptom management: breathlessness), item C11 only (the atmosphere/mood of the ICU waiting room) or a combination of items C2b (symptom management: breathlessness) and C2c (symptom management: agitation).

FIGURE 24.

Common patterns of missing data for the satisfaction with care domain.

The distribution of the numbers of items with missing responses is presented in Figure 25.

FIGURE 25.

Distribution of the number of items with missing responses for the satisfaction with care domain.

Satisfaction with decision-making

Six of the 10 items that constitute the original satisfaction with decision-making domain score all had the same response options, ‘excellent/very good/good/fair/poor’, with a further response option of ‘not applicable’. The response options for the remaining four items were as follows:

• Q7 (being included in the decision-making process) – I felt very excluded/I felt somewhat excluded/I felt neither included nor excluded/I felt somewhat included/I felt very included

• Q8 (feeling supported during the decision-making process) – I felt totally unsupported/I felt slightly unsupported/I felt neither supported nor unsupported/I felt supported/I felt very supported

• Q9 (control over the care of the patient) – I felt really out of control and that the health care system took over and dictated the care my family member received/I felt somewhat out of control and that the health care system took over and dictated the care my family member received/I felt neither in control nor out of control/I felt I had some control over the care my family member received/I felt that I had good control over the care my family member received

• Q10 (adequate time for making decisions and to have concerns addressed and questions answered) – I could have used more time/I had adequate time

• Q7–10 all had a further response option of ‘not applicable’.

The distributions of responses to each of the 10 items are presented in Figure 26.

FIGURE 26.

Responses to the 10 items constituting the satisfaction with decision-making domain score. N/A, not applicable. (a) Information needs. Q1. Frequency of communication with ICU doctors? (How often doctors communicated to you about your family member’s condition.) (b) Q2. Ease of getting information? (Willingness of ICU staff to answer your questions.) (c) Q3. Understanding of information? (How well ICU staff provided you with explanations that you understood.) (d) Q4. Honesty of information? (The honesty of information provided to you about your family member’s condition.) (e) Q5. Completeness of information? (How well ICU staff informed you what was happening to your family member and why things were being done.) (f) Q6. Consistency of information? [The consistency of information provided to you about your family member’s condition (did you get a similar story from the doctor, nurse, etc.)]. (g) The process of making decisions. Q7. Did you feel included in the decision-making process? (h) Q8. Did you feel supported during the decision-making process? (i) Q9. Did you feel you had control over the care of your family member? (j) Q10. When making decisions, did you have adequate time to have your concerns addressed and questions answered?

Of the 7019 questionnaires received, 3476 (40.5%) had complete responses for all 10 items constituting the satisfaction with decision-making domain. The proportion of missing data (no response) for all items was low but was highest for the items about the process of making decisions (had adequate time for making decisions, 2.0%; felt control over the care of the patient, 1.8%; felt included in the decision-making process, 1.6%; felt supported during the decision-making process, 1.6%). When combined with a response of ‘not applicable’, the proportion of missing data was high – again, highest for the process of making decisions items (had adequate time for making decisions, 44.8%; felt supported during the decision-making process, 28.6%; felt included in the decision-making process, 26.5%; felt control over the care of the patient, 24.4%).

When we examined patterns of missing data (Figure 27), we found that most frequently a response was missing for item D10 only (‘when making decisions, did you have adequate time to have your concerns addressed and questions answered?’), a combination of items D7 (‘did you feel included inthe decision-making process?’), D8 (‘did you feel supported during the decision-making process?’), D9 (‘did you feel you had control over the care of your family member?’) and D10, or a combination of items D9 and D10.

FIGURE 27.

Common patterns of missing data for the satisfaction with decision-making domain.

The distribution of the numbers of items with missing responses is presented in Figure 28.

FIGURE 28.

Distribution of the numbers of items with missing responses for the satisfaction with decision-making domain.

Satisfaction with information

Six items from within the satisfaction with decision-making domain (items D1–D6) constitute the satisfaction with information domain score. Of the 7019 questionnaires received, 6238 (88.9%) had complete responses for all six items. Although the proportion of missing data (no response) was low, a response was most frequently missing for either D1 only (‘frequency of communication with ICU doctors’) or all six items (Figure 29).

FIGURE 29.

Common patterns of missing data for the satisfaction with information domain.

The distribution of the numbers of items with missing responses is presented in Figure 30.

FIGURE 30.

Distribution of the numbers of items with missing responses for the satisfaction with information domain.

Satisfaction with the decision-making process

Four items from within the satisfaction with decision-making domain (items D7–D10) constitute the satisfaction with the decision-making process domain score. Of the 7019 questionnaires received, 3476 (49.5%) had complete responses for all four items. Most frequently, a response was missing for D10 only (‘when making decisions, did you have adequate time to have your concerns addressed and questions answered?’), all four items – D7 (‘did you feel included in the decision-making process?’), D8 (‘did you feel supported during the decision-making process?’), D9 (‘did you feel you had control over the care of your family member?’) and D10 – or a combination of items D9 and D10 (Figure 31).

FIGURE 31.

Common patterns of missing data for the satisfaction with the decision-making process domain.

The distribution of the numbers of items with missing responses is presented in Figure 32.

FIGURE 32.

Distribution of the number of items with missing responses for the satisfaction with the decision-making process domain.

Overall family satisfaction

Twenty-four items (items C1–C14 and D1–D10) constitute the overall family satisfaction score. Of the 7019 questionnaires received, 2898 (41.3%) had complete responses for all 24 items. Most frequently a response was missing for either D10 only (‘when making decisions, did you have adequate time to have your concerns addressed and questions answered?’) or a combination of D7 (‘did you feel included in the decision-making process?’), D8 (‘did you feel supported during the decision-making process?’), D9 (‘did you feel you had control over the care of your family member?’) and D10 (Figure 33).

FIGURE 33.

Common patterns of missing data for the overall family satisfaction score.

The distribution of the number of items with missing responses is presented in Figure 34.

FIGURE 34.

Distribution of the number of items with missing responses for the overall family satisfaction score.

Multiple imputation

As a considerable number of respondents had not completed the question regarding number of visits but instead had given free-text responses such as ‘every day’ or ‘stayed continuously’, we derived a frequency of visits variable categorising a family member as a frequent visitor if they visited at least once per day (including these free-text responders). Dependency was recoded into three categories combining the middle two groups (none, some and total dependency) and type of admission was recoded into non-surgical, elective surgical and emergency surgical. We categorised ethnicity combining all non-white into one category. Age of the family member was analysed as categories rather than continuous, as exact age was available only for respondents.

Preliminary analyses indicated a low percentage of missing values across the family member characteristics used in the main analyses. Where possible, family member characteristics were taken from questionnaire responses; however, before imputation, we recovered information for missing data in key variables from available data recorded on the web portal (age group, sex, relationship to patient, next of kin and lives with patient).

Overall, a complete case analysis model-fitting procedure would eliminate 59% of the sample, potentially biasing the inferences.

Apart from UK FS-ICU-24 items, data were imputed for family member age group (1.1% missing in the full data, respondents and non-respondents), family member sex (0.8%), next of kin (1.9%) and frequency of visits (2.4% missing among respondents, after taking account of free-text responses, but 42.8% missing among all family members, as it was not recorded on the web portal).

The evaluation of the imputed score values (scale-level imputation) for the FREE study showed that, although the observed and imputed values had the same mean and variance, they did not have the same distribution of values, since the imputed values had no skew. When each item was imputed as ordinal (item-level imputation), the mean and variance were consistent as well as the distribution, so this was the approach used for the final analysis.

Summary measures for the overall family satisfaction score and domain scores after multiple imputation of missing values, compared with the alternative approaches reported previously, are shown in Table 42. A comparison of the scores for ICU survivors and non-survivors in the multiply imputed data is shown in Table 43. Family satisfaction was substantially higher among family members of ICU non-survivors than among family members of ICU survivors, particularly for the satisfaction with decision-making domain score and its subdomains.

Selection of data

The final analysis of the FREE study was based on family members of patients (excluding readmissions) who were discharged from, or died in, the ICU up to 15 August 2014. Family members of patients remaining in the ICU beyond 15 August 2014 were excluded, as they would not have had an opportunity to complete and return a questionnaire before the follow-up database was closed on 12 September 2014 (note that, as family members were not recruited for patients admitted after 30 June 2014, there were very few patients remaining in the ICU beyond this date). Patients with no family members recruited to the FREE study were excluded from the analysis.

Primary analyses were undertaken using responders only (both complete responders and partial responders with missing items imputed). To test if our findings were influenced by using imputed data, we also conducted sensitivity analyses using complete case data (i.e. complete responders only), the modified traditional approach to scoring (using complete responders and partial responders with at least 60% of items completed) and all family members (additionally including non-responders, with their satisfaction imputed).

Statistical analysis

The following family member characteristics were described by mean and SD, median and quartiles, or number and percentage for all family members included in the final analysis, stratified by the patient’s ICU outcome (alive/dead): age in years; age group (< 30 years, 30–39 years, 40–49 years, 50–59 years, 60–69 years, 70–79 years, ≥ 80 years); sex; ethnicity (white, mixed, Asian or Asian British, black or black British, other ethnic group); relationship to patient (partner, parent, child, sibling, other relative, other non-relative); next of kin (yes/no); lives with patient (yes/no); highest level of education (NVQ level 1 or 2, equivalent to GCSE or O level; NVQ level 3, equivalent to A level, AS level or Higher School Certificate; NVQ level 4 or 5, equivalent to degree, higher degree, HNC or HND; other); deprivation (quintiles, using the Index of Multiple Deprivation 2010 for England, Welsh Index of Multiple Deprivation 2008 or Northern Ireland Multiple Deprivation Measure 2010, assigned according to the family member’s postcode); distance from home address to hospital (calculated in kilometres as the straight-line distance from the mid-point of the postcode for the family member to the midpoint of the postcode for the hospital); previous experience of ICU as a family member (yes/no); and frequency of visits.

The following patient characteristics were described by mean and SD, median and quartiles, or number and percentage for patients with a family member included in the final analysis, stratified by the patient’s ICU outcome (alive/dead): age in years; age group (< 30 years, 30–39 years, 40–49 years, 50–59 years, 60–69 years, 70–79 years, ≥ 80 years); sex; ethnicity (white, Asian or Asian British, black or black British, mixed ethnicity or other ethnic group); deprivation (quintiles, defined as for the family members); distance from home address to hospital (defined as for the family members); severe chronic conditions in the past medical history (liver, renal, respiratory, cardiovascular, metastatic cancer, haematological malignancy, immunocompromise – defined according to APACHE II26); prior dependency (able to live without assistance in daily activities, either minor or major assistance with daily activities, total assistance with daily activities); surgical status (planned admission direct from theatre following elective or scheduled surgery, unplanned admission direct from theatre following surgery of any urgency, non-surgical); ICNARC physiology score27 (mean and SD); APACHE II score26 (mean and SD); ICU length of stay (median and quartiles); receipt (number and percentage) and duration (median and quartiles in calendar days) of organ support in the ICU (advanced respiratory support, advanced cardiovascular support, renal support, neurological support); and, for ICU survivors, whether or not the patient died before ultimate discharge from acute hospital.

Variation in family satisfaction was analysed across the following factors: family member characteristics (as above); patient characteristics (as above); ICU/hospital characteristics (hospital teaching status and number of beds in the ICU); and other contextual factors (month of ICU admission and, for family members of ICU survivors, whether or not the questionnaire was received while the patient was still in hospital). These were explored using univariable and multivariable, multilevel linear regression models. There were three levels of the models: family member; patient/ICU admission; and ICU/hospital. The outcome for the primary analyses was the overall family satisfaction score. As secondary analyses, separate models were fitted for each of the original domain scores as well as those resulting from the full psychometric assessment (see Chapter 4). All models were stratified by the patient’s ICU outcome (i.e. separate models were fitted for family members of survivors and non-survivors). The multilevel models were fitted using the ‘xtmixed’ command in Stata/SE version 13.0 (StataCorp, College Station, TX, USA) using ‘mi estimate’ to analyse multiply imputed data.

A simple random intercepts model was fitted to decompose the amount of variance that existed between patient and ICU/hospital levels. The level effect was assessed by the relative contribution of these variances (note: likelihood-based approaches to testing are not applicable with multiply imputed data). The variance partition coefficient reports the proportion of the observed response variation that lies at each level of the model hierarchy and was used to confirm that both the ICU/hospital variance and the patient variance are separately significant.

In order to evaluate the distribution of variance between levels while analysing the contribution of each level’s characteristics, a two-stage model building approach was applied, as follows.

Stage 1: to select the significant characteristics at each level, the step-by-step approach was adopted, following an approach suggested by Raudenbush and Bryk51 and Snijders and Bosker. 52 Three models with fixed slopes were developed. In the first model, family-member-level variables were included in a simple random intercepts model. The second model was extended to include patient-level variables. The third model additionally contained the ICU/hospital-level variables and contextual factors. The opposite, backwards, approach of entering all possible variables at one time and then removing the non-significant ones was not feasible because of the large number of variables and multicollinearity problems. At each step, relationships between the outcome and each variable at that level were tested, one at a time, in multilevel models adjusted for the outcome and the lower-level variables. A significant Z-test for the variable coefficient confirmed that adjusting for this variable significantly improved the fit of the model. The differential explanatory power of the variable was evaluated by examining the total and level-specific changes in variance and adjusted intraclass correlation coefficient. A number of key variables were identified a priori to be carried forward to the full multivariable models (family member age, sex, relationship to the patient, next of kin and lives with the patient, and patient age, sex and ICU length of stay). Other variables were carried forward to the multivariable model if the results of the significance test and or the evaluation of explanatory power showed an important effect in the model. At each step, a Wald test was used to confirm the joint significance of all variables added at that level.

The variables that were not selected in the original multivariable model were added into the final model to assess the joint significance of these variables, adjusted for the selected variables, and changes were made to the model, if necessary.

Stage 2: in the final, three-level, model, with random intercepts at level 2 (patient) and level 3 (ICU/hospital), interaction terms specified a priori were considered (patient age with sex and ICU length of stay with receipt of advanced respiratory support). Determinants in the model that might better be modelled with random coefficients at level 2 were investigated, one at a time, to judge the impact of setting them as random effects. The inclusion of interactions as well as models with random slopes were judged by the size of the variance corresponding to the slope.

Descriptive statistics and univariable analyses

The comparisons of family member and patient characteristics for ICU survivors and non-survivors are presented in Tables 44 and 45, respectively. Univariable analyses of the association between each variable and family satisfaction are shown in Table 46 (family characteristics), Table 47 (patient characteristics) and Table 48 (ICU/hospital characteristics and contextual factors).