A service-user digital intervention to collect real-time safety information on acute, adult mental health wards: the WardSonar mixed-methods study

Alignment with the measurement and monitoring of safety framework

In 2014, Vincent et al. published the Measurement and Monitoring of Safety Framework (MMSF),100 a theoretical framework for the measurement and monitoring of safety in health care. This framework proposed a shift of focus from the measurement of past harm towards the prevention of future harm, through assessment of current safety. It involved co-design with service users and was evaluated in a range of healthcare services, including mental health services. The MMSF has five domains:

• Past harm: has patient care been safe in the past?

• Reliability: are our clinical systems and processes reliable?

• Sensitivity to operations: is care safe today?

• Anticipation and preparedness: will care be safe in the future?

• Integration and learning: are we responding and improving?

The current study is aligned with the MMSF100 domain ‘sensitivity to operations’. This domain highlights the crucial but often overlooked activity of monitoring the safety of care as it is delivered in real time, recognises patients and families as important sources of information, and highlights the need for staff to develop a collective awareness of the workings of the service and to be sensitive and responsive to subtle changes and disturbances. Proactive day-to-day monitoring of patient perspectives could potentially bring greater benefits than the traditional reactive approaches relying on retrospective review and could be part of a broader vision of how to improve ward safety.

Mechanisms for patient feedback

At present, there is no mechanism by which leading (i.e. moment to moment) safety data from patients on acute mental health wards can be captured and made available to staff in real time. There are different potential mechanisms for patient feedback about safety to be received and discussed by ward staff in an acute mental health ward. For example, in the PRASE intervention,92 feedback was collected over a 3- to 4-week period, and staff came together to consider the feedback report and produce action plans. 20,91,92

Routine shift handover meetings between outgoing and incoming staff can successfully be modified to incorporate standard items such as evidence. 101 Another option is ‘safety huddles’,102 which originated in secondary care as short daily meetings to brief staff on immediate safety issues. They were found to facilitate real-time identification of, and response to, safety concerns,103 and have been implemented successfully in a mental health setting. 102

Phase 1 design

Phase 1 consisted of overlapping stages progressing towards conceptual clarification, as illustrated in Figure 1, followed by technical specification and delivery of a testable intervention. Key theoretical concepts underpinning the development of the tool were the MMSF domain ‘sensitivity to operations’, contagion and milieu. These were explored in the literature at an early stage in the study113 to aid conceptual clarity and inform the research design.

FIGURE 1.

Phase 2 data categories synthesised for evaluation of feasibility and acceptability.

Specific Phase 1 components were:

• A co-design approach.

• A series of consultations with patient advisory groups, current inpatients and health professional staff to explore possible conceptualisations of the intervention and refine prototypes.

• A review of the literature on patient involvement in safety interventions in acute mental health care.

• An evidence scan of the use of digital technology in mental health contexts.

• Semistructured qualitative interviews with mental health patients and health professional staff, to elicit their views on ward safety.

• Ongoing engagement with stakeholders to guide and inform concept development and refine technical iterations.

• Development of a logic model and programme theory to act as a framework for organising and refining ideas about a safety monitoring tool.

• Technical development of a testable tool that uses patient feedback for proactive safety monitoring.

Phase 1 resulted in a testable monitoring tool that used a tablet computer device for real-time safety monitoring of patients’ perceptions of safety on the ward.

Phase 2 design

Phase 2 was a mixed-methods process evaluation of the feasibility and acceptability of the monitoring tool via four categories of data (qualitative interviews, focused ethnography, pre- and post-intervention measures, real-time measures and routinely collected ward data; Figure 1).

Specifically, these categories were:

• A focused ethnography to explore how staff communicate and use safety data; this included on-site observations by researchers.

• Qualitative interviews with patients and staff (health professional staff).

• Simultaneous collection of routine data (i.e. incidents, workforce and ward occupancy).

• Pre- and post-intervention measures relating to staff perceptions of safety culture and ward atmosphere, together with real-time measures of these concepts.

• Analysis of data from all the above.

Methods

Search strategy and study selection

Systematic searches of academic databases (CINAHL, PsycINFO, MEDLINE, EMBASE, Web of Science and Scopus) and grey literature were designed using medical subject headings and the research question: ‘To what extent are patients involved in interventions to improve patient safety in acute mental health care?’ Searches (2000–20) were conducted during March–June 2020 (last academic database search 1 April 2020; last grey literature search 19 June 2020).

Grey literature was included because of the likelihood of finding unpublished research around safety in acute mental health conducted by clinical teams. Additionally, 14 mental health-specific sources (e.g. Centre for Global Mental Health) and 25 non-mental health-specific sources (e.g. Royal College of Nursing) were explored, together with the National Institute for Health and Care Excellence (NICE) evidence database, ProQuest Thesis and Dissertations database, and three social media platforms (Twitter, Facebook, YouTube). Additional grey literature sources of interest identified by authors’ expertise and hand searching were screened. Review papers produced by the search were scanned for potentially relevant papers. Duplicates were identified and removed using EndNote X9 (Clarivate Analytics, Philadelphia, PA, USA) bibliographic software.

The same eligibility criteria applied to both academic and grey literature. Eligible studies involved patients in active research and/or patient or staff safety improvement role, as classified by the Involvement Matrix. 122 Eligible outcomes related to patient involvement in patient safety research or interventions. Eligible settings were inpatient mental health care contexts. All study designs were eligible for inclusion.

Search results were exported to Covidence (Melbourne, Australia) for screening at title, abstract and full text level by two researchers. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow chart124 is shown in Figure 2.

FIGURE 2.

Preferred reporting items for systematic reviews and meta-analyses flow chart.

Data extraction used the Covidence online tool and was informed by the Joanna Briggs Institute reviewers’ manual. 125 Study quality was appraised using the Mixed-Methods Appraisal Methodology Tool (MMAT),126 which is suitable for assessing diverse study designs. Patient involvement was evaluated using Smits et al.’s Involvement Matrix. 122

Results summary

A total of 52 studies from the grey and published literature were included. They were conducted in the UK, United States, Canada, Finland, New Zealand and Europe. The majority (n = 33) focused on reducing staff use of restrictive practices. About half of the studies reported only limited patient participation in the research, and the involvement of patients as decision-makers was reported in only four studies.

Safety interventions ranged from system change at the organisational level, through operational decision-making at ward level to interventions for individuals, such as a smartphone application (app). Patients were involved mainly as co-thinkers, advisers and partners rather than decision-makers. The more extensive their involvement, the more likely they were to have active roles in the research. Research with high patient involvement tended to focus on forensic mental health and to be associated with reduction in restrictive practices. Low patient involvement tended to be associated with less reduction in restrictive practices.

Narrative synthesis of the included literature identified a possible association between studies with high levels of patient involvement and more effective safety interventions, but methodological quality of the surveyed papers was inconsistent. The review findings were incorporated into the ongoing synthesis of all data and learning from Phase 1.

Methods

To further inform the development of a safety monitoring tool, an evidence scan127–129 of the literature around digital technology in a mental health context, was conducted in November 2020, using the databases CINAHL, PsycINFO and Web of Science. The search terms and included studies detail can be found in Appendix 1.

Results

Research in the field of application of digital technology in mental health contexts appears to be concentrated primarily on therapeutic interventions, such as assessment of suicidal ideation or suicidal risk, and psychological support therapies, such as counselling. A limited body of research on telecare and the design of mental health apps for use in various other contexts was identified but digital technologies specifically related to mental health care are relatively new. Many digital technologies for health (mainly apps) have been designed within the tech industry, not always using mental health expertise. 130

Regarding technology design, there was an emphasis on usability and accessibility and some concerns around confidentiality. Poor digital literacy is seen as a barrier to utility and some degree of personalising technology to the user was seen as potentially helpful. Preferred design features included images and the use of colour. Other themes included the possibility of under- or over-reporting by patients and the impact of the technology on staff. Establishing a threshold for staff action and building it into the technology to trigger an appropriate response were considered important. The reviewed literature also indicated that patients’ cognitive abilities, health status and confidence in the technology affect their engagement and that, provided that the digital technology does not cause patients any harm, it is seen as potentially helpful and useful to them.

Factors that support successful technology implementation and staff engagement included connectivity (e.g. access to a reliable internet connection), training for people who may be using the technology, the provision of technical support, and ensuring effective communication about the technology so that its rationale is understood.

Recruitment

Two NHS mental health trusts in the north of England (NHS trust 1 and NHS trust 2) supported the overall project and facilitated recruitment of patients. Patients were recruited via posters placed in NHS trust premises. Patients who were interested in taking part notified designated health professionals, who passed on contact details to the researchers. Health professionals were recruited via advertisements on social media (Twitter, now X, X Corp., San Fransisco, CA, USA) inviting them to contact the research team about the study. Potential interviewees were provided with further information and consent forms. Before commencing each interview, researchers explained the purpose and process of the interview to the participant, read the consent form out loud and recorded consent verbally. Interviews took place via the telephone/Microsoft (MS) Teams (Microsoft Corp., Redmond, WA, USA). Patients who were interviewed received a £10 shopping voucher as a thank you for their time.

Data collection

A topic guide was used to explore participants’ views about safety issues in acute adult mental health inpatient services and was adapted as interviews progressed so that pertinent topics were raised with subsequent participants. Appendix 2a contains the interview topic guide. Staff participants were asked to describe their job role and experience in acute adult inpatient mental health services. All interviewees (patients and staff) were prompted to discuss what they considered to be the features of safe and unsafe wards, their experience (or otherwise) of sensing that an incident had occurred or was about to occur, and their understanding of why safety incidents appeared to be temporally clustered. Interviews were audio recorded, anonymised and transcribed professionally.

Analysis

Members of the core research team coded the transcripts and jointly developed a coding framework based on inductive open coding of transcripts and deductive application of codes relating to the research questions and the topics covered in the interview. The constant comparison technique131 was used throughout, and analyses were conducted by hand and in Nvivo 12 (QSR International, Warrington, UK) software. Thematic analysis informed by Braun et al. 132 identified themes that cut across the descriptive categories.

Sample

A sample of 25 patients and mental health professionals participated in semistructured interviews via telephone or video link. Patient interviews were conducted between November 2020 and April 2021. Health professional interviews were conducted September–November 2020. Interview duration was driven by the interviewee and ranged between 15 and 71 minutes. The interview sample is summarised in Table 3.

A total of eight patients (female n = 2; male n = 6) over the age of 18 years with current or recent experience (during the past 2 years) of being an inpatient on an acute mental health ward were recruited for interview after responding to publicity material displayed in clinical areas. They made contact with the researcher and received further details about the study with a consent form. This sample was smaller than originally intended, with many fewer patients than staff. It is likely that the COVID-19 restrictions and the general social context influenced patients’ motivation to participate in interviews.

From health professionals, 17 (13 nurses, a psychologist, a psychiatrist, a speech and language therapist and a social worker) were recruited for interview. All had experience of working in UK inpatient psychiatric settings; 12 were based in one UK region and 11 were female; 8 were currently working in an acute mental health inpatient setting. Experience of working in mental health inpatient settings ranged from under 5 years to over 15 years.

Thematic data analysis133 explored aspects of contagion and milieu, highlighting the relationship between the milieu and contagion, and identifying themes of contagion via risk amplification, contagion via therapeutic depletion and contagion via increased exposure to safety incidents. Individual factors, the social and physical environments, and wider service factors (such as staffing levels) were considered as influences on safety.

The relationship between behavioural contagion and the milieu

Participant accounts of the physical, cultural and therapeutic space that constitutes a mental health ward described a complex, multilayered milieu in which attributes such as the social interactions of patients and staff and the wider service context were important influences. Their narratives suggested three mechanisms by which the milieu contributed to behavioural contagion: amplification of risk; a diminished therapeutic milieu; and increased exposure to safety incidents.

The extract below illustrates how different layers of the milieu and the interactions between them may be seen as facilitating, if not directly causing, perceived clusters of safety incidents.

When I enter a ward, I’m often told, you know, so-and-so has become distressed and then it’s set off someone else … I wonder whether, as professionals, we put that [interpretation] on the situation, because we’re trying to make sense of it ourselves, … You know, if someone becomes distressed, it then, kind of, ruminates with another person [and] they then feel upset. I think there might be a little bit of trauma stuff behind that, in terms of seeing someone be distressed is really upsetting … I also wonder whether there’s a relationship between that availability of staff. So, you know, if someone’s distressed because they haven’t been able to see the doctor, then it’s very likely that another person is going to get distressed, because they also can’t see the doctor. So I wonder whether there’s a more tangible reason why more than one person might be upset at that time.

Senior leadership team member

In this extract, the participant described being briefed about an incident (‘so-and-so has become distressed’) resulting in contagion (‘set off someone else’). She offered two explanations for contagion: (1) individuals’ characteristics (rumination, retraumatisation or increased sensitivity due to trauma, difficulty interpreting, regulating and expressing emotions); (2) the service context; specifically, that a diminished therapeutic milieu might frustrate patient attempts to access care (‘they haven’t been able to see the doctor’), leading to unmet needs and distress for that individual and potentially others.

Perspectives on safety

Participants referred to safety incidents that they said stemmed from patients and/or staff, and/or deficiencies within the service context. Aggression, violence, absconding, self-harm, suicidal behaviour, bullying, intimidation, substance use, property damage and use of restrictive practices were seen as safety issues.

Staff also referred to other events as having implications for safety, such as unexpected admissions or meetings, a chaotic patient, boredom, a fire alarm in the night, missed patient leave or staff being perceived (by patients) as acting unfairly. Many staff acknowledged that individual-level interpretations of service-level policy might vary between patients and staff or even different professions and that some incidents ‘never get recorded because it’s not recognised as an incident by a clinician’ (social worker, patient safety lead). This reflects differences in perspectives of patients, staff and organisational processes.

The quotation below illustrates how bullying by patients, staff behaviour and the wider service context combine to influence safety.

There was a lot of bullying. I felt really uneasy with some patients’ company, I spent a lot of my time in the activity room or in my own room, [or] on the occasions when I could get out, [in] the communal garden … Sometimes it was behind your back, if I was in the activity room but as soon as staff leave you could feel that uneasiness, you know what I mean, and there was one occasion when a member of staff went in and the panic alarm started but this female patient was always bullying people and the staff literally didn’t know how to handle her. It was so understaffed but the lack of staff, staff cannot be everywhere patrolling the wards, and they’re escorting patients and doing medication.

Patient 6, F

Patients drew on personal experience and gave examples indicating a broader range of the factors affecting physical and psychological safety. Other patients were a potential safety issue, as some were wary of the possibility of being attacked, without being protected by staff:

You get attacked in the hospital, by other patients and the staff don’t do nothing about it … I did get attacked in the hospital and they did nothing about it … I rung the police but the staff were saying he’s a liar.

Patient 3, M

Various norms and values around safety and risk were expressed, forming an impression of the cultural aspects of the milieu from reflections on policy, training, individual interpretations and staff–patient relations. Staff described shifts in attitudes to risk, safety and the use of restrictive practices (e.g. physical restraint, as-needed medication) had shifted over the past decade. They recalled feeling safe when ‘we could all restrain people and we had access to medications and that there were loads of staff’ (ward manager 1) and how the focus of post-incident reviews had shifted from explaining ‘why the incident occurred from our perspective’ (nurse 4) to considering patients’ psychological safety.

Staff also revealed how qualities they associated with a safe milieu could produce harm and clusters of behaviours they interpreted as ‘contagion’. For example, being ‘risk averse’ or performing well-intentioned efforts to prevent anticipated safety incidents could actually increase risk and safety incidents by imposing a culture perceived as oppressive: ‘the more restrictions we put on people, the more chance we’ve got of violence’ (ward manager 2). To add to the complexity of the context, it was possible that, if staff were out on the ward, visibly monitoring safety, patients could interpret this as a sign of disinterest in the patients:

I just think that they’re too busy, they’re moving around constantly, they’re not really putting the patients first.

Patient 5, M

Temporal clustering

One feature of social contagion is ‘temporal clustering’,134 where incidents of a behaviour are clustered together over a short period of time, like a ‘chain reaction’ (patient 2, M). Staff reported temporal clustering of incidents using terms such as ‘knock-on’ (ward manager 2) and ‘ripple’ (nurse 2). Arguably, this type of language represents a view that fails to consider contextual factors (such as staff shortages or subsequent COVID-19 restrictions).

Staff shared their perspectives on how patient aggression may be amplified and incidents of one type (violence, shouting) might have consequences (patient traumatisation, staff redeployment, patients need support, patients’ needs go unmet) that lead to incidents of another type (self-harm, leaving the ward without permission):

If a patient is very violent and aggressive on a ward, you may see a contagion for other violence and aggression … you may also see an incidence of increasing self-harm because people are distressed.

Social worker, patient safety lead

If you ever get a violent incident, you always end up with knock-on events … people requiring more one-to-one input [but] they can’t get it because the staff are redeployed somewhere else so that … it has a massive effect on the ward. And it’s traumatic, isn’t it, if you see something like that and people really struggle …

Ward manager 2

Information sharing between staff was seen as integral to constructing and maintaining a predictable, safe milieu, and indicative of a good team. Handovers between shifts provided an opportunity for outgoing staff to update incoming staff but could also ‘set off a chain of reaction with [staff]’ (nurse 6).

You get people going, oh you might as well go home, you’re going to have a terrible shift, it’s been a bloody horrible day, so-and-so’s been awful. You just think, for God’s sake, I don’t need this, this is not going to help me.

Ward manager 2

Although sharing patient information might help the team to feel prepared and safe, it could also lead staff to make negative assumptions, create anxiety and impact on how they interpret and respond to that patient.

Oh, Helen’s [name changed] been on this ward before and she’s really violent and then suddenly the next four people get this narrative of this person. And then […] maybe you’re scared or you’re unsure of what to do or your experiences inform how you interact with Helen.

Ward manager 1

Through their hypervigilance and information sharing, staff were primed to interpret cues based on their experience of previous instances that felt similar, information about patients, patients’ diagnoses and unconscious biases. Participants reflected that preparedness could be almost self-fulfilling: ‘when you’re expecting that situation, it can almost create that situation’ (social worker). They described how thinking about ‘scenarios that might arise, and how you’re going to manage them’ (nurse 4) generated feelings of tension and anxiety which affected relations with patients and also within the clinical team and could produce a ‘never-ending kind of spiral round of frustration, agitation and aggression’ (social worker, patient safety lead). Participants also described how they might arrive for a shift ‘already on the defensive’ (nurse 2) and full of ‘dread’ and this influenced how they processed information.

Contagion via therapeutic depletion

Therapeutic environments can become depleted when target-driven care models are prioritised over holistic philosophies of care. 136,137 Participants viewed the therapeutic potential of the milieu as another fundamental component of safety, suggesting that without it, ‘everything starts to fall to pieces’ (nurse 8, manager) and the ward becomes merely a ‘holding pen’ (senior leadership team member). Some pointed out that a depleted therapeutic milieu was harmful because ‘you’re leaving people just in their own distress or disturbance’ (ward manager 1) and creating an opportunity for aggression and unrest to develop.

They might sit for a couple of hours longer in the bedroom and then something that might have been quite easily resolved earlier on becomes another incident, whether it be self-harm or whether they go AWOL [absent without leave] or it could be anything.

Nurse 5

If you’re not allowing people to have access to leave, access to activities, access to things that make them feel well, you’re going to find that it’s going to be more unsafe because people are going to be more aggressive, people … there’s going to be a lot of unrest, people are going to be unsettled.

Ward manager 2

Furthermore, staff who looked disinterested or unhappy could influence the whole milieu:

[There should be something for staff] to remind themselves why they’re actually there, they’re there to help people, you know, to help those in need. Some having that negative, very down, low attitude, it’s going to translate into the team and translate into the patients.

Patient 5, M

It should be included in their training about showing empathy and compassion with patients no matter what their situation is because some of them can be really dismissive and just not interested … I think again it comes down to like, you know, more staff, attentive staff, staff that are not overworked to really sort of help dissipate these sorts of problems.

Patient 4, M

Patients were also sensitive to a ward milieu that made life difficult for staff.

A patient punched [a nurse] in a private area and all she did was go into the toilet and cry about it and then come back out and get on with her job … She should have reported. I mean if you’re a female in a hospital and you get touched inappropriately by a patient then you should report it. There were another female nurse that happened to, it were the same patient.

Patient 3, M

The topic of smoking was particularly highlighted, as restrictions on smoking could build up tension within the milieu without staff noticing.

This lad who kicked off, they wouldn’t let him have a smoke and that wound him up.

Patient 1, M

One of the examples of what staff don’t know about is smoking. In xxx you have to get an e-cig off one of the nurses and smoke it. I smoke myself and when I were in xxx I got fed up of asking for cigs so for a week I ended up on [nicotine] chewing gum, but that were the same, and every time I asked it were like ‘ten minutes’, ‘not now’, and it ended up they were like that with the chewing gum as well … I think that annoys people, not having a cigarette it affects people in different ways, some of those misunderstandings come from not having a cigarette.

Patient 2, M

Safety could be considered therapeutic in itself: ‘a vital thing for somebody to feel well’ (nurse 1). The therapeutic potential of the milieu could be diminished via exposure to an incident, or due to staff availability/unavailability, because of redeployment to an incident or retreat to their offices. Staff saw enormous potential for patients to feel ‘pissed off’ (nurse 8, manager), ‘rejected’ (nurse 7) and unsafe which may lead them to ‘self-harm or to try and abscond’ (nurse 7) or to try to elicit care, by using ‘verbal aggression’ (nurse 5) and ‘competing for volume’ (ward manager 2). These processes would damage trust: ‘it ruptures the patient–staff relationship hugely’ (psychologist).

Contagion via increased exposure to safety incidents

Potentially, exposure to safety incidents can create a norm whereby even in quiet periods, everyone within the milieu is anticipating the next incident, with resulting tensions contributing to the likelihood of it developing. When they do occur, the impact is widespread.

All the staff, kind of, run away into an office and say, are you alright, is everyone alright, and we’re all checking with each other. But we forget that we’ve got 20 guys either on the ward or in the bedrooms that are probably just as terrified or unsure of what’s happened, but we’ve left them to dwell in it. And then they don’t know what to do with it so then … we get that ripple effect of incidents.

Nurse 2

Finally, staff and patients shared concerns about the impact on patients of asking them to monitor their feelings of safety:

It is important to give feedback of any experiences, but … it is extra stress to be on the lookout for anything, you know, bad going on, having to report it and take the responsibility of reporting it, and everything.

Patient 4, M

One phenomenon in the data that can be interpreted as behavioural contagion was anxiety. Anxiety could be transmitted between staff and between patients and staff, leading to an overall increase in anxiety across the ward. The data presented here also included examples that did not qualify as contagion. For example, convergence, in which group norms lead to increased similarity, may be seen in the hypervigilance that stemmed from shared norms around the predictability of safety incidents.

Pre-prototype discussions: virtual stakeholder engagement sessions

In the summer of 2020, virtual stakeholder engagement sessions were facilitated by a member of the core research team with each of two stakeholder groups already established and hosted by an NHS trust (‘trust A’). One 30-minute session was held with a group that has a monthly meeting for people who currently access or have accessed trust services and their relatives, friends and carers. The second session (60 minutes) was held with a forum that provides researchers with an opportunity to involve public, service user and patient and carer involvement in research. These sessions focused on three key areas of discussion:

• How to phrase or ask patients about whether wards feel safe or unsafe.

• How best to talk about patient safety (e.g. the use of language, words or phrases that should or should not be used).

• Suggestions for what to consider when developing a digital intervention to be used on acute mental health wards.

One example of the impact of stakeholders is in preferences around language. For example, in the academic literature there are clear distinctions between the two concepts of milieu, with its therapeutic potential, and atmosphere, which is widely perceived as a measurable construct. 60,149,150 Atmosphere has been defined as the ward’s psychosocial climate,60 although it is argued that physical factors are important and should not be overlooked. 151 Milieu and atmosphere are closely connected concepts60,152 and are sometimes used interchangeably. 152

In the current study, stakeholder consultations concluded that the word ‘milieu’ was linguistically and conceptually challenging outside academic spheres; service-user networks preferred to refer to ‘atmosphere’ and ‘ward atmosphere’ as more familiar and self-explanatory terms. The use of inaccessible language can have an alienating effect. 153,154 In view of the stakeholder feedback, the term ‘atmosphere’ was applied in a broad sense within the user-facing components of the WardSonar project. It was agreed that it was more important to use language that was inclusive to diverse groups, than to complicate the syntax to avoid conflation of technical terms.

Ward visits to discuss prototypes 1 and 2

In late 2020, NHS trust A permitted a clinical team member of Thrive by Design to visit two wards (‘ward X’, male, and ‘ward Y’, female), to meet with staff and patients, on two occasions. This approach meant that contributions were made at a time and place that suited patients and ward staff. The permission was granted because this team member was also an occupational therapist who was familiar with the infection control policies and personal protective equipment in the trust. The first ward visits were in December 2020 and focused on prototype 1 of a digital product. The second ward visits were in January 2021 and focused on prototype 2 of the digital product.

Wards X and Y shared a reception area. Ward X was a working-age adult acute ward caring for males aged 18–65 years. In the reception area, six people from ward X contributed opportunistic feedback (three patients and the clinical team manager, ward staff of all grades, including administrative staff). Ward Y was a working-age adult acute ward caring for females aged 18–65 years. Opportunistic feedback from ward Y was elicited in the reception area from five patients, the clinical team manager and 12 other ward staff (clinical and administrative). The difference in participation between wards X and Y was due to overall engagement from each separate ward, particularly with respect to ward Y’s clinical team manager, who was very keen to be part of the work and ensured that staff were well informed about the visit and its purpose. In addition, three members of ward staff and five patients participated in co-design workshop activities (approximately 40 minutes’ duration) at the end of an existing meeting forum.

Feedback from these two ward visits was collated and prioritised jointly by co-design partners ‘Thrive by Design’ and developers ‘Ayup Digital’ using the MoSCoW method (Must have; Should have; Could have; Won’t have this time). 155

A bank of user stories was collected during the co-design activities to guide the conceptual and technical development of the tool. A sample is shown in Table 5. The full set is available in Appendix 2d.

Ward visit 1: prototype 1

The first prototype patient recording interface formed the basis of discussions in ward visit 1 in December 2020 (Figure 3). Draft ideas for the digital ‘products’ (i.e. patient recording interface, health professional dashboard) and the potential for a public-facing ward interface were presented and discussed, for example, with regard to wording, format, colour and how they would work in relation to existing systems. People wrote comments and fed back on all aspects of the design. Health professionals were consulted on whether the visuals looked like materials already in use, to avoid unintended consequences arising from confusion.

FIGURE 3.

Prototype 1.

Staff were keen on the use of colour and traffic light systems, as these elements would align with other systems. Following the first set of ward visits, the feedback and discussion content was represented in a spreadsheet as user stories, as described in Methods, to facilitate prioritisation to guide development.

Key areas where decisions were made following this first visit included that people suggested having different ways to be able to express how the ward was feeling, including pictures, text and being able to write in specifics. It was thus agreed to include free-text options. It was also suggested that more options for how the ward was feeling could be helpful to reflect that this is often more than ‘good, bad and okay’ and that the feeling is more of a spectrum. Consequently, it was agreed to have five options in the tool to describe how the ward was feeling, and that an additional question would be included to show the direction things are moving in (i.e. getting better, worse or staying the same; see Figure 3).

Ward visit 2: prototype 2

Ward visit 2 took place in January 2021 and focused on prototype 2 (Figure 4). Multiple versions of the latest iteration of digital ‘products’ were presented and feedback elicited from the patients, clinical staff and administrative staff, in addition to revisiting discussions from visit 1. People commented on components including the wording, icons, colours and representations (e.g. was it what they imagined?). The ward staff perspective on the requirements of the health professional dashboard were revisited, and the patient and ward staff perspective on the acceptability of a public-facing ward interface was considered (see Figure 4).

FIGURE 4.

Example content of prototype 2.

In this second ward visit, prototype 2 (see Figure 4) was viewed as a set of slides on a digital tablet. (When implemented on the wards, the tablet computers used in the study were often referred to as ‘devices’ to avoid confusion with pharmaceutical tablets.) As there was a high turnover of patients, many new patients were involved, although many of the same staff contributed, which enabled discussions from the first visits to be revisited.

People shared general perspectives on the prototypes and commented on wording and visual components, for example, the hospital image in prototype 2 was deemed to be confusing, as it looked like a church, and the weather descriptor ‘sunny’ was considered to be the wrong word and was subsequently changed to ‘calm’.

Practicalities of using a tablet on the ward was also discussed. The idea of using patients’ personal mobile phones had been rejected as unfeasible, but many staff raised concerns around the security of the tablets and who would have responsibility for them. This discussion led to key development decisions. Whereas it was not reasonable to rely on all patients having a smartphone and data, the NHS trusts did not permit use of iPads^® (Apple Inc., Cupertino, CA, USA) or the hospital Wi-Fi, so options for the monitoring device were limited to Android computer tablets and the patient Wi-Fi.

Around the time of this visit, the team also met to agree how and when the tool would be distributed to patients (three times a day facilitated by a member of staff), and this information was included within the introductory information in the tool itself.

Conceptualisation of safety theme

Facilitator notes from the stakeholder consultations demonstrated that although some service users said that the tablet would be a safe way to report safety issues, many remarked that reporting safety information on a digital tablet could create its own safety concerns, for example, it might be ‘bad’ for their mental health or could make other service users suspicious.

Physical, mental and emotional safety were all relevant: ‘It was noted that safety might at first be assumed to be physical [an example of tripping over the cleaner’s bucket was mentioned]’ (facilitator notes). Alternative terms representing safety were suggested: ‘reassurance, recovery caring, security, calm, responsive, comfortable, protect, concern, and safeguard’ (facilitator notes).

There was general agreement that it was important to report matters to staff in person, regardless of digital monitoring, and the suggestion that using the technology could reduce the amount of time staff spend actually checking in with patients: Please make sure it’s clear from the start that it is an optional extra NOT a replacement.

Facilitator notes

A group participant mentioned the difficulty in distinguishing ‘what is in my head and what is the atmosphere’ – this would make answering the question ‘other patients/staff/how I’m feeling/ward environment’ very difficult.

Facilitator notes

Anonymity theme

Stakeholders reinforced the need for anonymity, as a prerequisite for safety.

It’s not safe – if someone has an issue with another patient and it’s known about.

Stakeholder feedback

The tablet was seen to enable anonymous feedback: for instance:

If particular staff create an atmosphere – this method will pick this up.

Stakeholder feedback

This is great, feel more comfortable anonymously saying how I feel, I probably wouldn’t say otherwise.

Stakeholder feedback

Milieu (ward atmosphere), contagion and incidents theme

The term ‘milieu’ was discussed and rejected by the stakeholders:

All agreed to the removal of the word ‘milieu’ from the interface. The group agreed that ‘atmosphere’ is more understandable and that using plain English is important.

Facilitator notes

Stakeholders also noted that if there was a bad atmosphere or the ward was understaffed, there would less likely be support with the device and so in that situation, some people would not be heard (facilitator notes); however, the device could encourage reporting:

I would feel more comfortable being able to say how safe I feel on a ward without queuing at a desk or not reporting at all out of fear.

Stakeholder feedback

Stakeholders expressed that an atmosphere in which alarms are going off and staff are running to an incident feels unsafe, so prevention was preferable. The need to capture whether the direction was changing that is ‘getting worse’ or ‘getting better’ was felt to be really important from a health professional perspective.

On the subject of placing researchers on the ward as observers, one stakeholder group advised that: patients would not be uncomfortable but that staff may feel they are being monitored and that they may not want patients to say they are uncomfortable on the ward, the staff might behave differently and it wouldn’t be representative of the norm.

Facilitator feedback

Digital technology on the ward theme

Stakeholders discussed the advantages and disadvantages of introducing a digital monitoring tool to the ward. There was a feeling among stakeholders that the device could be an unhelpful introduction on the ward as a digital system could potentially feel intrusive. It was also noted that some people might need help using the device.

Comments included:

Some people with mental health conditions dislike using technology, have suspicions or are affected by the digital divide, e.g. ‘If someone was tapping on a device – I would think it was about me’.

(Stakeholder feedback). Medication can also affect the ability to take in information and remember it – relating to using a digital system.

People need to know that input is anonymous otherwise (ironically) reporting ward atmosphere might not be a safe thing to do.

It needs to be explained when people first arrive on the ward including that is for everyone’s benefit – patients shouldn’t just need to figure it out or be told by other service users.

Most people like a screen but staff would need to support people not used to digital devices. It was pointed out that if there was a bad atmosphere or the ward was understaffed – there would less likely be support with the device and so in that situation, some people would not be heard.

Facilitator notes

Involving patients in measuring safety theme

With some exceptions, stakeholders agreed they would be happy to complete the survey and that the weather buttons were useful for determining an atmosphere.

Facilitator feedback

Apart from the possibility of triggering individuals’ symptoms (see Unintended consequences theme), concerns that were raised related to potential adverse impact on relationships between patients.

Stakeholders pointed to some of the implications of collecting patient perspectives on safety, such as: ‘If someone was on the device and someone else had their lighter taken off them by staff – the person with the lighter might think it’s related’ (stakeholder feedback).

The stakeholders also generated some ideas for sensitively crafted phrases explaining what the device was for, which it was felt would reassure people, for example: ‘We want you to have a good stay, we want to create a calm and responsive environment and to help you to feel safe’.

Feeding data back theme

Feedback included that the digital feedback method would be good for people who didn’t want to make a fuss. That if particular staff create an atmosphere – this method will pick this up.

Facilitator notes

The group asked what would be done with the findings – I said that the research is to see what difference it made having real-time feedback of how safe the ward feels. The group asked if the research team would be expecting the ward staff to act on the dial if the mood was turning on a ward to improve the situation before it gets ‘stormy’.

Facilitator feedback

They also agreed that if safety information is gathered it must be responded to in a timely way.

Unintended consequences theme

There was some critique of the possible impact of the device on how patients felt, such as:

[I]f the dial pointed to ‘stormy’ it would affect how [patients] felt; they also reflected on the impact on patients if the dial reflected something different from how they felt.

Facilitator notes

One stakeholder said, ‘I don’t like it – it’s triggering’. Another explained, ‘I would feel less safe because of it – I remember using a mobile phone on a ward and it triggered someone’s symptoms and I got water thrown over me’ (stakeholder feedback).

National Health Service standards

Technical considerations were discussed early and embedded in decision-making to lay the foundations for steps that would be required, should the WardSonar suite of digital products be deployed for use within NHS settings in the future. This includes the standards that NHS commissioners and funders will require to assess the adoption of the tools, namely the digital technology assessment criteria (DTAC),158 which bring together the baseline standards that digital tools need to meet for use within the NHS.

The standards include the following key areas: clinical safety, technical assurance, data protection, interoperability, usability and accessibility. The DTAC overlaps with guidance and standards around medical devices,159 data protection regulation [UK General Data Protection Regulation (GDPR)],160 Data Protection Act 2018;161 evidence and benefits [National Institute for Health and Care Excellence Evidence Standards Framework (NICE-ESF)]162 and data security and governance (NHS Data Security and Protection Toolkit). 163

Product developers creating technology for use in health and care are encouraged to gather evidence to meet these standards from the earliest stages. The evidence required is extensive. The steps taken and information generated from this study, including co-design, technical development and evaluation data all contribute to the evidence and documentation required for future use and deployment of the WardSonar products.

Approvals

Approvals for use of the digital products in Phase 2 implementation in the current study were managed through ethical and site approval steps. They were preceded by discussions with key stakeholders from the participating NHS trusts during the co-design process, including research and development, service/ward managers, information technology (IT) and information governance. These discussions supported hardware purchasing decisions that would meet trust policies and avoid any network restrictions. They also informed key functional and non-functional requirements for the tools alongside the co-design findings, such as:

• The requirement for the tool to be anonymous and not collect any personal information. There was a discussion around mitigating risk by not including any ‘free-text’ entry boxes within the user interface. Service users expressed a preference for a free-text option, however, so this was included.

• The requirement for the tool to allow for data collection in all areas of the ward environment. Risk here was mitigated by ensuring the tool had an offline option to mitigate against any potential data loss within areas of the wards that had patchy or no Wi-Fi access. Wi-Fi maps were provided to inform this.

• The requirement for data to be secure. Risk was managed by ensuring data could only be accessed by key personnel with login credentials.

Medical device classification

A medical device classification assessment was completed in June 2021, based on the features of the products that were intended to be used in Phase 2. The assessment referred to Medicines and Healthcare products Regulatory Agency guidance for software159 and concluded that the products do not classify as medical devices. Preliminary assessments against the NICE-ESF suggest that WardSonar would be classified as a tier B product. Full assessment against this framework was not completed during the study as this is not a requirement for tools used within research but would be required in the future to support deployment.

The steps taken during this study go some way to supporting the WardSonar suite of products to be ‘DTAC ready’; however, this is an ongoing process of evidence generation and documentation that will need to continue past the current study, should the product be developed further for use within NHS settings.

The changing context of National Health Service technology standards

It is relevant to note that there is an ever-changing landscape for use of technology within the NHS and the frameworks and standards that are applicable. Over the time that the current study was running, significant changes occurred that affected this landscape, including the merger of NHS Digital with NHS England. This particular change affected who is responsible and accountable for these standards and for iterations to the standards, frameworks and assessments. For example, the DTAC was previously the Digital Assessment Questionnaire, which was a self-assessment tool for product developers, but the responsibility for assessment is now held by the commissioning organisation.

Access and implementation

The WardSonar monitoring tool was implemented during the first half of 2022 on six acute adult mental health inpatient wards: three in each of two NHS trusts in the north of England. The study was subject to various ward-specific delays and interruptions stemming from COVID-19 outbreaks and/or restrictions and some wards experienced more interruptions than others, but on each site a 4-week baseline period (weeks 1–4) was followed by 10 weeks of WardSonar implementation (weeks 1–10).

The study team made preparatory visits to the participating wards to explain to ward managers how WardSonar worked and the rationale behind it. Ward managers agreed that staff would attempt to take the devices around to the patients on their ward, optimally three times per day, so that they could enter real-time data regarding how they were feeling about ward safety at that moment. When they used the device, a patient would find brief, illustrated, multiple choice questions based on a weather analogy, for example, ‘How does the ward atmosphere feel to you today? Very calm; calm; neither calm nor stormy; stormy; very stormy’ (see Figure 7).

On commencement of the study, each ward manager took charge of two tablet computers (Galaxy Tab A7, Samsung, Suwon-si, South Korea). The WardSonar screens (see Figures 7 and 8) were accessed on the tablet via a web app.

Data collection

Pseudonyms are used here to protect the identity of individuals, wards and organisations.

Two health researchers (R1 and R2) with psychology backgrounds were each allocated to one of the NHS trusts to conduct ethnography-informed observations164 and qualitative interviews with staff and patients. R1 was allocated to wards Apple, Cherry and Elderflower in NHS trust 1. R2 was allocated to wards Bramble, Damson and Fir in NHS trust 2.

The implementation period was 10 weeks (weeks 1–10). It was preceded by a 4-week baseline period (weeks 1–4) that allowed for the wards to get used to the monitoring tool (Figure 9). R1 and R2 visited their respective wards during weeks 1–4 to conduct baseline interviews and observations to start to build an impression of the ward and the views of staff and patients. They subsequently visited their wards at weeks 4, 6 and 9 for further interviews and observations. At each visit, they tried to encourage ward managers and other ward staff to use the device as agreed. Additionally, they conducted post-implementation interviews with the ward managers.

FIGURE 9.

Schedule for ward-based implementation and evaluation of the WardSonar monitoring tool.

Each ethnographic observation involved being present on the ward for 4–6 hours on two occasions in that week and taking detailed fieldnotes. The aim was to develop insights into each ward’s milieu to understand the context for implementation.

The purpose of the qualitative interviews with staff and patients was to understand technology adoption and elicit perspectives on feasibility and acceptability. R1 and R2 were advised by staff on which patients they could approach to invite to participate in an interview. Staff could be approached directly to invite to participate. Interviews focused on patient or staff experience and views on using the device. The Phase 2 interview topic guide is available in Appendix 2a.

Additionally, the technology automatically recorded ‘dashboard completions’ (see Figure 8), that is occasions when a patient entered real-time data about their perspective on the ward milieu. Other categories of data were gathered and subjected to statistical analysis (see Chapter 9).

Interviews were professionally transcribed. Observations were either written or audio recorded by the observer. Audio recordings of observations were professionally transcribed to MS Word®165 documents. Each transcript and observation were allocated a unique number. Original and anonymised audio files and transcripts were stored in the (organisation detail removed for peer review) secure drive. Only anonymised files were permitted to be downloaded from the secure drive.

All transcripts were imported into a Nvivo 12+project to support data management, retain links to source data during analysis, and to facilitate within- and between-case exploration of the data.

Analysis

Each of the six wards was a study site (i.e. a case) for analysis. Qualitative data collection and analysis were concurrent. Researchers R1 and R2 met frequently with other research team members to start preliminary data interpretation while the study was ongoing. This meant that their data collection could be responsive to questions suggested by the data. The statistical data analysis and qualitative data analysis informed each other to generate a rounded interpretation of WardSonar’s feasibility and acceptability. Data were examined across and within cases.

Pen portrait analytical process

A pen portrait analytical process was used to help clarify and communicate key messages found across the observations, interviews and dashboard completions. Pen portraits are familiar tools for humanities scholars wishing to convey a sense of how different strands of data interact within the complete data set. 166,167 The use of pen portraits is consistent with key analytical strategies for qualitative data, including framework,168 constant comparisons131 and Template Analysis. 169 In health services research, however, they are still an emerging method170 for working with multidimensional data sets. 75,170–172

Sheard et al. (2019)170 argued that the study of change has often focused on individual patients over time, and they recommend considering a broader data set. They identified a need for broad guidelines that researchers can draw on for the synthesis and reporting of multidimensional data, suggesting that this could replace study-specific bespoke methods, while enhancing methodological rigour via a consistent and theoretically sound approach. 173 The pen portraits in Sheard et al. (2019)170 broadly consist of tabulated summaries of fieldnotes, taped discussions, follow-up interviews and comments on how the ward engaged with the study, but the structure is adapted to differences between ward data sets. Louch et al. 75 used a more narrative style with commentary and quotations.

The guidance offered in Sheard et al. (2019)170 is a four-stage application of pen portraits to analysis of longitudinal data from qualitative health research:

• Understand and define what to focus on.

• Design a basic structure relevant to the data set in question.

• Populate the content.

• Interpret the data. 170 (p. 5)

Each of the two pen portrait exemplars in Sheard et al. (2019)170 is a one- or two-page, tabulated, chronological account of what happened and what it means. There is no rigid formula, and the exemplars illustrate how the proposed guidelines can be used to suit the particular data set. This flexible approach was applied to the analysis of qualitative data in the current study.

Data set

Of the six acute adult mental health inpatient wards, two were psychiatric intensive care units (PICUs); two were all female, one was all male, two were mixed (partly as a response to COVID-19) and one changed from mixed to all male during the study. The number of beds per ward ranged from 12 to 21. The impact of COVID-19 meant that some of these wards were underoccupied at times, so the number of beds did not always reflect the number of patients. To protect the identity of the trusts, wards and individual participants, some detail has been aggregated and names have been removed or changed.

Table 7 summarises the data set used for qualitative evaluation. R1 and R2 conducted 39 observations and 67 interviews. Each observation was for a minimum of 3 hours; total observation was 145.45 hours. Interview duration depended on each interviewee’s (patient or staff) preference and availability. Staff interviews ranged from 4 to 12 minutes (mean 8.3 minutes). Patient interviews ranged from 5 to 15 minutes (mean 7.8 minutes). Mean interview duration was 8.3 minutes (staff) and 7.8 minutes (patients). In total, 602 dashboard completions were recorded.

Pen portraits

Six pen portraits, one per ward, were constructed from these data. Several possible pen portrait structures were considered. The research team experimented with focus, length and level of detail,75,170 before finally agreeing on the format presented below. A high proportion of quotations was included to highlight the richness of the data. 174 The approach was underpinned by well-established principles for analysis of qualitative data, in which, for example, the researcher differentiates between semantic content and latent meaning. 175

The pen portrait for Elderflower ward is presented in Timeline below, as an exemplar. All six pen portraits are available in Appendix 4.

Focused ethnography

Access to the device

Access to the devices was subject to technical and practical problems including uncharged devices, missing devices, software updates required, poor Wi-Fi connections to the patient Wi-Fi and absence of visible dashboard data. The latter issue was a problem for researchers attempting to demonstrate to staff how they might benefit from using the tool.

I can see the devices are on the table and I say, oh, [staff name], I’ll just show you how to use them, but they’re both out of battery.

R1 fieldnotes, Elderflower ward

Concerns about protecting the study hardware meant that the devices were liable to be locked away, meaning less opportunity for data collection.

Additionally, COVID-19 outbreaks, periods of busyness and malfunctioning equipment seemed to be deterrents to taking the device round the patients. Patients frequently acknowledged how busy and overworked the staff were and understood this to be one of the reasons why they might not get to use the device. Ward managers seemed to agree that unless the device use was embedded into routines, staff would be unlikely to remember to take it round the patients. Figure 10 summarises challenges affecting opportunities for patients on the wards to enter real-time data on the WardSonar device.

FIGURE 10.

Challenges to collecting real-time safety data from patients.

Neither patients nor staff saw with data collection three times daily as feasible, and some did not see it as necessary. One member of staff said that low patient numbers meant each patient was approached too often with the device and lost interest in it; generally, however, staff cited the busyness of the ward as a barrier.

Pre- and post-implementation analyses: staff questionnaires

Research question:

• Does collecting data from patients and feeding it back to staff in real-time improve the safety of the ward?

Data sources

This part of the quantitative evaluation used a pre- and post-implementation design, focusing on staff perceptions of safety culture and ward atmosphere. Measurements were taken at baseline (week 1) and at the end of the implementation period. Staff across the six wards were invited to complete a paper questionnaire on these two occasions. The staff questionnaire is available in Appendix 2b.

Staff perceptions of ward atmosphere were measured using the EssenCES scale. 176 This measure has three subscales: patients’ cohesion, experienced safety and therapeutic hold. Respondents indicate the extent to which they agree with 15 statements on a 5-point Likert scale from 1 = not at all to 5 = very much.

Staff perceptions of safety culture were measured using the Agency for Healthcare Research and Quality (AHRQ) Hospital Survey on Patient Safety Culture (HSOPSC). 177 Despite recognised issues surrounding the assessment of safety culture,178 the AHRQ HSOPSC is a suggested tool for evaluating the cultural impact of patient safety initiatives and interventions. 177 This survey includes 10 composite measures and two additional measures – number of events reported and patient safety rating. the composite measures include teamwork, staffing and work pace, organisational learning – continuous improvement, response to error, supervisor, manager or clinical leader support for patient safety, communication about error, communication openness, reporting patient safety events, hospital management support for patient safety, handoffs and information exchange. Respondents indicate the extent to which they agree with statements on a 5-point Likert scale from 1 = strongly disagree to 5 = strongly agree. The single item ‘number of events reported’ asks respondents to indicate the number of patient safety events they have reported in the last 12 months, and the single item ‘patient safety rating’ asks respondents to provide an overall patient safety rating from ‘excellent’ to ‘poor’.

Data preparation and statistical analyses

Results

A total of 124 questionnaires were completed by staff from across the six wards. Invalidated questionnaires were removed, leaving 119 questionnaire responses for analysis. Table 8 shows the number of valid questionnaire responses, split by ward and pre/post WardSonar implementation. There were 84 questionnaires completed pre implementation and 35 questionnaires completed post-implementation across the six wards. The number of responses varied by ward with the most responses on Cherry ward (n = 29) and the fewest responses on Apple ward (n = 12). Given the small sample size, analyses were not conducted at the individual ward level.

Table 9 shows the mean average scores for each of the three EssenCES subscales, pre and post WardSonar implementation. The EssenCES patient cohesion and experienced safety scores were approximately normally distributed; however, the therapeutic hold score had a skewed distribution. There was a slight increase in the average scores of each subscale post-implementation, compared with pre implementation, but this difference was not statistically significant for any of the subscales.

Table 10 shows the average percentage of positive responses pre and post WardSonar implementation for each of the 10 AHRQ HSOPC composite measures. The measures with the highest per cent positive responses pre-implementation were: reporting patient safety events (74%), communication openness (74%), teamwork (73%), communication about error (73%) and supervisor, manager or clinical leader support for patient safety (73%). The measure with the fewest per cent positive responses was staffing and work pace, for which only 32% of responses were positive. There was an increase in the average percentage positive responses for 5 of the 10 composite measures, but none of these differences were statistically significant at the 0.005 significance level.

Table 11 shows the responses for the items number of events reported and patient safety rating pre and post WardSonar implementation. For number of events reported, pre implementation, the most common response was either no events reported (29%) or 1 or 2 events reported (25%). post-implementation, the most common response was 11 or more events (28%), which was an increase from 18% pre implementation; however, these differences were not statistically significant. The most common response to the overall patient safety rating question both pre- and post-implementation was ‘very good’ (49% and 40% of respondents, respectively). The proportion of respondents rating patient safety ‘excellent’ increased from 15% pre implementation to 26% post-implementation; however, these differences were not statistically significant.

Discussion of results

With respect to the research question ‘Does collecting data from patients and feeding it back to staff in real time improve the safety of the ward?’ there were no statistically significant differences in staff perceptions of ward atmosphere or safety culture pre and post WardSonar implementation. Therefore, WardSonar implementation did not improve staff perceptions of ward safety. A limitation of the work is that due to the sample size, analyses were not conducted at the individual ward level.

Data sources

Data preparation and statistical analyses

Statistical modelling

A number of statistical models were used to explore the relationships between wards, WardSonar use and occurrence of incidents over time. Owing to differing types of outcome measures, different types of models were used, ranging from zero-inflated negative binomial models for count data with a high proportion of zero counts to ordinal logistic regression for ordered response data. However, the underlying structure of the explanatory variables, the design matrix, was consistent across all models and consisted of fixed effects to capture time-invariant ward-specific effects, time of day variables and lagged values of incidents of WardSonar responses, depending on whether the model was explaining current incidents or current response. A simple linear trend was also included to determine any systematic deviation in use or response over time. Patient to staff ratios were not included in the analysis as the data were not considered universally valid.

WardSonar submissions, incidents and staffing data

There were 627 WardSonar submissions (602 completed submissions and 25 abandoned submissions) across the 6 wards over the 10-week implementation period. Table 12 shows average hourly volumes and case mix for staffing and patients, the total observed incidents, the number of WardSonar submissions and average hourly WardSonar scores by ward, split by day and night periods.

There were some instances of multiple submissions at exactly the same date and time. Insights from the qualitative data suggest data entry patterns may reflect ward routines (e.g. group meetings, shift changes), so there may be a group of service users entering data all during the same short period of time (potentially multiple entries inputted within one minute). There also appears to be some repeat submissions (with the same comments), but as they are indistinguishable from duplicates or different service users entering the same information, these records were retained within the analysis data set.

The number of completed WardSonar submissions varied by ward. The most submissions were on Cherry ward (n = 216) and the fewest were on Bramble ward (n = 45). On average, across the implementation period, the perceived ward atmosphere was calm on all six wards during daytime hours (8:00–22:00). Average WardSonar scores for the night-time hours were not calculated as there were either no, or very few, completed submissions during this period. On average, there was no perceived change in ward atmosphere on any of the six wards.

In total, 624 incidents were reported during the 10-week implementation period. Of these, 126 cases (20%) were missing information about the time the incident occurred, leaving 498 cases containing the time of the incident. Apple ward had the highest number of (complete case) incidents (n = 156) and Cherry ward had the fewest (n = 33). Of the incidents with complete case information, 149 (30%) were classed as causing harm. The proportion varied by ward, with 42% (n = 34) of incidents on Fir ward classed as causing harm, compared with 14% (n = 9) of incidents on Elderflower ward.

Average patient to staff ratios were slightly higher during night-time hours than during daytime hours on Apple, Cherry and Elderflower wards. On all six wards, the average number of registered staff was higher during daytime hours compared with night-time hours; but on Bramble, Damson and Fir wards, the average number of unregistered staff was higher during night-time hours than during daytime hours. On the smaller Cherry ward, with an average of 12 patients, the patient to staff ratio was lower (on average 1.5 patients to each nurse during the day) compared with the larger wards (Apple and Elderflower wards, with an average of 23 and 21 patients, respectively) which had an average day time ratio of 3.5 and 3.4 patients to each nurse, respectively.

Figure 11 shows WardSonar submissions (across the 10-week implementation period) by time of day. There were very few, if any, submissions during night-time hours on any of the six wards. There is some variation by ward in the pattern of submissions by time of day. Apple, Cherry and Fir wards appear to have peaks in submissions at particular time points during the day (i.e. in the morning, in the middle of the day and in the early evening). This trimodal pattern could reflect the stated intention to take the WardSonar devices around the wards three times a day. In contrast, Bramble ward has a single peak in the morning. Submissions on Elderflower ward are concentrated in the afternoon and on Damson ward around the middle of the day. Apple, Bramble, Cherry and Fir wards all have a peak in submissions in the morning around 8–10 a.m. (which from some of the earlier qualitative insights may correspond with a morning meeting).

FIGURE 11.

WardSonar submissions on each ward by hour of the day.

The pattern of submissions by day of week (Figure 12) also varied by ward. There were no submissions on Elderflower ward (and very few on Fir ward) Friday to Sunday. Apple ward recorded very few submissions during the beginning of the week (Monday–Wednesday) and Bramble ward had no submissions on Mondays and very few on Saturdays. Submissions on Cherry and Damson ward were more spread out across the week, with peaks on Cherry ward on Tuesdays and Thursdays and on Damson ward on Mondays and Fridays.

FIGURE 12.

WardSonar submissions on each ward by day of week.

As with submissions, there were also fewer incidents during night-time hours compared to during daytime hours on all six wards (Figure 13). There was, however, some variation by ward in the pattern of incidents by time of day. While there were fewer incidents during the night-time hours on Damson and Elderflower wards, the number of incidents throughout the rest of the day was fairly flat. In contrast, there was a peak in incidents on Apple ward in the evening and on Bramble ward in the late afternoon. Cherry and Fir wards had smaller peaks in incidents in the late morning.

FIGURE 13.

Incidents on each ward by hour of the day.

Figure 14 shows ward atmosphere responses by ward and week over the WardSonar implementation period. There was a general decline in responses over time on all wards. The most common response on all wards over the entire 10-week time period was that the atmosphere was ‘calm’ and the least common response was ‘very stormy’. However, Figure 14 shows that there was variation in the ward atmosphere rating within each week, with responses recorded across the five categories. This could reflect change in the perception of the ward atmosphere throughout the week or the differing perception of the ward atmosphere by different patients.

FIGURE 14.

Ward atmosphere over time by ward.

Figures 15 and 16 show the reasons given for the WardSonar submissions on each ward, split by ward atmosphere (see Figure 15) and direction of ward atmosphere (see Figure 16). When the ward atmosphere was rated as calm (‘calm’ or ‘very calm’ categories combined), the most common reason given for this on all but one ward was ‘the staff’. On Apple ward, the most common reason given was ‘the ward’. Similarly, the most common reason given for the ward direction ‘getting calm’ on all wards, apart from Apple ward, was ‘the staff’, although this was more pronounced on some wards (Cherry, Elderflower and Fir) than others. On Apple ward, the most common reason given was ‘the ward’.

FIGURE 15.

Reason for ward atmosphere by ward.

FIGURE 16.

Reason for direction in ward atmosphere by ward.

In contrast, when the ward atmosphere was rated as stormy (‘stormy’ or ‘very stormy’ categories combined), the most common reason on all but one ward was ‘the other patients’. On Fir ward, the most common reason given for a stormy atmosphere was ‘how I’m feeling’. The most common reason given for the ward direction ‘getting stormy’ on all wards was ‘the other patients’, although on Cherry ward this was closely followed by ‘how I’m feeling’.

Figure 17 shows WardSonar submissions and incidents by ward over the 10-week WardSonar implementation period. Cherry ward had the most WardSonar submissions but the number of submissions tailed off over the implementation period. Submissions on Apple ward increased over the first couple of weeks of the implementation period, dropped off and then increased again in week 7, after which no more submissions were recorded. Submissions did not appear to follow any clear pattern on the other wards and there were gaps in data collected during which no WardSonar submissions were recorded. In particular, there were large periods with no submissions on Fir and on Elderflower ward. Where there are peaks in incidents, there appear to be few (if any) WardSonar submissions.

FIGURE 17.

WardSonar device submissions and incidents by ward over 10-week WardSonar implementation period.

Exploratory regression analysis

Exploratory regression analysis was used to identify any systematic patterns between wards, time, incidents, submissions and types of submission. Of primary interest is whether information obtained from tool submissions may be used to predict the likelihood of incidents above and beyond alternative information, such as knowledge of recent previous incidents and whether or not there is any evidence of a contagion effect that persists after an incident has occurred.

All models were estimated using SAS^® 9.4 (SAS Institute Inc., Cary, NC, USA) analytical software.

Graphical inspection of the volume of submissions revealed differing patterns across wards, time of day and duration of study; alongside a mix of other notable factors such as a lack of submissions during weekend for Fir ward and long periods of non-response within the implementation period (e.g. Apple ward after 44 days, Bramble between 25 and 38 days, Elderflower between 34 and 54 days and Fir between 33 and 63 days). Submissions also tended to be made in clusters when any submission was made. Given the large number of periods where no submissions were made, combined with a clustering of submissions, conventional ‘count’ models (Poisson and negative binomial) were considered. Owing to the overdispersion of zero counts, zero-inflated Poisson and negative binomial models were also estimated, in which there were two models: a logistic regression looking at the probability of any submission and a count model conditional on there being a submission.

All estimations were conducted across all 30 imputed data sets with within and between variation synthesised using Rubin’s rule. The analysis data consisted of all ward by hour observations for which the WardSonar tool was in systematic use (i.e. excluding the hours after 22:00 and before 08:00) and also excluding those systematic periods for which the tool seemed to be unused or unavailable (e.g. Apple ward after 44 days). Thus, the results should be interpreted as conditional on the WardSonar tool being in general use. A sensitivity analysis was run in which only night-time data were excluded. The reduced data set contained 4695 ward by hour observations and the fuller data set contained 6300 observations.

Explanatory variables included: fixed effect/dummy variable for each ward to capture time-invariant underlying tendencies; a dummy variable for time of day (morning 08:00–10:59 hours; midday 12:00–14:59 hours; afternoon 15:00–19:59 hours and evening 20:00–23:00 hours); a weekend dummy variable to capture any systematic impact of weekends; a Fir ward by weekend interaction dummy variable to accommodate the noticeable, but not total, drop in submissions on Fir ward during weekends; and a series of lagged variables indicating whether an incident had occurred in each of the previous 5 hours. The rationale for the lagged incidents variables was to determine whether there was any evidence of social or behavioural contagion. Patient to staff ratios were not measured systematically across wards and so were excluded from the analysis. Finally, a days since trial started (days elapsed) category was included as a measure of time on the study. A positive (or negative) coefficient on this variable indicated whether the tool was being used more (or less often) over time.

For the fuller data set containing periods of systematic zero counts additional ad hoc dummy variables were included, indicating whether the data refer to an Elderflower ward weekend (Friday–Sunday) and/or a period in which no data submissions were made. These are only included in the zero-prediction stage of the zero-inflated models.

Table 13 shows the best fit zero-inflated negative binomial model (Zinf). Because of the large proportion of zero counts, the negative binomial distributional assumption is preferred to the Poisson and the two-stage zero-inflated strategy preferred to the standard traditional count models. The table shows the coefficient values and standard errors for both stages of the model with logistic model for predicting zero counts having coefficient prefixed by Zinf for the zero-inflation part of the model. Positive coefficients in the negative binomial part of the model represent increased volume of responses given a response, but as the zero-inflation part of the model models probability of a zero count then positive Zinf coefficients imply a lower count. Thus, the sign of the estimates should be interpreted differently, depending on which part of the model is being considered.

For the zero-inflation part of the model there was substantial heterogeneity between wards with, all other things being equal, Apple, Bramble and Elderflower were all more likely to post zero submissions at any time of the study and Cherry ward least likely to post zero submissions. There were also systematic differences across the time of day, with all periods more likely to have zero submissions compared with midday, and a response much less likely to occur during evenings. As indicated by the graphs, the positive and significant coefficient on Zinf days elapsed indicates a higher likelihood of zero submissions over time.

The coefficients for the lagged incident variables in the zero-inflation part of the model were mostly negative but not statistically significant. The 1-hour lagged variable was of the largest magnitude and had the predicted sign and was the closest to being statistically significant.

In terms of the conditional count second part of the model, significant differences across wards were again observed. However, in this case, wards Apple and Elderflower were more likely to have higher numbers of submissions, if any submissions should occur. Bramble ward, on the other hand, was not only most likely to submit zero responses, but also submitted the lowest volume, even when a submission was made. There were no time effects bar a slightly increased rate of submissions for morning, when at least one submission was made.

The expected coefficient sign for lagged incidents is positive, but only two of the five lagged variables had a positive coefficient and none was statistically significant. The coefficient for the lag of 1 hour, as in the zero-inflated regression, was of the hypothesised direction, largest magnitude and the closest to being statistically significant.

The days elapsed variable was not significant for the second part of the model.

The addition of the periods in which there were no submissions with accommodating dummy variables did not change the results. The estimated coefficients on the systematic non-use and Elderflower weekend demonstrated predictable signs of quasi-complete separation.

Conclusion to explanatory variables analysis

The observed variation in response rates across wards observed in the descriptive plots were found to persist in a multivariate statistical analysis. Similarly, there was a trend for submission rates to become lower over time, and for greater numbers to be submitted at midday. The two-part statistical model looking at WardSonar responses identified that much of the variation in overall number of reports was due to differences in an underlying probability to submit rather than in the volume or clustering of reports when any submission is made.

The estimated coefficients for lagged incidents was a notable finding. Although nothing was statistically significant in the model, the 1-hour lag was of the expected direction, had the largest magnitude and was closest to being statistically significant. This finding is broadly true across all estimation models estimated (including those not reported). Indeed, in the non-zero-inflated models, the coefficient attached to incidents lagged by one was always significant and positive, indicating that a response was more likely in the hour following an incident, all other things being equal. It is only when the mechanism of count generation is split into two in the zero-inflated models that significance is lost. Thus, in the exploratory analysis it is concluded that there is weak evidence to suggest that incidents may increase the probability of a response in the following hour, but no evidence to suggest that the effect has a longer duration.

Ward atmosphere

This section addresses the type of response for ward atmosphere and atmosphere direction, given that a response has been made. It represents a secondary part of the analysis considering the impact of ward characteristics on response. This regression used all 602 complete submissions and adopted a set of explanatory variables similar to that used to estimate volume of responses. Table 14 shows the estimated coefficients.

In both cases, the response category was treated as ordered with five categories for ward atmosphere (very calm, calm, neither calm nor stormy, stormy, very stormy) and three for atmosphere direction (getting calmer, no change/same, getting stormy). The ordered regression model also estimated thresholds between these categories within each regression (four thresholds for the five categories in ward atmosphere and two thresholds for the three categories).

Positive coefficients represent higher probabilities of calmer outcomes.

In the atmosphere model, very little was statistically significant. There appeared to be no variation across wards. Lagged incidents appear to have no impact. The negative and significant coefficient attached to evening indicates that stormier atmospheres were more likely to be reported later in the day. The statistically significant coefficient for days elapsed suggest that although the number of submissions declined over time (see previous analysis) they were more likely to report a calmer atmosphere.

For the atmosphere direction, significant differences were found across wards with Fir ward responses being more proportionately likely to report the atmosphere is getting calmer than other wards and Bramble ward being more likely to report the atmosphere deteriorating. No other variable was significant.

Overall, in terms of volume and types of display, there is evidence to suggest greater variation in volume of response rather than type of response across wards and time of day with weaker evidence that an incident occurring in the previous hour and only previous hour increases the likelihood and volume of response.

The final set of regression models looks at whether lagged WardSonar responses can help predict the likelihood of future incidents. Such a link would support the argument that measurements of ward atmosphere can be a proactive instrument in reducing ward incidents.

This relationship was explored by looking at the probability of an incident occurring as a function of fixed ward effects, time of day, the lag of previous incidents (up to 5 hours) and had four models of lagged WardSonar responses: mean ward atmosphere, mean ward atmosphere direction, volume of WardSonar submissions and the lowest atmosphere reading. Owing to the categorical nature of lagged responses and the low volume of some observed values, quasi-complete separation meant that lags of more than 1 hour could not be estimated in the data.

All non-night observations were used. This included periods when the WardSonar tool appeared not to be in use at all. To accommodate this an additional category was created for the WardSonar response ‘WardSonar not in systematic use’. The coefficient attached to this variable was intended to capture whether the periods for which the tool was not used happened to be periods of high incident occurrence. A significant result could suggest underlying tensions which cause incidents and may also affect the use of the tool. Tables 15 and 16 together summarise the lagged WardSonar responses.

As discussed, a positive (or negative) coefficient on the series of lagged variables indicated whether the tool was being used more (or less often) over time. In all four regressions, the results of the common variables were very consistent. There was significant variation in the rate of incidents between wards. There was no impact of time of day. Importantly there was compelling evidence that lagged incidents increase the probability of an incident in the current period – with statistically significant positive coefficients for 1, 3 and 4 hours. The lack of significance for 2 hours is puzzling but, overall, the data showed clear evidence of a lingering effect of previous incidents.

In all models, the ‘WardSonar not in use’ category was not statistically significant. This indicates that the periods for which the tool appeared never to have been used were not different from the other periods in terms of numbers of incidents.

In all, there was little evidence that lagged WardSonar responses were predictive of future incidents. No category for median atmosphere nor median atmosphere direction showed any statistical power in predicting incidents from the omitted no response category; and nor were the coefficients in the hypothesised directions. For example, the ‘calm’ category for ward atmosphere had a positive coefficient when a negative coefficient would have been expected. Furthermore, the size of the positive coefficient is greater than that for the ‘stormy’ category. A value that predicted fewer incidents when calm compared with stormy would have been expected.

There was a positive relationship between the 1-hour lag of volume of submissions and an increased rate of incidents, but it was not statistically significant. However, there was weak evidence (significant at 10%) that any reporting of a stormy atmosphere in a period leads to a higher likelihood of an incident in the following hour than if no response. The estimated coefficient for ‘very stormy’ as the lowest response showed clear signs of quasi-complete separation.

Discussion of results

There is substantial and significant variation in the use of the device across wards, both in terms of the likelihood of any submission at a given time and the number of submissions. There does appear to be a time element, with more responses in the morning. There is weak evidence to suggest that submissions are more likely in the hour following an incident. There is evidence to suggest that the use of the device declines over time; that is, that any submission is less likely over the duration of the study.

In terms of the type of responses for both direction and current atmosphere, although a response appears to be more likely given a recent incident, the type of response is not sensitive to whether or not there has been an incident. In terms of atmosphere, the only seemingly related factor was that evenings lead to greater likelihood of a worse atmosphere being reported, given a submission was made. Although the volume of submissions decreased over time, the probability of a submission reporting a better atmosphere increased slightly. There was surprisingly no variation across wards. In terms of direction of atmosphere, there was significant differences across wards, but this was the only significant variable.

The most important regressions look at the ability of the tool responses in terms of predicting future incidents. However, this did not generate useful insights, especially in comparison with the information that a recent lagged incident gives. This can be viewed in four ways while also allowing non-response as a predictor: (1) as a function of median reported atmosphere; (2) median reported direction; (3) volume of submissions; and (4) as a function of the worst reported atmosphere reading. Given the nature of the data, it was only possible to look back at a 1-hour lag each time. Lagged incidences (up to 5 hours) were included and the analysis allowed for time-invariant ward characteristics.

In summary, there were substantial significant differences in the probability of an incident happening across wards, which is consistently measured across all models. Similarly, there was strong evidence to suggest that incidences happening up to 3–4 hours prior adversely affected (i.e. increased) the probability of an incident occurring in the current hour. This was consistent across all models and offers a measure of the duration of contagion. There is nothing to suggest that the medians of responses are useful for predicting the likelihood of an incident in the current period. However, there is weak evidence to suggest that if the volume of submissions is high in the past hour, then an incident is more likely, and slightly stronger evidence (significance at 90% confidence interval) to suggest that if there is a single ‘stormy’ atmosphere reported then an incident is more likely.

Background to study

The WardSonar patient safety monitoring tool uses proactive real-time safety monitoring rather than traditional retrospective data review and is part of a broader vision for ward safety. Previous studies have found differences between patient and staff perceptions of the milieu in psychiatric settings. 31,150,179–181 The WardSonar study was partly a response to evidence gaps around the large numbers of reported and unreported safety issues on acute mental health wards; common practice is that data are collected retrospectively and almost none of the data relate to the patient perspective. Our data illustrate the relevance of this point, in suggesting patients may perceive that a ward is not safe on the basis of the interpersonal dynamics they are experiencing, but staff may take the absence of visible disruptions to the milieu as a mark of ward safety.

The current study preceded the recent publication by the UK Department of Health and Social Care of a review of data on mental health inpatient settings. 109 The review sets out ‘a vision for the future where the potential of data and evidence is fully exploited so that the healthcare system is able to ensure the highest standards of care in all mental health inpatient wards and pathways’. 109

Five themes were identified in the review:

• measuring what matters;

• patient, carer and staff voice;

• freeing up time to care;

• getting the most out of what we have;

• data on their own are not enough.

These are closely consistent with the aims and findings of the WardSonar study, thus demonstrating the currency and relevance of our work.

The study plan was disrupted by the COVID-19 pandemic. One possible consequence of this was low uptake of the tool, affecting data validity.

Study outputs

The study’s outputs were the development and evaluation of a co-designed monitoring tool to improve patient safety on adult acute mental health wards, through the collection of daily data about the patients’ perceptions of safety, to support staff in monitoring and improving the safety of the clinical environment and to test the intervention and explore its feasibility and acceptability.

The study delivered on its aims, which were to co-design a monitoring tool to improve patient safety on acute mental health wards, through the collection of daily data about the patients’ perceptions of safety, to support staff in monitoring and improving the safety of the clinical environment (completed in Phase 1); and to implement the tool and explore its feasibility and acceptability (completed in Phase 2).

The five study objectives are mapped to their corresponding report chapters below:

• Objective 1, to co-design with service users and staff a digital innovation that will allow real-time monitoring of safety on acute mental health wards. Chapters 4–7 address objective 1.

• Objective 2, to explore the feasibility and acceptability of capturing real-time feedback from service users about safety. Chapters 8 and 9 address objective 2.

• Objective 3, to explore how staff use this information when reported during daily handovers (or other mechanism). Chapter 8 addresses objective 3.

• Objective 4, to explore how the resulting data are related to quality and safety metrics. Chapter 9 addresses objective 4.

• Objective 5, to explore how these data can be used longitudinally to promote safety. A barrier to achieving objective 5 was the compressed time-scale for the study following pragmatic revisions to the protocol in the context of COVID-19. Chapter 8 partly addresses objective 5.

General patterns

Statistical analysis highlighted some patterns in the data, and construction of an hour-by-hour data set over each ward permitted detailed analysis of the use of the WardSonar tool over the duration of the trial and the relationship with recorded incidents. This approach revealed substantial variation in the use of the tool over time across wards, particularly in the volume of reports generated from the tool rather than the levels of ward atmosphere recorded. Peaks in incidents appeared to correspond with time periods where there were few or no WardSonar submissions. The reasons patients gave for the ward atmosphere varied by whether the atmosphere was perceived as calm (often attributed to staff) or stormy (often attributed to other patients).

Tool use was most common in the middle of the day and there was a small but systematic general trend to using the tool less over the duration of the trial. There were very few (if any) submissions during night-time hours; devices were used less during the weekend (and not at all or very rarely on some wards). Additionally, device use reduced over time and there were occasions during which there were no submissions for days or weeks at a time.

Predictive potential of WardSonar

There was strong evidence that an incident leads to increased probability of further incidents in the next 4 hours. This supports the idea of social/behavioural contagion and puts a measure on the extent to which the contagion persists. There is weak evidence to suggest that an incident leads to a greater use of the WardSonar tool in the following hour but does not appear to influence the type of response. There is no evidence to suggest that mean atmosphere nor changes in atmosphere predict the likelihood of future incidents; however, the presence of any individual ‘stormy’ response or an increased volume of submissions in a hour is associated with an increased likelihood of an incident in the next hour, but is not as strong a signal as is given by recent incidents.

Further work

Further work could explore relationships between high levels of staff sickness and ward atmosphere and between ward atmosphere and the reason for sickness, such as anxiety or stress. A separate analysis of seclusion incidents could explore possible contagion affecting how incidents in the seclusion room impact staff and patients on the main ward. The finding that use of the tool trailed off over time on most wards may suggest a need to review implementation processes in future work.

Use of pen portraits

As suggested in the literature,75 pen portraits were an effective tool for highlighting barriers to, and facilitators of, the implementation processes, and generated an holistic interpretation of how the monitoring tool was perceived and adopted on the wards. 75,166,167

These pen portraits supported the presentation of large quantities of data from a range of sources in a concise format. They facilitated rapid appraisal of rich data findings across and between cases, helping with data management, data familiarisation, synthesis, comparison, clarity around semantic ideas and latent themes, and summarising. The use of NVivo qualitative analysis software allowed an efficient approach to maintaining navigable trails between source and context, similar to the framework method. 168

Focused ethnography and researcher positionality

Focused ethnography is characterised by clear research positioning182 and reflexivity,183 a concept that is highly relevant, since researcher subjectivity shapes data interpretation,184 especially in qualitative research. 185,186 Theoretical frameworks for analysis of multimodal data can overlook researcher positionality, but it is critical to the process,187 as reflexivity helps to understand complexities of the context. 193 As insiders in terms of being familiar with WardSonar, and outsiders in terms of limited previous exposure to mental health wards, the researchers conducting the focused ethnography brought the valuable perspective of what Barnes describes as ‘the liquid inbetweener’182 (i.e. occupying a different space somewhere between the two extremes).

In this study, the focused ethnography facilitated a narrative in which the observers’ perspectives changed as they became used to the wards and the ward communities became used to them. ‘Othering’ theory suggests ‘three types of shocks — those based on a violation of a social taboo, those deriving from professional role reversal, and those that are based on stereotypes’ (p. 950). 189 The observers started out as ‘other’189 – strangers who did not know what is normal in this environment but, over time, they settled into the ward culture. A possible unfortunate effect of evolving observer perspectives over time may be that the observers became accustomed to and tolerant of ‘rudeness’ from some staff. This may have wider significance, as acceptance of unpleasant behaviour from colleagues can be one of the factors contributing to toxic nursing cultures. 190–194

The observations therefore offer a lay view, but an evolving one, on the ward milieu. The fieldnotes showed changes in the observers’ interpretation of the ward milieu. Wolgemuth and Donohue195 refer to ‘a beneficial shift from dualistic, categorical, and entrenched subjective positionality to a more ambiguous engagement with social reality’ (p. 1022), another phenomenon that can be seen in the observer fieldnotes. During their observations, the researchers occupied, both physically and conceptually, a space of work and treatment. They attempted to remain neutral and uninvolved in ward daily life. But from a patient perspective, the ward was home, however temporary, so some of the things that appeared noteworthy to the observers (one example might be idiosyncratic dress), may have been of no particular significance to patients or staff. Analysis of these data by other members of the core research team reinforced the importance of understanding researcher positionality.

Possible contextual influences on implementation and evaluation

Qualitative analysis provided potential explanations for the data patterns identified through statistical analysis. For example, patients’ positive overall response to WardSonar seemed to be linked with patients’ views that staff do not really know how they feel or what is going on among the patients; staff tended to have limited understanding of the WardSonar tool’s facility to aggregate and display patient reports; there was a positive response from staff as well as patients to the design features of WardSonar; it seemed important to have an engaged ward manager who would enthuse the rest of the staff to use the tool.

The qualitative analysis offered potential explanations for the fluctuations in submissions, such as that the tool was not much used during busy times when staff were dealing with incidents, and if the device was lost or out of action, the tool was unavailable for use. This appears to explain a common tendency for staff to collect patient submissions at quiet times, and when patients were gathered in communal areas such as during the mornings. Additionally, it raises questions about data quality (i.e. the extent to which the submissions genuinely reflect patients’ perceptions of ward atmosphere).

The insights described above add to the literature concerning how research studies can best be implemented in hospital settings196–199 and can inform a future iteration of WardSonar implementation and evaluation.

Implementation taxonomy

We adapted and applied Proctor et al. ’s taxonomy108 to help understand the implementation of WardSonar (Table 17 and Figure 18). Table 17 sets out the eight components of the taxonomy108 with comments on what was highlighted by the evaluation of the WardSonar monitoring tool and what this means for future implementation. According to Hernan et al.,107 one fidelity indicator could be the data collection processes (i.e. the entries on the devices for each ward); however, our evidence highlighted non-adherence to the protocol, for example a member of staff entering their own data, or staff potentially influencing the data entered by patients.

FIGURE 18.

Understanding WardSonar implementation: application of adapted taxonomy. 108 N.B. the outcome ‘Costs’ was not examined in the current study and is therefore not included in Figure 19.

Figure 18 maps the results from Phases 1 and 2 WardSonar study to the taxonomy; that is, whether stakeholders perceived WardSonar to be useful for improving ward safety, and suitable for the ward settings; personal responses to using the device to enter or access data; the extent to which WardSonar was accepted and integrated into ward routines and norms; the challenges and successes of data collection; and other practicalities such as whether the device was physically available (i.e. not locked away and with a charged battery and internet connection).

These implementation issues resonate with the conclusions of the evidence scan that formed part of Phase 1 of the current study, for example, the importance of connectivity, and effective communication about the technology so that its rationale is understood.

Engagement issues in digital health care

Data from the current study showed reductions in research engagement over time. The implementation context appeared to be profoundly shaped by the effects of COVID-19; however, staff burnout, which is a documented effect of the pandemic,200–205 has been highlighted previously as a barrier to change in health contexts. 192,197 Analysis of log data is an emerging field in health technology research that may improve understanding of these processes. 206

Digital health care seems more likely to succeed where it suits individual patient needs and preferences. 207 There are some indications that provision of personalised support may be a useful strategy to promote patients’ engagement with health technology;199,208–210 yet overall, the wider digital health literature suggests that engagement with digital health technology is not well evaluated and its mechanisms are poorly understood. 211–214 Further research into this field could support effective engagement strategies.

Tentative conditions for successful implementation of WardSonar

The evaluation of WardSonar highlighted conditions that appeared to enhance how the WardSonar study was perceived and implemented and perceived across the six sites. Broadly, these conditions may be summarised as:

• The design features are acceptable to staff and patients.

• Staff and patients alike understand the study’s purpose and the study protocol.

• Staff perceive that using the device will benefit patients and/or enhance ward safety.

• Patients perceive that using the device will benefit staff, patients, or the ward in general.

• The process of entering and accessing data via the WardSonar device is user friendly.

• The device works with minimal hitches.

• Ward leaders including managers and nurses in charge of shifts, are positive about and engaged with the study.

It seems that where the conditions are not met, adoption of the technology is adversely affected. Figure 19 illustrates similarities and differences between factors affecting patients’ and staff’s individual motivation to use the device.

FIGURE 19.

Factors affecting motivation to engage with the WardSonar device.

Technical assessment of challenges and further development

The prototype tool was designed as a standalone web app that did not receive or send information to any other systems during its implementation during the study. This was for several reasons:

• During discussions with the first NHS site involved in the co-design, it became apparent that the timescales for developing a tool that connected with existing systems would have created unmanageable delays to the study timelines.

• Different systems were used across different sites.

• At the early stage of design, it was not clear which systems the tool would need to talk to; that is, which systems would support use of the tool and where data collected by the tool may need to be transferred to.

The developers created an application programming interface that links the digital products together with the underlying data storage engine. This was developed using an open data format standard so that the data could be interoperable with other systems/products in the future, recognising that this is of importance to NHS staff and also a requirement of the DTAC. 161 More information is available in the technical report (see Appendix 2d).

Post-deployment challenges

This section highlights selected challenges reported to the core research team by the technical developers. A detailed account of post-deployment challenges is available in the technical report (see Appendix 2d).

• Connectivity issues: at times, certain wards were unable to connect their devices to the Wi-Fi:

  1. What we did: issues resolved by trust IT department.

  2. Future recommendations: more investigation/testing on wards before deployments and deployment of higher capacity servers. Map the coverage of the patient Wi-Fi across the wards. Ensure that staff know the patient Wi-Fi login details.

• Some dashboards were showing a ‘no data’ error:

  1. What we did: this was because the date range ‘remembered’ what date was selected previously. We updated the dashboard logic so the date defaulted to ‘today’ to avoid confusion.

  2. Future recommendations: investing in training users of the dashboard prior to use.

• Two of the devices were lost on ward A:

  1. What we did: change the password on the Google account to limit data access.

  2. Future recommendations: consider other security measures.

Considerations for further development of WardSonar technology

Conclusion: considerations for data interpretation

The idea that all data are socially constructed215 challenges the notion of objective reality in analysis. Gelman and Hennig184 propose eight markers (‘virtues’) of high-quality analysis that avoid using categories of subjectivity and objectivity: transparency, consensus (i.e. drawing on knowledge and providing rationales), impartiality (considering bias and alternative points of view), correspondence to observable reality, by connecting theory with empirical observation and creating conditions for reproducibility and testing, awareness of multiple perspectives; awareness of context dependence and investigation of stability (i.e. considering how conclusions might have been different if different decisions had been made). Therefore, in the current study, as with all categories of data,184 contexts for data collection and interpretation were subject to decisions about methods and meaning.

Figure 20 summarises possible influences on the data collected in Phase 2; for instance, what patients decided to report or not report, and why. This is a further consideration for data interpretation. It is possible that the context for entering data into the WardSonar monitoring tool was a mediating factor (e.g. whether or not staff and other patients were helping, watching, interested or reactive). Our work in Phase 1 of the current study highlighted that patients may be concerned that other patients think they are reporting on them or be reluctant to report that staff are creating tensions. This is consistent with the well-established argument that privacy preferences are an important driver of social interaction and behaviour. 216 Additionally, there seemed to be a possibility that staff could challenge patients’ perceptions (e.g. by assuming they were joking or expressing shock at what they had written). This possibility is supported by an observation in Salmon and Young,217 that ‘patients can regard being asked about care as an occasion to show gratitude rather than appear critical’ (p. 1516).

FIGURE 20.

Influences on the data; considerations for data interpretation.

Phase 1: IRAS project 242683

The health professional arm of the Phase 1 interview study received ethical approval from the University of Leeds, School of Healthcare Ethics Committee (reference: HREC 19-028) and the patient arm of the Phase 1 interview study received ethical approval from South Central – Berkshire B Research Ethics Committee (reference: 20/SC/0360).

Phase 2: IRAS project 300833

Phase 2 received ethical approval from East Midlands – Nottingham 2 Research Ethics Committee (reference: 21/EM/0247).

Search results

The included papers are listed in Table 18.

Introduction

The mental health literature indicates that staff, patients and carers all perceive safety slightly differently. Staff and carers are far more likely to focus upon physical safety, for example, avoidance of self-harm, whereas patients view emotional and psychological safety as equally, if not more, important.

What is reported below has emerged from interviews with mental health staff only. Each section follows the same format: headline findings, followed by an illustrative quotation(s), followed by what this means (potentially) for any technological development.

• Staff see patients as more sensitive to safety concerns than they are (this is both positive and negative).

• Threshold for safety is higher for staff than patients because staff know what is going to happen next to deal with an incident, patients don’t.

• Patients don’t have preloaded knowledge of other patients, they haven’t read the case notes, so an incident can be more scary for patients than staff.

• Staff see patients' feelings about safety as relating to their pathology, for example paranoia. Feeling unsafe can be seen as a proxy for suicidal intent.

• Staff see patients as having trauma responses and which leads them to having highly sensitive sensory reactions. This can be seen as positive or negative by staff, that is patients can be seen as informative or over reactive.

• Staff normalise distress, whereas patients, especially new patients, do not.

• ‘The Safety Thermometer [was flawed as there were] … thousands of reasons that patients didn’t feel safe … they’d … be absolutely distracted by everything else in the world that didn’t make them feel safe, whether those be symptoms or otherwise. So I don’t think something like that worked because it was far too vague’.

And

• ‘[Patients pick up on safety issues] … much earlier than we do. I’m always astounded by how … patients know exactly what’s going on … [they] pick up on the controlling nature of the micro interactions’.

• What this means.

• Any safety data fed through to staff might need to be anonymised (so it can’t be linked to a particular patient pathology).

• Safety data need to be specific, not simply asking about feelings of safety/lack of safety.

• Data need to be salient so that staff take them seriously: it needs to trigger the staff threshold for safety.

• Safety is a multifaceted concept that can mean different things to staff depending upon their professional role and career stage.

• Safety can be seen as primarily a nursing concern, whereas it is the responsibility of all staff, from consultants to domestics.

• Staff understandings of safety relate to experience of handling previous safety incidents and a building of confidence in doing so.

• Safety relates to having the skill set required to deal with incidents, for example, de-escalation and communication techniques.

• Safety relates to working with knowledgeable and engaged staff that can be relied upon, that is not too many bank/agency staff who may have no alignment to the ward.

• My experience of safety has definitely changed, my understanding, research, models of the interventions that we use more than when first trained which isn’t long ago compared to other people that I work with.

What this means

Any technology should allow for a multifaceted conception of safety according to role and career stage so that individual staff reading the data will recognise something as a safety concern.

Safety incidents are often preceded by micro-cues but in the context of complexity:

1. Word slip

2. ‘A look’

3. Demeanour

4. Way someone walks-with their ‘head down’

5. Voice – -both tone and content

6. A lot of activity or movement/noise on the ward

Space: how it is being used on the ward by both staff and patients. Are all patients in one place on the ward (or nowhere to be seen?)-sometimes referred to as ‘retreat and swarm’. Are staff visible on the ward? Where? (or nowhere to be seen).

Deviation from the normal, for example someone in their room who is normally in communal areas, or vice versa; someone eating in their room who normally eats in the dining room, or eating faster than normal in the dining room, or a norm breached by patients or staff.

Language changes, from proactive to reactive, staff become more directive:

1. ‘[if the] patient is acting different to usual … [gives example of patient who unusually sat in TV lounge] escalated through the day and she became more and more withdrawn and … isolative and then that wasn’t picked up by the staff and she was found ligatured in the lounge’.

And

1. ‘The wards are so complex … lots of things contribute to it … lots of small things … that then become a big thing … it’s pattern recognition as well’.

And

1. ‘when there’s some social norm that’s been breached, by the patient maybe but also by the staff’.

What this means

1. These micro-cues could be held in a database and if/when input into a digital technology by patients they can alert staff to a potential incident unfolding, particularly if multiple cues or ‘patterns’ are identified.

2. Staff may not respond to one cue but may need evidence of several before they intervene. (Thus, this can be conceptualised in synchronic terms, as events at one point in time, or as diachronic, as events that develop one after another that significantly alter the milieu, that is the ripple/domino effect.)

3. Cues could be differentially weighted.

4. Safety incidents and micro-cues are gendered.

5. Men more often elicit intimidating, violent and aggressive behaviours whereas women tend to withdraw or want increased staff attention.

6. Mixed wards can have a differ dynamic to single sex wards: either, sexual safety can be an issue on mixed wards, with both male and female patients reporting feeling safer on single sex wards, or, they can have more balance with men and women using informal controls to check each other's behaviour.

7. Men are more likely to get in the middle of an ongoing incident.

8. Women are more likely to self-harm.

9. A female staffed male ward can present particular safety challenges.

10. ‘men generally, I think, act more intimidating. And a lot of the ladies kind of retreat and go quiet, or will want to spend more and more time with you’.

And

1. [on an] acute male ward … when you have people … on the sex offenders register … they tend to be put on our ward … and we’re a largely female staffed ward … sometimes that actually affects our personal safety … we end up having to brief everyone from the domestic about not going in to rooms on their own.

What this means

1. All female, all male and mixed wards may all need different metrics to trigger a safety concern to reflect the gender composition of each.

2. Gender mix between patients and staff is another context variable to consider.

Milieu and contagion.

1. All staff seemed to agree that milieu and contagion exist even if they found these hard to articulate.

2. Many metaphors were used for contagion: ripples, pressure, domino effect etc. For milieu-environment, atmosphere, a feeling.

3. There was overall agreement that one safety incident could lead to others and that this effect could last from an hour up to 2 days on the ward. It was recognised that it can be transferred from shift to shift.

4. There are often differences between night and day shifts (the latter is more difficult).

5. One person can upset a whole ward, in part by diverting resources from other patients. Restraints particularly upset the ward.

6. Contagion spreads differently on male and female wards (men get aggressive/interfere, women withdraw or seek staff support).

7. Contagion can be more acute on wards with patients with learning disabilities:

8. [it lasts] probably an hour or two … sometimes it’ll go into the nightshift … it carries on again because then there’s more people available … that are refreshed … it’s not intentional, people don’t intentionally think right, we’ll just drain these new staff, that’s not the way it works, it’s just that’s human behaviour, isn’t it, that’s what happens.

And

1. In my head 2 days is the maximum to it. I think sometimes.…resentments rumble on and then there’s retaliatory things but usually after 2 days things are starting to calm down, that arousal curve, again.

And

1. ‘I definitely saw the difference between day and nights … [so I] speak to people straight away, start turning the lights down a little bit, ask people if they want a brew and a toast, you know, it’s late at night, you start putting those routines in place plus it’s quieter, there’s not as many people around, it’s a bit easier’.

And

1. The whole ripple goes through the whole ward … so, it doesn’t take long for everybody to know that XXXX was restrained and … that then causes upset for everybody

And

1. I think [contagion] … depends on the dynamic of the service users … some … love to get involved in an incident … [others will just]. … listen, and if you ask them to move away from the situation …

And

1. As soon as we have one lady going up there, everybody else is jumping in and either competing for volume or trying to soothe someone …

What this means

1. Any technology may have to incorporate an alert status (for up to 2 days) when a safety incident has occurred. The technology may have to become more sensitive during this period.

2. The technology would need to be able to pick up and collate input from patients perhaps in different areas of the ward to indicate that an incident was spreading, for example, increased anxiety or input activity by patients into the technology.

3. The technology needs to see night and day as different contexts and incorporate that into any alerts.

4. Milieu and contagion operate according to ward profile thus the technology would ideally incorporate this factor (e.g. all male, all female, mixed wards).

A digital technology on the ward: divided opinion.

There is some variation between trusts in the capacity for a digital technology to be incorporated into ward life some have superb Wi-Fi, others don’t have enough computers and are still using pens and paper.

Views were divided over the usability of iPads and phones by patients to input safety data. Some felt that patients could easily do this, whereas others highlighted that some patients cannot read and write.

Direct patients to bespoke website.

Not all patients speak English or have English as their first language.

Any technology would have to be very structured and focused to work.

A simple technology of smiley/sad faces or traffic light system was thought a good idea by some and usable by patients, though others thought patients would report in the negative all of the time with such a system.

A mixture of words, pictures and sound clips was suggested.

It is no good knowing that patients feel unsafe staff need to know why.

Most staff thought that patients could report on ward safety.

Some staff would struggle with interpreting the data.

Any technology may need to operate independently from nursing staff.

There’s a … role there for education because it’s that assumption … patients know how to work these things. Some … can’t even read or write, so how can we expect [them to] engage with technologies [they need] … skills to engage with it meaningfully.

And

1. … we have to embrace digital ways of working as we move forward because that’s the way the world is moving … that’s the way younger people understand the world and they’re going to be the newly qualified nurses of tomorrow … the Band 6s … [and] Band 7s of the future.

And

1. We take a photo of the QR code for their friends and family and then they fill out our surveys on the iPads and they’re quite receptive to that, I think that’s probably better than paper and pens, I don’t know if you could maybe develop a QR code that directs you to a website and just do it that way? … we’ve got amazing Wi-Fi.

What this means

1. Any technology is going to have to engage with a wide range of patients and abilities, making it accessible to those without reading/digital skills but not being too simplistic and potentially patronising for more able patients.

2. Some staff would struggle to interpret the data and may need training or more training than others. Potentially problematic for staff qualified a long time.

Measures of safety

• Both quantitative and qualitative measures would be needed.

• Some cues might be given more weight, numerically, than others.

• Data should be collected daily and sent to an app (for patients/staff) or a dashboard/computer (for staff).

• Numerical scores could be assigned to feelings of safety by patients.

• Safety can be measured in language both body and verbal when safety is diminishing language becomes more reactive and here and now. Staff may become ‘shouty’.

• Patient reporting activity is a valid construct in itself and indicates safety concerns, even if the reporting is incorrect in the detail.

• ‘having a threshold where … a nought to ten scale where once you hit four, that triggers a safety huddle, once you get to six or eight … that triggers an MDT discussion. I think the issue … with that though is … to have that huddle or … MDT discussion because if you’ve got to … the higher end of the scale … the likelihood is that you need staff on the ward, observing to maintain people safety’

What this means

1. Any technology needs to incorporate the input of quantitative and qualitative data, perhaps giving patients the choice of drilling further and further into how they are feeling. Some may want to give basic information.

2. Stress for patients should be considered.

3. Thresholds/scales need to be built into the system, triggering alerts to action.

What to do with the data

There appear to be four options to streaming the data to staff that staff interviewees thought would work:

• Live, continually and directly to staff phones via apps while on the ward.

• At daily meetings huddles and handovers this could be via phones or existing computer systems.

• Digital display boards.

• Direct patients to bespoke website.

1. At weekly meetings-reporting, community meetings, mutual help meetings, again adapted to existing computer systems

2. At monthly meetings – ‘community’ – often involve patients.

3. Data could be displayed in the patient lounge.

4. Every ward is different.

5. There was recognition that any technology would be experimental at first.

What this means

1. Whichever route is chosen, or chosen for any ward, means it would have to work within existing ward and trust systems. There may have to be several versions of the same technology, or a suitably plastic technology, to adapt to different wards structures, systems and conditions.

Potential harm to patients.

1. Only 1 of 10 of the respondents mentioned potential harms to patients but it worth considering.

2. We spend our time trying to … help patients relax and … slow down their thoughts and process … it can only harm them, to … say, your job today is watchman of the ward … that constant stress state that they’re going to be in, it’s going to harm them psychologically, it’s going to harm them physically.

What this means

1. We may need to offer patients different levels of engagement so that the most vulnerable to stress can contribute (or not) as they feel able to.

Technical brief

Technical approach

High-level technical architecture overview

The digital products developed will be built using open web standard technologies (e.g. HTML5, CSS3, JavaScript, PHP) in a componentised and scalable way, to allow modifications and future developments to easily happen.

All digital products will be hosted in a custom cloud environment built on top of the Amazon Web Services (AWS) Infrastructure. Technical details of the infrastructure are described below attached to each of the digital product's descriptions.

Figure 22 shows all of the digital products to be developed, how they interact with each other and sit within the wider context.

FIGURE 22.

Digital products in context.

Developed digital products

Recording interface

The recording interface facilitates the collection of patient safety data and measures within acute mental health wards. The interface is presented as an app on a tablet device on which patients are asked to answer a few simple questions about how safe they feel on the ward. No patient data is collected and all data collected on the interface is anonymous

Data are recorded in real time and sent to an API (see below) for data processing. The questions that patients are asked on the interface were determined from the environmental scan findings and co-design workshops. Particular attention has been paid to the UX of the interface, especially in understanding contextual constraints and pressures that patients may be experiencing (e.g. location when entering data, what device they will be using, what might their mental state be at the time of data entry). Similar attention to detail with respect to accessibility is also important to ensure the interface can be used by a wide range of users with varying abilities.

The interface has been built to WCAG 2.1 AA standards and is device agnostic (i.e. it is usable across most modern devices, platforms, operating systems and internet browsers). Patients are able to respond via a device supplied by ward staff.

Technologies used

The core technologies used to develop the recording interface are:

• HTML/CSS (HyperText Markup Language/cascading style sheets) – the underlying technologies of the web. HTML and CSS were used to develop the core structure and provide visual styling.

• ReactJS 17+ – an open-source JavaScript framework for building component-driven web applications and user interfaces.

• MaterialUI 4.11 – an open-source comprehensive suite of user interface tools that can be reused to develop interfaces faster and with consistency.

Infrastructure

Each ward has its own unique deployment of the core recording interface product with each ward having its own unique ‘instance’. The recording interface is completely separate from other wards and accessible via a unique URL to ensure an easy-to-use experience for patents submitting data and to avoid increasing the number of questions asked, for example, ‘which ward are you on?’. This approach also helped to eliminate the potential of malicious data submissions across wards.

Each ward has its own subdomain which is automatically generated using a random string of characters as the subdomain. For example, ‘{4–6 char}.wardsonar.co.uk’.

The instances of the product are stored in an S3 Bucket fronted with a Cloudfront distribution to handle caching and SSL termination. Route 53 DNS records have been updated automatically through a Cloudformation script which generates the unique subdomain values. Secrets manager has also been used to store API keys, credentials and configuration values for the recording interface.

FIGURE 23.

Patient interface.

Staff dashboard

The staff dashboard is the main interface used by staff to view data submitted by patients. It is primarily accessed in specific, authorised locations, such as the ward office and provides real-time snapshot data and greater informational insights through the use of interactive data visualisations.

Co-design work with healthcare staff, statisticians and researchers determined the exact requirements of how data should be represented on the dashboard, including barometer-style visualisation of ‘How is the ward feeling’, SPC charts, graph and statistical metrics. Monthly/weekly/daily/shift aggregate data is available and comparable to historic time periods.

Each ward has their own staff dashboards independent of other wards, so no data can be exposed between wards. The dashboards are behind a layer of authentication and can only be accessed by authorised staff members with login credentials.

FIGURE 24.

Screenshot of the ward F staff dashboard (identifiers removed).

Technologies used

The staff dashboards have been developed within the openly available Data Studio tool, which is produced and provided by Google. Budget and time constraints meant that an ‘off the shelf’ product for data visualisation – a scan of the market highlighted Google Data Studio as the most appropriate and cost effective.

Data Studio was connected securely to the underlying database where the raw data recorded by patients was held. This enabled much greater control over reporting and presenting of the data than would be possible through the relatively simple API that was created.

Having access directly to the raw data set meant data was able to be computed, reformatted and presented with the in-built scripting language, in the ways which the co-design work highlighted.

Infrastructure

The staff dashboards are hosted directly within the Google Infrastructure, however a couple of alterations to our infrastructure were required in order to achieve a stable and secure connection from Google to our database:

• A read-replica MySQL database was deployed – this is a read-only and real-time version of the live database but is exclusively used for data computation and reporting. Using a read-replica means these high resource intensive tasks do not affect the performance or stability of the master database.

• By default, the database within our infrastructure is not accessible publicly and is behind several layers of network security and firewalls. For the dashboards to connect, a secure link had to be created to allow Google servers to connect. This was achieved through whitelisting a set of intellectual properties that belong to Google, which allowed access into Virtual Private Cloud to connect to read-replica database instance.

Ward interface

The ward interface communicates the output from a specific measure for public consumption. Through co-design work, the specific outcome measure was determined to be ‘How does the ward atmosphere feel today?’. The outcome measure is displayed in the form of a barometer and shows the average milieu rating for the current shift.

The ward interface has been designed to displayed on a television, desktop computer monitor or tablet devices.

FIGURE 25.

Screenshot of the ward F interface (identifiers removed).

Technologies used

The ward interface is a subsection of the recording interface product, accessed by appending/dashboard to the URL and thus uses the exact same technologies.

Using the same code base for the ward interface allowed us to entirely reuse code and develop the interface much quicker, especially as the existence of the ward interface product was unknown from the start and something that co-design informed.

Application programming interface

An API, that is central to the digital products, has been developed in line with as many best practice government and health standards as feasible. The API is used to link all of the digital products together with the underlying data storage engine. It facilitates data recording, processing, representation and exporting. Where possible, we used an open data format standard so that the data can be interoperable with other systems/products (e.g. JSON, REST, OpenAPI).

The API has a one public endpoint (/stats) which returns a single computed payload that is used to power the ward interfaces. The rest of the API endpoints require a simple authentication bearer token which is unique to each ward. This token identifies where submissions are coming from, as well providing an authentication layer.

Technologies used

The core technologies used to develop the API are:

• PHP 7.2 – The underlying programming language used to create the API.

• Laravel 8.12 – An open-source application framework written in PHP that provides some reusable code for common tasks (e.g. validation).

• MySQL 5.7 – The central database engine used where all data is stored and accessed.

• PHPUnit: 9.3 – A library to help with automated testing of the API.

Infrastructure

The API is hosted on a dynamically sized group of application servers, fronted with a load balancer. All requests to the API first are forwarded to load balancer which then decides which application server to send the request to. This model allows for automatic scaling to handle traffic spikes.

The API code is packaged up into a Docker container (a unit of software which contains the application code and all of its dependencies) which is deployed to the application servers via the CI/CD pipeline Travis.

We make use of CloudFormation which allows most of the infrastructure to be templated and scripted, which automates all of the tasks of deployment.

Application programming interface exist at the endpoint (api.wardsonar.co.uk).

FIGURE 26.

Database schema extract.

TABLE 19 - Description of terms in database schema

Table name	Description
clients	Clients = wards. Each ward has an entry in this table with a unique id, secret (for authentication) and urlkey (random 4–6 char which forms the URL for the recording interface)
submissions	The main table where all user submissions are stored. Each submission is tied to a client_id (the ward they are on). If abandoned indicates a user didn’t want to submit data. Atmosphere and direction fields store the value of the user selection – see below data structure for the options.
causes	A populated table with predetermined options of ‘valid’ reasons/causes
cause_submission	A table that stores the reason/cause responses by users. Maps submissions to causes via their unique ids

Post deployment challenges

Throughout the deployment of the technology onto the wards and throughout the intervention, a small number of challenges came up. We highlight the main challenges faced here, along with what action was taken and future recommendations.

As the technology partner we provided a service level agreement throughout and were on-hand throughout the deployment and implementation process.

To aid with troubleshooting and debugging, application and infrastructure level monitoring was implemented through BugSnag and AWS respectively, to proactively monitor the application and platform for any issues.

What we’d do differently

Many of the challenges faced with the staff dashboard were related to the use and limitations of Google Data Studio, which was developed for much different use cases to this study. However, in terms of minimum required functionality and cost, it was deemed to be the most appropriate piece of technology to utilise.

Developing a bespoke dashboard that is designed around this specific use case should overcome the challenges and create a simpler, more refined experience. However, this clearly is budget dependent and would require investment.

Investing in training workshops on the wards facilitated by the technology partner prior to deployment would have reduced some of the friction and challenges faced – as a number of the issues were down to understanding/education about the system and its usage, not necessarily bugs or issues with the technology itself.

Annex (a) to technical report: user stories