All-Wales Licensed Premises Intervention (AWLPI): a randomised controlled trial of an intervention to reduce alcohol-related violence

High outlet density

The distribution of licensed premises in urban centres has been identified as a key contributor to levels of alcohol-related harm. 39 Areas with high concentrations of licensed premises have disproportionately higher levels of disorder, suggesting a cumulative, non-linear effect of outlet density. 40–42 Clearly, there can be no causal association between outlet density and harm, as areas with a large number of premises but few patrons would be expected to exhibit low levels of alcohol-related harm. High density most likely encourages behaviours that are associated with harm, such as pub hopping and competition between premises that leads to inappropriate promotions. While outlet density is not in itself in the control of premises staff, it would be possible for premises to mitigate those features that are causally associated with harm and are correlated with heightened outlet density, such as crowding and competition. 15

Customer management

Generalising from BWT, the entrance and façade of a premises inform potential customers of the characteristics within. While it is unclear if customers can accurately predict the risk of disorder in a premises based simply on approach, it is likely that interpretations of social norms are informed by these external characteristics. Therefore, door staff behaviour, queue management and the management of intoxicated or disorderly customers are fundamental in providing cues about descriptive and injunctive norms within. Interactions outside premises also represent potential flashpoints for disorder. The congregation of people outside after closing time represents a considerable risk factor, as this usually occurs at times when staff have finished their shift and are busy emptying the premises of its last few customers. This leaves the external area of the premises without a designated guardian at a time when customers are likely to be most intoxicated.

Security and door management

Generalising the social control theory outlined above, door staff represent both the expression and the actuality of informal social control, or guardianship, in a licensed premises. It is, therefore, important that they are adequately trained and present a professional and welcoming demeanour. In England and Wales it is illegal, under the Licensing Act, to allow disorderly conduct on premises. Furthermore, any member of staff who is authorised to prevent disorder and allows it to happen on the premises is legally culpable. Therefore, the refusal of admission to disorderly customers is regarded as a main role of door staff in England and Wales. The deployment of sufficient numbers is essential and should be informed by the capacity of the premises, the number of expected customers and past history of disorder.

In order to obtain a door licence, applicants are required to complete an examined training course. Moreover, in the event of a violent incident, customer ejection or injury on the premises, all details of the incident should be recorded clearly, accurately and promptly in a log book and subsequently reported to the police and, depending on circumstances, to local authority (LA) environmental health officers. Recording events linked to disorder, extreme intoxication and violence also enables premises managers to explore trends in disorder and to determine how to best allocate door staff.

Vigilant serving staff

Premises serving staff play a key role in the safe service of alcohol and the prevention of disorder, as they are responsible for refusing service to underage customers and intoxicated customers and for identifying signs of disorder within the premises. It is essential that serving staff are aware of their legal responsibilities and that they take these responsibilities seriously. Server training has been shown to have limited, short-term effects in improving serving practices in respect of the refusal of service to intoxicated patrons. 8 As serving staff act as informal guardians, it is important that sufficient numbers of staff are deployed in order to facilitate this role. Insufficient numbers of serving staff are associated with increased levels of disorder in a premises, as this increases competition for service between customers,43 as well as crowding. 15 Furthermore, a premises with a high proportion of male staff is associated with disorder in licensed premises,44 although this phenomenon may be a reaction to past disorder as opposed to a causal factor. Clarke45 further suggests that an overly sexualised dress code for female serving staff can contribute to heightened levels of arousal in a premises and this is further implicated in disorder.

Environmental factors

A number of studies have aimed to identify the environmental aspects of a premises that influence the likelihood of disorder. Graham et al. 46 conducted a detailed multilevel analysis of risk factors for bars in Toronto, ON, while Green and Plant38 collated a detailed description of these risk factors. Evidence suggests that showing sport in premises increases the length of customers’ visits47 and is associated with increased levels of aggression. Music has also been related to levels of disorder and intoxication. For example, poor-quality bands can be an irritant,48 while slower tempo country music is associated with an increase in drinking speed. 49 Loud music may further impair communication between customers, preventing the de-escalation of fractious encounters. A range of other environmental factors can act as irritants such as poor air quality,43 increased temperature50–52 and uncomfortable furniture. 43,53 Moreover, dim lighting reduces the capacity for formal surveillance by premises guardians, impairs communication and increases the likelihood of collisions. Areas that are difficult to view and guard, such as thoroughfares and stairways, can also provide increased opportunities for collisions and injuries. Glassware also presents a significant risk factor for serious injury owing to its portability, accessibility and the level of harm that can be caused with a single blow. 12 Furthermore, the presence of empty glasses and other litter on tables may signal low levels of social order, and there is a relationship between untidy premises and disorder. 19,44,48,53,54

Promotions

Stockwell et al. 55 found that alcohol promotions were associated with intoxication but not associated with the risk of alcohol-related harm. More recently, however, studies suggest that promotions and becoming drunk are associated with police-recorded violence. 14

Customer behaviour and characteristics

Disorderly customer behaviour, according to BWT, contributes to perceptions of a permissive environment, thereby increasing the likelihood of further disorder. However, the relationship between gender ratio and disorder risk is unclear and evidence is scarce. 9 Although men are at far greater risk of violence,4 the presence of women may actually serve as a risk factor for violence owing to competition for sexual resources. Similarly, while a younger age is associated with increased risk of violence, customer age is not a strong predictor of disorder. However, age may interact with several other factors, such as premises type, which ultimately contributes to disorder. Typically, persistent offenders who are frequently violent when intoxicated56 are usually well known, emphasising the need for door staff and premises managers to share data across premises in an area.

Intervention development and refinement

In order to develop a suitable and workable intervention, the AWLPI trial incorporated an intervention refinement stage, which involved collaboration with senior EHPs, researchers and web media experts. The specific aim of this stage was to transfer the theoretical and empirical motivations for the intervention into a practicable and accessible evidence-based intervention programme. The intervention refinement stage of the trial is described fully below.

Trial stage

The effectiveness of the intervention developed in the intervention refinement stage of the trial was assessed using a RCT design described more fully in Chapter 4. The primary outcome in this trial was the difference in police-recorded violence between intervention and control premises over the follow-up period. Secondary outcomes were assessed using an embedded process evaluation (see Chapter 6) and economic evaluation (see Chapter 7).

Overview

The previous chapter described the theory behind premises-level interventions and areas of premises operation that could be targeted so that improvements would bring about a reduction in alcohol-related harm. This chapter describes how formative work was translated for use by EHPs and into a format that would be acceptable to premises staff.

Key research findings from a feasibility study of a risk-led intervention showed that (1) an enhanced multiple risk-audit approach can successfully identify appropriate targets and approaches to prevention; (2) the engagement of licensed premises and intervention efficacy were maximised when implemented by statutory authorities; and (3) police-recorded data on violent incidents were a valid measure of harm, sensitive to change at the premises level. 69

The theoretical basis to the intervention was that identifying risk of alcohol-related harm and motivating changes in premises to mitigate those risks would be expected to reduce alcohol-related harm. This approach is enabled by current legislation. The Licensing Act 2003 places a requirement on DPSs to adjust premises operation if they become aware that their operation increases the risk of harm. 13 Therefore, practitioners who are able to identify and advise premises staff on such risks should expect their advice to be heeded and that premises will respond to such advice. Furthermore, and specific to EHPs, the HSWA facilitates practitioners’ access to premises and premises staff and also affords EHPs the statutory remit to investigate workplace violence. 64 Thus, EHPs provide formal governance and are able to require change in small businesses to reduce harm.

While EHPs were not susceptible to the intervention barriers identified in the feasibility trial, there still existed a need to translate formative theoretical work into a format that was both consistent with EHPs’ normal working practice and acceptable to premises staff. Intervention materials were coproduced with EHPs to translate formative work into a format that communicated premises, obligations to minimise harm in a way that was acceptable to all premises staff (e.g. via an accessible website and informative films). The expectation was that these materials would engage all staff in the premises hierarchy, from door security staff to servers and management. Second, a risk audit was developed that covered multiple risks. This comprised a written booklet that both outlined the statutory and research evidence for areas in which EHPs should attend and provided a uniform means of identifying and collating evidence for risk in premises (see Appendix 1). The risk audit was intentionally developed from earlier audits used by EHPs.

All-Wales Licensed Premises Intervention aims and objectives

The primary research aim for this component of the trial was to refine a risk-led intervention that aimed to identify environmental risks in premises and that could be subsequently mainstreamed into EHP usual practice.

Audit development

Three core members of the research team and five senior EHPs were involved in the AWLPI audit development. Development began initially by identifying the most pertinent indicators on an EHP health and safety audit that had previously been developed by EHPs. Selected indicators were translated into a series of items and mapped onto a spreadsheet to provide a comprehensive matrix whereby each item was associated with a potential response. This matrix was sent to the subgroup EHPs for feedback. This process was repeated during the development phase until all parties were satisfied. The finalised matrix was then formatted in the style of a health and safety audit (see Appendix 1). The AWLPI audit was then piloted by EHPs (naive to the AWLPI trial) to check for completeness, accuracy and ease of use. Final iterations of the audit were made as recommended in the piloting feedback.

Web-based training and instructional materials

Representatives from a web development company, a graphic design company, the British Association of Anger Management, an organisation representing door supervisors and a communications consultancy along with three core members of the research team were involved in the development of the AWLPI website and online training films.

The focus of the short films was explored during the developmental meetings between the research team and EHPs. The overarching topics chosen for the films were influenced by the literature, previous research conducted in licensed premises and industrial experience of both the researchers and the EHPs. The visual concepts of the films were developed using storyboards.

Once built, the system underwent user acceptance testing by the research team and all content on the website was checked by EHPs to ensure legal compliance.

Risk audit

The risk audit included items that could be used to describe basic features of the premises being audited. This included the number of full- and part-time staff, whether or not food was served and whether or not the premises hosted live music. The date and time of the audit was also recorded as well as the distance travelled by the EHP to the premises.

Eleven operational domains were examined and included:

1. Records management: written risk assessments, related policies, premises opening/closing logs and incident logs must be available and up to date. Incidents logged should be cross-referenced with policies and risk assessments to assess whether or not action had been taken to minimise the risk of further harm.

2. Visibility and lighting: visibility and lighting should be good throughout the premises. Blind areas can impede surveillance by premises staff, as can low levels of lighting.

3. Health and safety observations and checks: heating, ventilation and the overall condition of the premises were assessed.

4. Surveillance: surveillance arrangements should be sufficient to protect health and safety. This includes where security staff are usually located and the use of closed-circuit television (CCTV).

5. Noise and communication: staff should be able to communicate with each other effectively about potential risks during times when the premises is open.

6. Risk planning: there should be evidence of regular engagement with PubWatch (www.nationalpubwatch.org.uk) or similar. There should be no evidence of irresponsible drinks promotions. There should be sufficient numbers of front-of-house staff present during busy periods.

7. Door management: effective door management during busy periods should be in place. This includes maintaining appropriate door-staff registers and policies.

8. Managing people: visibly intoxicated and/or disorderly patrons should be effectively managed. Those who are disorderly must be escorted off the premises and those who are intoxicated must not be served alcohol.

9. Staff training: there should be evidence of staff induction and ongoing training that encompasses disorder, violence and aggression.

10. Incident reporting: it is a legal requirement to record violent incidents that have occurred on the premises, records that should inform future practice.

11. Glassware policy: literature suggests that the use of toughened glass or plastic reduces risk of injury.

Web-based training and instructional materials

Films

Training and educational films were designed to increase awareness of the physical and social environment, and increase knowledge of policies and practices that prevent and reduce excessive alcohol consumption and violence (Figure 1 and Table 1). Each film included animations, summary text and a spoken script. The messages of the films were positively framed by focusing on the benefits to be enjoyed from implementing the techniques demonstrated in the films rather than dramatising the consequences.

FIGURE 1.

An example of a storyboard used to inform the design of the short films. ID, identification; VO, voiceover.

TABLE 1 - An excerpt from one of the scripts

Chapter One: the benefits of a safer environment
Voiceover: creating a safe environment not only provides a better atmosphere for your customers; it can also help to improve staff retention by making your venue a more pleasant place to work. And with happy customers and happy staff, your reputation can bring you more business Everyone working at the venue can play a role to help reduce risk within your venue. And it needn’t be time-consuming; this video provides a quick snapshot of the ways staff can get involved and work together to provide a safer environment	Visual: presenter piece to camera, walking through scene of happy customers, happy staff, and busy bar
Duration: 30 seconds

The training and educational films provided guidance in the following areas: premises environment, security, crowding and how to de-escalate fractious encounters between customers. The four film topics were:

1. safety and your colleagues

2. keeping them happy

3. tables and chairs

4. staggering crowds.

Angling the films towards depictions of emotions, and anger in particular, was felt to be a good inclusion, as staff may feel empowered to recognise and understand conflict at the bar more easily. The films also aimed to communicate that customers’ pride (or ‘power base’) needs to be kept intact throughout interventions in order to avoid escalating or displacing aggression.

Website

The intervention website contained information about harm-reduction practices in licensed premises and provided guidance documents that could be downloaded and used by premises staff (Figure 2). The website also contained a due-diligence quiz that was designed to provide instruction on how premises staff can reduce excessive alcohol consumption and violence. The website was made available in English and Welsh. 70 A diagram of the site layout is in Appendix 2.

FIGURE 2.

The homepage of the SMILE website.

Intervention input

The key ingredients during this phase of the intervention are (1) the auditor, (2) the audit and (3) the training films.

1. The auditor is an EHP who has been trained to conduct the audit in a systematic and standardised way.

2. The audit materials consist of:

   1. the 11-point audit of the premises, this is grouped into:

      1. – the physical environment

      2. – policies and procedures

      3. – staff training

      4. – risk assessment and planning

   2. guidance leaflets and notices, to be given at the end of the audit if necessary.

3. Training films, which were accessed using the SMILE website.

Expected output arising from the audit

The second component of the intervention is the generation of an action plan arising from the audit itself. The action plan given to the DPS should be tailor made for each premises as a result of the findings from the audit. There is a need for this to be a list of points (‘tell me what to do’) rather than a long document. The way the action plan is written should take HSE reports on how small businesses perceive and implement legislation into consideration.

Conducting a follow-up audit

Depending on the severity of the risks identified during the initial audit the EHP would serve an improvement notice or a prohibition notice, or arrange for a second audit to take place. When a formal notice is served the EHP would specify the provision(s) that had been contravened and give reason(s) why they feel these had been contravened. The requirements necessary to remedy the contravention should then be given followed by a reasonable time period in which to comply with the notice (not fewer than 21 days). A second audit would be arranged to ensure that premises had complied with directions.

As a part of the intervention, second audits were expected to have been completed within 3 months after the intervention to check that actions had been taken. In premises where no risks were found, no further visits would be necessary and usual practice would resume.

Expected outcome

It was expected that, through dialogue with auditors immediately following the audit, DPSs would feel some ownership over the action plan they receive. This was key in facilitating premises to carry out the action points, as any disagreement over risk factors should have been resolved and there should not have been any surprises in the action plan.

Trial population

The population comprised premises licensed for the on-site sale and consumption of alcohol residing within any of the 22 Welsh LAs.

Closure and replacement premises

It was anticipated that a number of premises screened and allocated to the intervention or control groups might cease business or become ineligible (e.g. changing to a restaurant) before the intervention was to be conducted. If premises in the intervention arm became ineligible before the intervention phase began, it was replaced with a premises randomly selected from a list of any remaining premises in that LA, matched by strata. When a replacement premises meeting the necessary strata criteria was unavailable the premises was not replaced. Following the intervention period, the research team investigated closure of premises (in both experimental arms) by contacting LAs. Duplicate premises, or premises in the intervention arm that did not co-operate with EHPs or could not be accessed by EHPs in the allotted intervention period, were not replaced with any of the remaining premises.

Permission to participate

Environmental health practitioner intervention training

At least one EHP from each LA attended one training workshop. The EHPs were provided with a training manual and were supported throughout the trial period by the research team.

Data collection

Sample size calculation

A previous exploratory trial21 suggested that an overall group size of 274 premises provided a power of 90% to detect a 10% reduction in the failure rate (a day in which one or more violent incidents occurred) at a significance level of 0.05. Attrition was not a factor in this trial, as EHPs have a statutory authority to enter premises and therefore adjustments for withdrawals were not applicable. Temporary and permanent suspensions of premises licences are valid outcomes, as they represent a form of intervention and are, therefore, accounted for in the group size. A 12-month follow-up period was chosen (12 months following delivery of the first audit), as the earlier exploratory trial suggested that this offered the most likely opportunity to detect a significant effect in a full trial and could control for annualised cyclical variation within premises.

Randomisation

The unit of randomisation was each licensed premises within the 22 LAs in Wales. There were two groups (intervention and control). The total sample size of eligible premises for the randomisation was 837. A minimum sample size of 274 licensed premises per arm was required but this was rounded up to 600 total for simplicity. Therefore, 600 premises, stratified by LA, were randomly selected from the total eligible population for randomisation. Licensed premises were randomly assigned to intervention and control in a 1 : 1 ratio. Optimal allocation was used to randomise licensed premises to intervention or control, stratified by LA. Opening hours [low (0–4 hours after 11 p.m.), high (≥ 5 hours after 11 p.m.)] and number of incidents [low (1 and 2 incidents), high (≥ 3 incidents)] were used to balance the randomisation. 72 LAs that did not have the capacity to carry out as many audits as required for the randomisation were not supplemented with other LA premises, as EHPs do not generally go beyond their boundary, and also the approaches can differ. Premises that were closed prior to intervention delivery were replaced with premises randomly selected from a matched list of any remaining premises within that LA. However, as these spare premises were not randomised they were included only in sensitivity analyses.

Allocation by local authority

The number of premises in each LA receiving the intervention was determined by LA population. Table 3 shows the available number of eligible premises by LA and the number of required premises for the intervention group according to LA population. For information, the same allocation process was repeated for total violent crime by each LA and the number of audits conducted by EHPs in a similar but unrelated project completed the year before the current project began.

Because the number of available premises was not sufficient to meet what was required for some LAs, these LAs had their number reduced to the maximum available. The number of premises required to meet the target sample size was met through increasing the number of premises in those areas where there were sufficient premises and according to LA population (Table 4).

Treatment of trial data

Analytic strategy

The primary analysis was a comparison of police-recorded violence between intervention and control premises over the follow-up period, with time-zero being 1 January 2013, the earliest conceivable date an audit could be delivered to a licensed premises, for the ITT group. No interim analyses were undertaken. The primary data set used those data extracted using automated search procedures; sensitivity analyses were conducted on the manually extracted data.

Using the date and time of violent incidents, incidents were organised into sessions. A session was defined as 12 p.m. to 12 p.m. the following day and took the date of the first 12-hour period. Each session was coded with a binary (0, 1) indicator. If a premises had registered one or more violent incidents in a session that session was coded 1; otherwise, it was coded 0.

The Andersen–Gill model was used to analyse failure in premises in the follow-up period in order to account for potential time-varying covariates, censoring, multiple events and discontinuous risk intervals. 73 All incidents of violence at premises from the police forces records were recurrent events with the intervention effect as one predictor, also adjusted by opening hours and number of previous incidents strata (both coded 0, 1) at baseline. As the randomisation was stratified by LA the primary analysis was further explored by the addition of LA as shared frailty. This allowed for any potential clustering to be examined.

Sensitivity analyses were conducted on the per-protocol and non-randomised groups.

Dose was primarily explored through nature of feedback provided by EHPs (none, formal advice) and by any additional follow-up audits undertaken to enforce changes recommended to premises.

Secondary analysis of primary outcomes was undertaken using the Andersen–Gill model to assess the hypothesised intervention wane over the follow-up period. This was achieved by the use of a time by treatment interaction term.

In addition, secondary analyses were used to explore the effect of the intervention on the volume of violence (total counts of incidents) attributable to trial premises, to inform the economic analyses.

Additional exploratory analyses considered the relationship between health and safety issues, incident reporting and EHP confidence in management. In premises with repeat audits the changes were examined in relation to reports of violent incidents. Audit data were also used as part of the process evaluation detailed in Chapter 6.

Intervention costs

We report the costs that would be incurred if the intervention was to be introduced into routine practice. Costs associated with developing the intervention are thus considered as historic and are excluded. As SMILE was additional to usual practice, all costs are incremental. All costs are reported in 2014 prices. As the study period was 1 year, no discounting has been applied.

There are three main elements to the costing of this intervention: training EHPs, the audit process and costs incurred by the licensees as a direct result of advice received during the intervention.

Training costs were all recorded prospectively. These included trainer and trainee time, travel costs, venue fee and administration costs. The hourly value of trainer and trainee time was based on reported salary plus on-costs assuming a 42-week working year and a 37.5-hour working week (assumption from Curtis). 74 Where these details were not provided, the median salary/on-costs were imputed. Travel costs included both time spent travelling and car mileage cost of 40 pence per mile. 75 Other costs such as materials and travel expenses were recorded in money terms.

The cost of printing training manuals, audit documentation, etc., was based on number of pages charged at 5 pence per page (assuming black-and-white printing). Postage was via the Royal Mail’s price finder for first-class postage of large envelopes at £0.93 and small parcels at £5.45. 76

While training could be considered a one-off investment producing a flow of benefits over time, we attributed the whole of training to the sites which received the intervention during the period of study on the basis that retraining would be required annually owing to changes in legislation, etc. It was likely that costs of retraining would be lower than those of initial training and this was explored via sensitivity analysis.

Audit costs included EHP time and travel for the audit (initial plus follow-up, where these occurred) of each premises.

As EHPs have a statutory right to undertake the audit, participation in the study did not require informed consent from licensees of intervention premises or their DPSs. Accordingly, we felt that attempting to identify the costs of implementing any improvements that arose as a result of the audit by questionnaire would produce a low response rate, which would be potentially biased. However, intervention and control premises that participated in the process evaluation did so via informed consent. This provided an opportunity to gain information on licensee-borne costs through the addition of a number of questions to the interview schedules.

Cost of violent incidents

As our data on violent incidents include only those which have been reported to the police, our cost estimates should be regarded as minimal.

The costs of violent incidents were taken from the Home Office data on crimes against the person77 in line with Dubourg,78 which was the basis of many of the Home Office estimates. However, as the Home Office estimates currently exclude three elements which were in Dubourg,78 ‘lost output’, ‘physical and emotional impact on direct victim’ and ‘NHS costs’, we have included these here using the estimates in Dubourg inflated to 2014 prices. As not all violent incidents will incur criminal justice system costs, we have explored via sensitivity analysis the effect of reducing the percentage of incidents which incur criminal justice system costs. These base-case unit costs of a violent incident are reported in Table 5.

This cost estimate of a violent incident was then multiplied by the number of total incidents that occurred during the follow-up period in order to calculate costs of crimes at premises level.

Effects

The unit of effectiveness for the cost-effectiveness analysis was number of sessions with one or more violent incidents, based on police records.

Cost-effectiveness

The cost-effectiveness analysis was carried out on premises in the ITT sample. The total cost for intervention premises includes the intervention cost plus costs of violent incidents that occurred during the follow-up period. For control premises this was only the cost of violent incidents. As costs data are likely to be skewed, a non-parametric bootstrapped approach was applied to estimate bias-corrected accelerated (BCa) 95% confidence intervals (CIs) for differences in costs. The mean differences in total costs were further adjusted for baseline incidence group and opening hours, in line with the primary analyses.

If the intervention group was less costly and more effective, the intervention would be shown to be unambiguously cost-effective. If the intervention group was more effective but more costly, then results will be reported in the form of an incremental cost-effectiveness ratio showing the additional cost per unit of effect. This would provide an evidence-based benchmark for comparison in future cost-effectiveness evaluations of interventions aimed at reducing violence in the NTE. As some premises had follow-up audits, a separate exploratory economic analysis was done to investigate the cost-effectiveness of the intervention where it included a follow-up audit.

Dealing with uncertainty

Two one-way sensitivity analyses were undertaken to examine how sensitive cost-effectiveness results are to changes in base-case parameters. In the first, the assumption that training costs are apportioned to premises that received the intervention during the study period was replaced by an assumption that training lasts for 5 years. In the second, it was assumed that only 50% of police-recorded violent incidents incur criminal justice system costs.

To explore whether or not the intervention represents good value for money, a stochastic sensitivity analysis was carried out using a non-parametric bootstrapped approach on the joint distribution of costs and effects. Results are reported in the form of a cost-effectiveness acceptability curve (CEAC) showing the probability that the extra cost per unit of effect lay below a range of societal willingness-to-pay thresholds.

Aims and objectives

The process evaluation aimed to explore the intrinsic processes operating during SMILE adoption and implementation and their influence on what was delivered.

The process evaluation objectives were to:

1. understand the adoption of the intervention by EHPs and the adaptation required to integrate SMILE into normal practice

2. assess intervention implementation – reach, fidelity and dose delivered

3. assess intervention efficacy – receipt and acceptability

4. understand the implementation context for intervention maintenance and any intervention reconfiguration

5. construct a logic model of the intervention as delivered.

Process evaluation design

Table 6 outlines the research design and associated methods that were used to explore each of the research aims in three successive phases. Phase 1 explored the intervention codevelopment and adoption within the environmental health organisation; data were obtained from focus groups with senior EHPs involved in SMILE coproduction and implementation. Phase 2 investigated intervention implementation through interviews with EHPs responsible for delivering the intervention and explored issues of reach, fidelity, dose delivered and longer-term maintenance. Phase 3 examined how the intervention was received by licensed premises. Its perceived efficacy, acceptability and likely sustainability were explored through semistructured interviews with a sample of licensed premises staff. Results from these phases were drawn together to construct a logic model of intervention delivery (research objective 5) and refine the intervention. Ethical approval for the process evaluation was given by the Cardiff University Dental School Research Ethics Committee.

Sampling and recruitment

The process evaluation explored SMILE development and implementation with senior EHPs and a representative sample of EHPs who delivered SMILE. Two focus groups were conducted with senior EHPs (n = 4). The first focus group that was conducted after the main phase of SMILE coproduction had been completed sought to explore SMILE codevelopment and adoption. The second focus group took place following delivery of the intervention and was concerned with intervention implementation. The senior EHPs were invited to take part verbally; information sheets and consent forms were completed at one of the regular research group meetings.

Telephone interviews were conducted with one EHP from each LA in Wales (n = 22) who had been involved in the delivery of SMILE. EHP recruitment took place by e-mail or telephone, with information and consent forms supplied and returned electronically. Purposive sampling ensured that practitioners working in premises with the greatest possible variation of risk in both control and intervention arms and significant experience of delivering SMILE were invited to participate. In order to explore the reach and impact of the AWLPI training, three EHPs who had not attended SMILE training workshops were also purposively sampled.

To explore intervention receipt and its relationship with normal practice within premises, a sample of licensed premises were drawn from control (n = 321) and intervention premises (n = 285). Purposeful sampling ensured that these premises represented those found in a range of geographical areas across Wales (North, West and South East) and location (urban, rural, town/fringe) and previous incidences of violence (high/low). EHPs who had delivered SMILE to each premises were asked to provide the owner/manager with details of the study and request permission for a researcher to contact them to discuss participation. Where the EHP failed to gain a response (n = 4) or the DPS refused participation (n = 2), a researcher contacted the DPS directly or selected new premises and conducted recruitment themselves. Recruitment continued until theoretical saturation was reached. This resulted in data from 30 premises across Wales (Table 7).

Data collection

Each focus group with senior EHPs was facilitated by two researchers. The focus group schedules for the two focus groups contained different guides and prompts, designed to focus on the RE-AIM framework (see Appendices 4 and 5). The initial focus group was held to gain better understanding of the routine work of the environmental health agency; organisational views on increasing their role in addressing ARV; and some insight into senior environmental health managers’ role in adopting and developing SMILE. The post-implementation focus group encouraged reflection on the process of SMILE implementation, and on how well the interventions had been embedded in environmental health practice.

Semistructured interview schedules for EHPs again drew on the RE-AIM framework (see Appendix 4) and were conducted by a single researcher. Most EHP interviews were conducted by telephone, although one EHP requested a face-to-face meeting. All interviews took place as soon as possible after the intervention phase had ended, as the aim was to identify and explore the process of intervention delivery in varied premises and different parts of Wales, with particular interest in barriers and facilitators experienced during SMILE delivery and assimilation.

Interviews with premises managers/owners (n = 30) were conducted face to face when possible (n = 18) or by telephone (n = 12). All semistructured interviews were held shortly after the intervention phase had ended and were organised around RE-AIM constructs (see Appendix 5).

Data analysis

All focus groups and interviews were audio recorded. Recordings of interviews and focus groups were transcribed, anonymised and entered into password-protected university files before analysis using NVivo 10 (QSR International, Warrington, UK). All transcripts were scrutinised for errors. The first phase of analysis categorised data into dominant themes determined a priori by the constructs of the RE-AIM:81

• SMILE adoption and diffusion

• SMILE implementation – reach, fidelity and dosed delivered

• SMILE efficacy – acceptability and receipt

• SMILE maintenance – reconfiguration and sustainability.

Subsequent analysis drew on NPT constructs to explore whether or not and to what extent the use of a TDAR approach facilitated these processes. Using this approach, an analytical framework matrix was generated and used to construct a logic model, which visually describes the entire intervention process.

Aims

• To investigate the impact of the SMILE intervention on police-recorded violence.

• To assess the reach and dose delivered of the intervention.

Objectives

• To analyse police data from 1 January 2012 for 455 days and determine whether or not the intervention was associated with a change in violent-event frequency.

• To conduct sensitivity analyses to determine the robustness of any intervention effect.

• To analyse police data from 1 January 2012 for 455 days and determine whether or not the effect of the intervention, if any, changed over time (e.g. intervention wane).

• To use data on delivery of the audits to determine the proportion of premises that received the intervention (reach) and what actions resulting from the audits were needed to reduce premises-level risk (dose).

Manual data

Police-recorded violent crime data were manually checked by researchers who sought to identify all premises included in the study: 20% of the data from each of the four police forces was randomly selected and sent to an independent researcher, who completed the same matching process. Proportion of agreement was calculated for each force (0.97, 0.98, 0.97, 0.98) and yielded an overall proportion, where raters agreed, of 0.973.

Automatic data

Baseline data, together with study premises name and address information, were used as a training data set for a bespoke automated text search algorithm that was created to identify study premises in police violent crime data (see Chapter 4, Automatic data).

Summing the total number of incidents identified using each method by day suggests the manual process identified more incidents [daily mean = 6.69, standard deviation (SD) = 5.27] than the automated approach (mean = 3.74, SD = 3.04; t = 19.38, p < 0.001); although there was a robust association over time (Spearman’s ρ = 0.83, p < 0.0001) indicating that rank was preserved over the two approaches.

The reason for these differences is that free text in police data would occasionally use abbreviations that were not picked up in the automated search, particularly where GPS codes were missing. Similarly, records would occasionally not have address information and just provide GPS co-ordinates.

Strata

The baseline violent crime data were used to stratify premises into high-violence (> 2 incidents across the baseline period) and low-violence (≤ 2 incidents) premises. Cut-points were determined through k-means cluster analysis. Opening hours past 11 p.m. on Friday and Saturday nights was also used to stratify premises. Premises open a total of ≥ 4 hours on Fridays and Saturdays constituted the ‘high’ group; premises open for < 4 hours in total constituted the ‘low’ group. Dummy variables were created for each (low = 0, high = 1 for each) and were entered into analyses as covariates.

Capacity was not used to stratify premises, the reason being that these data are not reliably available. Of the 837 eligible premises, capacity data were available for 144 from LA licensing records. For these 144 premises, baseline total violence was associated with capacity (Spearman’s ρ = 0.38, p < 0.0001), suggesting that stratifying on baseline violence was sufficient.

Censoring

All businesses are expected to pay business rates, a form of local taxation. Businesses, including licensed premises that cease to trade whether temporarily or permanently, do not pay these rates. LA business data were, therefore, accessed for all study premises to determine periods of economic inactivity. Violent offences recorded for a premises during a period of inactivity were dropped (many premises are also private residences and can, therefore, experience violence even if they are not operational). In total, six incidents were dropped from automated data and 17 were dropped from the manual data.

Audit completion rate

All intervention premises were audited by EHPs from January 2013 onwards. All premises had received the initial audit by 29 April 2013. A subset of premises received a follow-up audit. All follow-up audits (n = 18) had been completed by 4 June 2013. Figure 6 presents the temporal distribution of completed audits and follow-up audits and suggests that EHPs delivered the intervention in the expected time frame.

FIGURE 6.

Proportion of intervention premises audited by analysis time (0 = 1 January 2013).

Baseline data

Premises were stratified by opening hours and baseline violence. Table 8 presents descriptive statistics for each group.

Follow-up data

Subgroup analysis

To further explore the effect of the intervention on violence, intervention premises were divided into two groups, according to nature of the feedback given: (1) premises where EHPs did not identify any areas of risk and therefore premises received no advice and (2) premises where EHPs did identify areas of risk and provided advice (Figure 8). For the ITT group, using the automated data and controlling for violence and opening hours groups, premises receiving feedback (n = 217) yielded a lower incident rate than those that did not (n = 21; HR = 0.51, 95% CI 0.42 to 0.63).

FIGURE 8.

Subgroup analysis comparing intervention premises receiving EHP feedback with those receiving no feedback.

Secondary analyses

One explanation for an increase in police-recorded violence in this context was that premises receive greater attention from the police. As licensing committees involve the police, LA licensing practitioners and other responsible authorities, it is feasible that referral to any of these responsible authorities would mean that premises are placed under greater scrutiny and, therefore, the police are more likely to record violence associated with these premises. In the audit data there are two proxies for greater scrutiny: whether intervention premises received a referral to a responsible authority or were issued a formal notice. In the ITT group, 27 premises were referred to at least one responsible authority, seven premises received a formal notice and two premises received both. To simplify analyses a single referral indicator was created (1 if referred to a responsible authority or received a formal notice, 0 otherwise). Rerunning the main analyses (ITT group, automated data), interacted with the audit indicator yields a significant positive effect (Table 11). No premises receiving a follow-up audit received a referral or notice.

A Wald test on the equivalence of coefficients on audited with and without referral suggests additional referral activity increases event frequency (χ² = 16.39, p < 0.001).

Duration of audits

Data presented here are for the full non-randomised group.

The first audits took, on average, 82.62 minutes (SD = 34.49 minutes), with an average travel time to the premises of 17.8 minutes (SD = 11.75 minutes). Follow-up audits (n = 18) took, on average, 28.44 minutes (SD = 13.87 minutes), with an average travel time to premises of 13.39 minutes (SD = 7.99 minutes). All follow-up audits were conducted within 3 months of the initial audit (Figure 9).

FIGURE 9.

Number of premises by duration of audit.

Evidence of risk assessments

Of all premises, 73% (n = 204) had ≥ 5 members of staff and were, therefore, eligible to have completed a written risk assessment. Of these, 204 premises (70%) did not have a written risk assessment. In addition, 12 premises with < 5 members of staff did have a written risk assessment.

Risk control indicator outcomes

For each domain, EHPs completed a RCI, a Likert scale that ranged from 0 to 6. A score of 0 denotes not applicable and these scores were dropped from summary statistics. A score of 1 denotes good practice, while a score of 6 denotes heightened levels of risk for that domain. The expectation was that scores of 4–6 would warrant further action in respect of that domain. Figure 10 presents results from the RCI scores and shows that the areas requiring most attention included, first, record keeping (including written risk assessments), second, health and safety and, third, incident reporting.

FIGURE 10.

Mean RCIs (with standard error bars) and percentage of premises scoring ≥ 4 and, therefore, requiring further action.

Overall, 34.88% of premises audited scored > 3 on at least one RCI, warranting enforcement, while 6.41% received a follow-up audit.

Actions resulting from the audit

Actions available to EHPs were written and verbal advice, improvement notices (five premises received one, one premises received two) and prohibition notices (three premises received one prohibition notice) and referral to the police (none were referred to the police), referral to fire services (six premises received one referral and one premises received referrals for two issues), and referral to LA licensing (14 premises were referred for one issue, four premises were referred to two reasons and two premises were referred for three reasons). Reasons for referral to LA licensing were for premises not operating according to their licensing conditions. Notices covered lack of safety policies and records (n = 3), inadequate staff training (n = 1), poor condition of the premises (n = 2), poor lighting (n = 1) and significant failing in respect of gas safety (n = 1). In addition, one premises received a prohibition notice for inadequate fire safety but also demonstrated failings with regard to adequate CCTV and staff training.

In total, 24 premises (8.5%) received no verbal or written advice and no prohibition or improvement notices. Written and verbal advice could be given once for each audit domain, yielding a maximum of 11 for both verbal and written advice for each premises (Figure 11). On average, verbal advice was given more frequently (mean = 5.62 per premises, SD = 3.91 per premises) than written advice (mean = 1.97 per premises, SD = 2.22 per premises; t = 16.41, p < 0.001).

FIGURE 11.

The number of premises receiving written and verbal advice. The maximum number of areas on which premises could receive advice was 11.

Perceived efficacy in premises management

Environmental health practitioners rated their confidence in premises management on a Likert 1–6 scale across eight domains (Figure 12). Overall, items varied little both across and within premises and were associated with the numbers of domains receiving verbal advice (Spearman’s ρ > 0.15 and p < 0.01 for each comparison, except organising co-operation and communication, where Spearman’s ρ = 0.12 and p = 0.06) and the number of domains receiving written advice (Spearman’s ρ > 0.32 and p < 0.001 for each comparison) with lower confidence being associated with a greater likelihood of verbal or written advice.

FIGURE 12.

Confidence in management (6 indicates low confidence, 1 indicates high confidence).

Intervention premises characteristics

Opening and closing times

Premises were more likely to stay open later on Friday and Saturday evenings, with 8 p.m. to 11.30 p.m. being those times when premises were busiest (Figures 13 and 14). In total, 99% of premises were open both on a Friday and on a Saturday; with regard to these premises they would be open on average for 3.34 hours in total after 11 p.m. on a Friday and Saturday evening, with those premises open longest also experiencing greater levels of baseline violence (Spearman’s ρ = 0.30, p < 0.001).

FIGURE 13.

Intervention premises self-report busiest periods.

FIGURE 14.

Mean (with standard error bars) premises hours open after 12 p.m. by day of the week.

SMILE adoption

Intervention training

Following coproduction of the intervention at least one EHP from each of the 22 Welsh LAs attended a training day. These days aimed to increase wider organisational understanding at both group and individual levels by increasing knowledge and understanding of the intervention format and rationale within the extended EHP body. The training days consisted of expert presentations drawing attention to the extent of violence within the NTE, the potential role of EHPs and the aims and objectives of SMILE. The days also saw senior EHPs presenting the SMILE risk audit in detail and discussing it with the wider EHP body. SMILE audit tools and contact details for allocated intervention premises were distributed to LA environmental health teams once the training days were complete.

In total 74 EHPs attended the training days, a high attendance level which increased the organisational reach of SMILE. Comments from attendees suggested the days increased both individual and communal understanding of the intervention by providing details of project background, rationale and the proposed role of EHPs within it. Most EHPs felt that the research team presentations were useful as they gave ‘[G]ood insight to the background, you know to see where it comes from, the thought behind it and, you know, what you hoped to achieve from doing the project’ (EHP2) while additional medical presentations offered ‘visual sort of displays of injuries that were in A&E from the, from the medical practitioner that was on site, was, was quite . . . Well it was hard hitting’ (EHP5). In addition, the sessions appeared to play an important role in increasing the personal enrolment of EHPs in the project, and providing justification for their involvement in intervention delivery: ‘it did, you know, sort of get me sort of more enthusiastic about trying to tackle it’ (EHP5). The role of senior EHPs in coproducing SMILE was also perceived as important, both in facilitating EHP enrolment – ‘[B]ecause our colleagues had been involved in it from the start that did help because, you know, you weren’t sort of actually preaching to people who had no sort of feedback from peers’ (EHP10) – and in ensuring that the resulting intervention tools met existing skill sets: ‘[T]he actual forms I think are pretty self-explanatory in terms of, you know, filling them in, obviously you know we are used to going on site and taking various different forms with us, you know’ (EHP17). This positive view was later reinforced by the EHPs who did not attend a training day – ‘I had a look through [the risk-audit form] myself and any questions that I had about it I could ask my colleagues and they would sort of clear it’ (EHP6) – which suggests that the SMILE audit fitted into work practices easily even without training being given.

Despite this generally encouraging response, concerns about the exclusion of LA licensing teams remained: ‘the violence and the procedures around that, with regards to CCTV and all that, would have been better suited to licensing’ (EHP18) and ‘it would have been nice to have had the feedback and the input from the licensing officer side of it’ (EHP15). A minority of EHPs were concerned that ‘we don’t want to step on their [licensing’s] toes!’ (EHP18), thus highlighting the importance of agency boundaries within work practices and relationships. 83 A minority of EHPs also felt that the health and safety questions removed the project focus: ‘there were other more general health and safety questions . . . risk assessments and those types of things, maybe they could have been more specific and kind of into more detail perhaps on the violence and aggression’ (EHP9). This response can be attributed to lack of understanding of the role that these factors can play in levels of violence and highlights the importance of providing this knowledge clearly and ensuring that it is disseminated widely during this phase of the project.

Although most EHPs were impressed by the website and films – ‘I felt that I could say to the publicans oh you know it does give you a lot of good information, you know such as challenge 25. And so I thought there was a lot of good information on the DVD’ (EHP2) – two practitioners voiced concerns about one section of one film:

[A]s a team we were slightly horrified by was the contents of the training video which, you know, indicated that premises should put staff into harm’s way in order to protect customers, which is just abhorrent, it is just everything that we would advise against.

EHP8

Consequently, the videos were not extensively promoted by these practitioners.

Overall, the training days appeared to have a beneficial effect. They remedied or limited most EHP concerns by improving levels of understanding, and promoted individual legitimation and enrolment more widely throughout environmental health. The largely positive reaction to SMILE tools and delivery methods was also a testimony to the ability of coproduction to generate interventions that are perceived as valid and relevant to users84 and to the professional attitudes of EHPs, an attribute that was further evidenced during SMILE delivery.

Intervention reach

Although the subsequent delivery of SMILE by EHPs led to significant intervention reach, it is must be noted that a number of premises closed down before the intervention could be delivered, and a small number proved uncontactable or refused to participate. Among the 300 original intervention premises selected, 92 were ineligible owing to premises closure and four refused. Within the four premises refusals, one was as a result of an ongoing prosecution, while one landlord declined because he had recently been part of a similar project and another refused because he felt that SMILE selection reflected poorly on the premises: ‘he got quite upset about it and I think [research team member] got involved as well and you know he had to send you know an apology out’ (EHP21). The final refusal was based on disbelief: ‘they were like well, well, why, why are you here. You know I can’t remember the last time there was even an incident’ (EHP22). This was a feeling shared by managers of many premises in rural or quiet locations. Consequently, EHPs delivered the intervention to 69% of the initial sample.

As per protocol, replacement premises for premises closures and refusals were allocated. In total EHPs delivered SMILE to all but four of the available intervention premises that were open and could be contacted. Such a high implementation rate is consistent with studies in other areas involving statutory partners and strengthens the evidence that agents without legal powers are inferior when compared with those that do have legal power. 69,85 Indeed, nearly all managers consulted reported feeling obligated to co-operate with authorities: ‘it’s going to happen, so may as well go with it’ (DPS 342).

Intervention fidelity and dose delivered

The collaborative approach to intervention development appeared to have produced an intervention that was acceptable and proved to be usable by most EHPs, with high levels of implementation fidelity. However, a similar level of intervention dose was delivered across premises regardless of premises context and level of violence, with EHPs relying on less intensive verbal/written feedback rather than follow-up enforcement visits.

Visits typically consisted of study explanations, an internal and external inspection of premises and necessary documents, a talk ‘that gave information about ‘health and safety at work and that kind of thing’ (DPS 347), and some discussion of the project website and DVD. Although collective EHP accounts provided strong evidence that SMILE delivery was conducted with high levels of fidelity, individual approaches to delivery varied. Some practitioners –

They were always looking at what you were writing and stuff, and it was quite easy for then for me to explain to them with the tick boxes, well you know, you just ‘well look, there is the question, this is the box that I am ticking’.

EHP11

– used more open attitudes than others:

I didn’t score in front of them, I would fill in the pro forma and then come back and do my scores back in the office, so I wouldn’t say to them at the end because obviously that would be quite confrontational.

EHP7

There was evidence that all EHPs found the risk audit easy to use and could provide the necessary advice and guidance: ‘I thought it was all very, the layout and everything was all sort of easy to use and I thought that it was fine’ (EHP1). Some used the audit exclusively – ‘I think that we just utilised your guidance’ (EHP9) – but many drew on previous experience and/or interaction with premises staff:

You work on the response of the person that you are talking to, you know how proactive they are, how responsive they are to what you are talking about. There is no point in serving notices on somebody who is very willing and very happy to put things in place.

EHP7

The EHPs in one LA doubly scored premises: ‘your rating system for this is different to our rating system on our system and so basically we had to rate them twice, we had to rate them for you and then for our own purposes’ (EHP12).

Full description and findings of the audit are reported elsewhere in the report (see Chapter 5). However to summarise, the risk audit involved completion of a RCI (Likert scale 1–6) that indicated levels of risk in 11 separate areas and recorded subsequent requests and actions. Descriptions of visits and the records produced indicated that premises tended to receive a similar dose of the intervention regardless of premises context. Overall, only 24 (8.5%) premises failed to receive some request for action (see also Chapter 5). The vast majority (91.5%) received verbal advice – ‘always verbal on site’ (EHP6) – with over half receiving additional written advice detailing changes that the premises were required to undertake. However, incidence of follow-up visits was low. Only 18 were made in total, most related to health and safety checks (n = 13) and staff training (n = 13). EHPs reported reverting to more formal procedures only when actions were not made or legislation had been compromised:

With issues then which I did pick up on it was written advice, I didn’t need to take any further enforcement action after the written advice and revisiting them, they had done all the, you know they had done all the issues which I had picked up.

EHP6

Few problems in promoting the website and DVD were reported, although the two practitioners who had held concerns about the advice in one film distributed DVDs with warnings to ignore the section in question.

Reports that SMILE delivery was easy together with high prescription fidelity leant further support to claims that use of a coproduction approach produces interventions that fit well into organisational systems. While the high levels of reach and implementation fidelity strengthen arguments that statutory authorities should deliver such interventions, this may need to be considered alongside evidence that EHPs preferred to deliver SMILE using verbal/written advice rather than enforcement and follow-up visits. The effect of this on premises’ response to intervention, therefore, needs to be considered in an examination of intervention receipt and acceptability.

Intervention receipt and acceptability

Findings suggest that the statutory nature of the intervention promoted receipt. However, there was a perception that premises were more responsive to SMILE if they were already engaged with prevention. This was most likely if they were in an urban area, were a large chain and had existing systems and structures in place. Smaller independent premises seemed to find the intervention less acceptable and appeared to require enforcement to promote action.

Half of the EHPs interviewed felt that premises reacted to the SMILE audit positively. Some ascribed this to the statutory nature of visits, others to pre-existing positive relationships with premises:

All of them were accommodating and you know quite a few of them were actually happy then . . . because . . . maybe they hadn’t had an inspection or visit off us for you know a number of years.

EHP6

Some owners supported this view: ‘if you are doing it with the local authority you know that you have covered everything and there is nothing that you have missed out yourself’ (DPS336). In general, EHPs reported that managers of large, often chain, premises tended to react more positively to SMILE – ‘some of the bigger ones . . . where they have got good procedures in place and well trained managers and a well-run place that are receptive to it’ (EHP3) – with poorer responses often coming from long-time owners of smaller pubs:

They said ‘have you ever worked in a pub’ and I said ‘no’, ‘well’ they said ‘well there we are, I have worked in a pub for 30 years, I know exactly what goes on here, I know exactly what to look for’.

EHP3

These EHP descriptions of visits were supported by 16 intervention premises owners/managers (chain, n = 7; independent, n = 9). In reaction to SMILE, managers of large chain premises tended to feel that issues were already being addressed by established in-house policies, practices, training and support: ‘all I have got to do is phone head office, and they are like there with an answer’ (DPS347). However, this opinion was not universal: four managers of chain premises, including two where SMILE identified no concern, felt that the intervention had benefited them by raising personal awareness of ARV and refocusing attention on the issue – ‘it pushes me to the right direction that you have got to be focused on these types of things you know’ (DPS84). In addition, some proprietors noted how having SMILE delivered by EHPs had encouraged maintenance of present standards and updated knowledge:

[F]resh eyes, do you know what I am saying, so I mean and anything new or anything I mean it is like if I have been here 6 years and . . . maybe, she will come and say oh why don’t you try it that way because it is a fresh idea you know. Sometimes so anything new really is always a good idea I think.

DPS124

Within independent premises, despite one reacting very positively –

[T]here are new things coming along . . . and I don’t have the information and there are lots of different things she brought with her, booklets and that as well but especially that DVD, it has just highlighted different zones . . . and say ok I am taking time out here now and I am doing this and showing the staff this as well, and it is all good isn’t it really. It is positive.

DPS131

– a significant number were unhappy at the idea of receiving SMILE. A couple of owners insisted that the intervention had no place in their premises and saw it as an unwelcome burden: ‘I suppose if you hadn’t ticked a lot of the boxes then yes it would be an eye opener and it would be of value. But personally speaking I had a lot of the work already done’ (DPS179). Others participated without complaint, but gave SMILE little thought afterwards, and did not remember it positively: ‘once is bad enough!’ (DPS100).

Participants managing premises near or in towns or city centres appeared more responsive to the aims of the intervention. All agreed that violence was an established ongoing problem, with the majority feeling that the situation had worsened in recent years as escalating use of stimulant drugs had compounded matters: ‘[i]t has got a lot worse over the last few years because there is so much, there is so much drugs in the [location redacted] now it’s sort of an accumulation of the two’ (DPS174). Despite this, a minority of managers working in urban environments perceived violence as something that occurred outside their premises or their area of responsibility –

[P]eople walking down from X Street two of them seen each other . . . history of a feud between them and they started fighting outside. And it has ended up our doormen have got involved because it is literally our doorstep and other people have got, and it just escalated and escalated and it was my doorman that got into trouble for it – for stepping off their door.

DPS33

– or as a consequence of drinking elsewhere either at other premises or through preloading. This inference that premises prefer to distance themselves from customer violence was reinforced by comments such as ‘[t]he biggest issue is once people have become intoxicated is having the foresight, as managers and door staff, to remove those people from the venue’ (DPS28). Such a comment supports evidence that violence and aggression are often displaced onto the street,86 which strengthens a call for some premises to be given stronger support to help them take responsibility for their role in ARV.

Intervention reconfiguration and sustainability

Most EHPs felt that SMILE had been an effective intervention that had had a beneficial impact on the knowledge and practices of EHPs and fitted well with routine work practices. However, initial unease and a lack of confidence in dealing with violence as part of their routine practice remained. Some suggestions to reconfigure SMILE to improve implementation targeting, efficacy and sustainability were also offered. Of particular importance was the involvement of other agencies to promote enforcement and premises receipt.

Using SMILE had raised awareness of ARV and the factors affecting it for many EHPs – ‘it has got to be done, as in like I didn’t realise the scale of the issue’ (EHP21) – and made the potential to address ARV within premises more explicit: ‘I would much prefer to go out there and give them all the information and the guidance that they need you know to help them you know make the premises a safer place rather than going in as an enforcement’ (EHP19). There was wide agreement among EHPs that SMILE fitted the organisational context well: ‘seems to be certainly something that would tie in naturally with the Health and Safety at Work Act and it is sometimes it is a very real, erm, safety issue to employees and to members of the public’ (EHP20). The importance of sustaining and integrating this work into professional practice rather than as a ‘one-off’ intervention was also stressed: ‘I know from like, from personal experience and they have got good intentions for that, the next few months and then it sort of slips off and it goes off their agenda and then something else’ (EHP13).

In contrast, most owners and managers were surprised to find EHPs delivering SMILE. Regardless of this, most felt that EHPs had a role in assessing risk factors given their knowledge and statutory powers: ‘you would make double sure that, you know on that night, you know, it is one of those things that you shouldn’t work like this, you do you just say I will go and double check’ (DPS308). EHPs agreed that statutory powers encouraged actions within premises:

[T]hese interventions highlight the issue to the managing agent and as a result, as is generally the case when we book an assessment, the first thing those in control of the business does is review their own risk assessments on that subject.

EHP8

Further reflection produced some suggestions for SMILE reconfiguration. The major concern stemmed from the data used to identify participant premises. Although virtually all EHPs felt that the premises visited had been representative of their areas, over half felt that the wrong premises had been selected: ‘some of the ones perhaps that I would have classed as being a problem pubs weren’t within that list. . .’ (EHP5). Many respondents agreed, feeling that SMILE effectiveness would have been increased if delivery had been confined to premises whose managers had little experience or knowledge of ARV and how to minimise it – ‘the ones that weren’t part of a chain, independent, no controls, ones with no, no real understanding’ (EHP22) – or which were known to have larger levels of problematic behaviours and violence. Many EHPs suspected that the police violence records were unreliable: ‘you said that a lot of the data came through from the police but when I checked up, certainly on the one pub they had nothing, there was nothing against it whatsoever’ (EHP7). Police violence records were also criticised for associating incidents with nearby premises – ‘three of the pubs in particular, they have called the police for problems outside the premises, you know it is nothing to do with their premises but yet they are, you know they are brought up on the list’ (EHP3) – and for not differentiating between police attendance to prevent trouble and managing ongoing violence: ‘people believed they were serving under age but they were very pro-active in wanting to kind of clean that up . . . and I think that is why they had a higher number of police incidents’ (EHP18).

Post implementation, the feeling that SMILE should have included licensing officers had strengthened – ‘a lot of things that sort of licensees have to do which ties in with this and [licensing] have got a lot of hands on knowledge of individual premises, individual licensees’ (EHP4) – even among some proprietors ‘if a policeman had come in and done that study with me I would have found it more appropriate than the EHP woman coming in to do it’ (DPS22). However, a few managers were concerned that SMILE drew more agencies into a field already negatively affected by decreased funding for police and local authorities in licensing over the last decade: ‘I think that perhaps bringing more bodies into the kind of . . . maybe . . . Yeah I think that, as I say at the moment there is way too much conditions on licences and stuff like that’ (DPS157). SMILE also introduced or reinforced appreciation of the value of multiagency work, especially for EHPs with similar earlier experience: ‘people like the health board, the police, you know fire service, other agencies . . . you know getting everybody interested’ (EHP17). EHPs also maintained that:

it is important to forge these links with everybody . . . so that they can actually, you know raise concerns and perhaps you know change licence conditions and deal with all these things rather than you know do it in isolation.

EHP10

Cost-effectiveness of the intervention

Table 22 shows the mean (SD) costs of training, audit, crimes, total costs and total cost differences, with p-values and 95% CIs. The results show that intervention premises incurred higher costs. The mean difference in total costs between intervention and control premises is £28,831.24, which is not statistically significant (p = 0.17, 95% CI –£12,748.34 to £70,602.91). When adjusted for opening hours and baseline incidents group (binary), the adjusted mean difference is decreased to £24,137.33 and remains not statistically insignificant (p = 0.22, 95% CI –£14,069.34 to £62,151.38).

As the intervention group is more costly and less effective, it is dominated by the control.

Exploratory cost-effectiveness of intervention with follow-up audit

Results in Table 23 show that intervention premises that had follow-up audits incurred significantly lower costs than those not audited (controls) or audited once only. The mean difference in total costs between these two groups is –£64,572, which is statistically significant (p = 0.005, 95% CI –£104,888 to –£18,752). When adjusted as above, the mean difference in total cost is reduced to –£47,795.50, but this difference remains (marginally) statistically significant (p = 0.051, 95% CI –£105,200 to –£1167).

This suggests that an intervention that includes a follow-up audit is more effective and less costly than no intervention or a single audit. This intervention leads to a 42% reduction in violent sessions in the group, which is equivalent to a reduction by seven sessions with violent incident. An intervention that includes a follow-up audit, therefore, could produce savings to society of £47,796, or £6828 per violent session averted.

Stochastic uncertainty in the above results (joint uncertainty in mean failures and costs) are presented via a cost-effectiveness plane that is based on 10,000 bootstrapped replications (Figure 17). The bootstrapped samples consider the mean number of failure sessions and mean total costs. Note that in conventional economic analysis a positive difference in effect means an intervention is more effective than control; here, a negative difference in effects means that the intervention is more effective. Most of the bootstrapped replications are in the south-east quadrant, which means that the intervention (initial audit with follow-up) is more effective and less costly and hence unambiguously more cost-effective.

FIGURE 17.

Cost-effectiveness plane showing the joint uncertainty in mean differences in costs and failures: intervention (including follow-up audit) vs. all other premises.

The CEAC (Figure 18) shows an intervention that includes a follow-up audit that has a 99% probability of being cost-effective over any value the society is willing to pay to avert a violent session.

FIGURE 18.

Cost-effectiveness acceptability curve showing the probability that the intervention (follow-up audits) is cost-effective at different willingness-to-pay values.

One-way sensitivity analysis

As not all violent incidents lead to a criminal justice system cost, we have considered a 50% reduction in this cost in sensitivity analyses. The new estimate of the total cost incurred from a violent incident is £22,846 or (£10,811 + £12,035). A sensitivity analysis is also carried out for training costs amortised over 5 years at a rate of 3.5%. Results are given in Table 24, showing that the intervention (follow-up audits) still remains as a cost saving. However, considering a lower estimate of criminal justice system costs indicates that the adjusted mean difference in total costs is now marginally not statistically significant (p = 0.054, 95% CI –71,687.95 to 2.69).

Intervention development

To date, EHPs have not been routinely involved in addressing ARV in premises licensed for the on-site sale and consumption of alcohol, despite having a violence in the workplace remit. LA licensing teams take responsibility in this area and at the expense of EHP involvement, even to the extent that EHPs would rarely visit premises for health and safety reasons. This is of concern. While Licensing are able to enforce the Licensing Act, they have neither health and safety remit nor expertise. Important issues such as fire safety and health and safety training in the workforce and other areas have, therefore, been neglected. This is evidenced in the audit data collected here, where it was found that some premises were operating in ways that placed their customers at risk. Recognising that a number of EHPs felt that licensing should be involved in such intervention work suggests that, at a minimum, joint visits with licensing should be undertaken so that both health and safety and licensing legislation are brought to bear. Alternatively, practitioners skilled in both areas could be trained.

While EHPs have considerable expertise in the area of intervening with small businesses, material and knowledge of what works in respect of reducing the risk of ARV in premises was lacking. An aim of the current study was therefore to develop materials that developed the existing academic knowledge base but also remained within EHPs’ statutory remit. This remit was fairly broad, as under Health and Safety at Work legislation there is an expectation that businesses conduct risk assessments and ensure that their staff are appropriately trained on the processes required to mitigate risk. Staff training is a theme in many interventions in licensed premises and is typically focused on responsible service and similar areas of operation. However, the EHP approach to staff training is that businesses should first understand what risks they are subject to, what the solutions are, who is responsible for mitigating these risks and what can be done. This highlights the need for premises staff to collectively work together to mitigate risk and, thus, generic training programmes are likely to cover only a portion of what is required, and idiosyncrasies will be missed, thereby highlighting opportunities for EHPs to help premises identify and manage risk.

In order to develop materials for EHPs’ use in licensed premises, a series of multidisciplinary development meetings were scheduled. These enabled the development and refinement of a programme model of SMILE, and developed programme models and materials to support EHPs. The final intervention package included a risk audit, a website and a quiz. The risk audit was the primary intervention vehicle. It contained clear instructions on the legislative basis for EHPs to intervene to target specific risk factors in premises, allowed for the quantification of those risks, provided instruction on what changes might reduce those risks and included materials that facilitated a follow-up audit. Instructions for the follow-up audit departed from the original feasibility trial. 21 Originally, all intervention premises in the feasibility study received a follow-up visit, whereas in the current study this requirement was relaxed to fit with EHPs usual working practice in which follow-up visits are scheduled only if a business has more serious issues. For EHPs, dose is commensurate with EHPs’ perceived risk. This reflects recent government guidance62 that aims to reduce the administrative load on small and medium-sized businesses. 88

The website and associated films (available online and on DVD) aimed to provide to premises staff with a positive reason for engaging in harm reduction (e.g. increased footfall), provided template forms consistent with those that EHPs would recommend that small businesses use and further inform premises staff (from senior managers to servers and door security personnel) on how a premises as a whole can work to mitigate harm.

Throughout the development process EHPs engaged with the project fully and played a central role in the coproduction of these materials. The end result was an intervention, SMILE, which proved acceptable to EHPs and was delivered with a high degree of engagement and fidelity.

The intervention

The SMILE intervention was evaluated in a RCT where police-recorded violence was the primary outcome measure. Police violence data were followed for 455 days following randomisation. The reason for following premises for this extended period was to assess any effect of the intervention over time. For example, if a reduction in violence was observed, would this effect be sustained over time or would it wane? Licensed premises are characterised by a high staff turnover and it was hypothesised that this may undermine any initial intervention effect.

Originally, the target sample was 300 premises recruited into each control and intervention group, with sample size estimates suggesting that a minimum of 274 in the intervention group would be required. Historical police data were accessed to identify premises that had experienced one or more violent incidents and these premises constituted the population from which premises were randomly allocated to each experimental group. Twelve months of police data were accessed in order to identify premises, together with a 6-month period in which data were analysed and material prepared. This meant that premises were identified in police data through incidents that occurred up to 18 months previously. Accordingly, a large number of premises were no longer amenable to receive the intervention. Reasons for this included closure and change of purpose (e.g. from a traditional pub to a restaurant). In order to confirm both control and intervention premises were operating further, attempts were made to access LA public licensing registers, although information in these proved to be, at best, sparse and not fit for purpose. Thus, business rate information was also accessed (as small business premises are expected to pay business rates if they are operating). This allowed us to note when premises closed, if they did so, during the follow-up period. In addition, and at the point of implementation, premises were dropped from the study for additional reasons. These included duplicates (e.g. some nightclubs may operate a bar in the same venue that differs in name from the nightclub itself), ongoing prosecution by either environmental health or a related agency, and occasions where premises claimed there had been no occasions of violence on-site (e.g. one premises name was identical to the village in which it was situated and thus it was not possible to disambiguate the exact location of the incident in police data). An advantage of working with EHPs was their capacity to access premises; all small businesses must provide access within 24 hours of receiving notice that they are to be visited. Despite this, three premises refused to participate. In these cases it was the local EHP who asked that they did not pursue the premises in question, mostly because of a preference to avoid conflict in what was effectively a project. These refusals were, therefore, anomalous.

The high level of attrition, mostly a result of closure, from the study meant that ITT analyses were underpowered, according to initial sample size estimates. Accordingly, and to mitigate this feature, sensitivity analyses were planned where the remaining population of premises were added into analyses. Effectively, fewer than expected premises in the analyses means that there is more uncertainty and these sensitivity analyses offer one method to assess the impact of increased uncertainly in analyses. No matter which way we looked at the data the results were consistent. The intervention was associated with an increase in police-recorded violence.

Over 98% of premises eligible to receive the intervention and that were open at the time received a visit from an EHP, suggesting a high level of engagement. Furthermore, analysis of data derived from the initial audit indicates that EHPs did find evidence of risk in premises. These risks varied considerably in form and severity and, therefore, the form of advice given to premises by EHPs also varied. It is usual EHP practice to titrate their response to findings in a manner that is commensurate with the risks they observe. Thus, for fairly mild incursions premises would receive verbal advice on how they might address issues and for much more serious issues EHPs’ response went as far as formal notices that placed a legal duty on premises to enact the required changes. If premises failed to make changes under a formal notice then they became liable to criminal proceedings. A small number of premises exhibited a range of problems to such a serious level that EHPs were obliged to issue formal notices. The review process, under the Licensing Act (2003, Section 53 A), takes 28 days for initial consultation followed by a 20-day period in which a hearing will be held. It is led by a senior police officer and is mostly focused on serious crime; there are opportunities for an expedited review, but these are rare. This process does not cover broader issues concerned with health and safety.

Although instruction was given on the criteria for conducting follow-up visits to premises in the audit materials, and at least one-third of premises met these criteria according to audit data, far fewer than expected premises received a follow-up audit. This may be a result of reluctance on the part of EHPs.

Both primary and secondary sensitivity analyses yielded a consistent intervention effect whereby police-recorded violence increased following the intervention and showed no change in this effect over time (it did not wane). Although the trial was underpowered, secondary exploratory analyses and comparisons with the original feasibility study21 provide insights into the possible reasons for this effect.

In the original feasibility study, all premises received an initial and follow-up audit. Further, audits were conducted by private contractors who, while experts in health and safety in the workplace, were not affiliated with statutory bodies. This meant that engaging premises in the feasibility study proved difficult, with few interested in any voluntary harm-reduction initiative; this was effectively overcome through collaboration with EHPs who delivered the intervention to nearly 100% of eligible premises. However, the process evaluation found that in many instances EHPs sought the advice of partners in LA licensing teams. It is therefore possible that this caused premises in the intervention group to receive greater attention across partners than control premises. As the police are able to record only the violence they are aware89 of, it is feasible that this increased attention meant that more violence was recorded in intervention premises than in control premises, an effect that has been observed in similar studies using police data. 17

A more pertinent issue is whether or not premises in the intervention group made any changes in consequence of the intervention and advice received from EHPs. The only group in which reliable data are available to establish whether or not premises made the recommended changes is those premises that received a follow-up visit from EHPs. In this case, analysis of the audit data strongly suggests that those premises receiving a second follow-up visit had, in fact, made the changes recommended by EHPs. While the process evaluation had covered this matter to some extent, how reliable these data are is open to question. Premises staff, in particular those in premises that are exhibiting greater risk of alcohol-related harm, are unlikely to be reliable, particularly if changes are viewed as being voluntary, which, without a follow-up visit, might have been the case. Further secondary analyses, in particular comparing premises in which written or verbal advice had been given with premises in which no advice had been given, appear to support this. Those that received advice showed lower levels of violence over the follow-up period. We therefore suggest, accepting issues concerned with the fewer than expected premises, that dose would appear to be a key component in the effect of the intervention. As EHPs provided greater motivation for premises to make changes, levels of violence appear to have fallen.

Process evaluation

Study findings suggest that environmental health is an agency that possesses the required infrastructure, expertise and skills to become more involved in addressing the incidence of ARV in licensed premises. Their existing professional practice and audit tools provide a context for the development of appropriate interventions that draw on their professional skills and statutory powers.

The provision of a developmental phase and coproduction approach in intervention development ensured high levels of intervention approval and implementation fidelity as the intervention mapped well on to existing working practices. Utilising a TDAR approach in future intervention development phases appears to be a promising approach. However, adequately addressing professional skills and understanding how to achieve sustainable changes in professional practice is also important. There was some professional reluctance and a lack of confidence in dealing with ARV as a new area of work among some EHPs at the start of the project. This appeared to have been addressed by the training days and diffusion of the intervention within the organisation. However, these concerns remained at the end of the study and might have contributed to the intervention implementation failure of a key mechanism of action for intervention effectiveness, the use of statutory enforcement.

Statutory powers appeared to be key in facilitating the high level of reach of the intervention and were identified as an important influence on promoting intervention acceptability and receipt in premises. However, EHPs rarely drew on these powers. Enforcement follow-up visits were hardly ever used, regardless of premises context or levels of violence associated with premises. Examining the potential mechanisms of action, premises’ responses suggested that these visits could be key in promoting responsibility and addressing resistance at the premises level. Larger premises engaged in prevention may require more follow-up visits and the involvement of licensing officers to counteract assumptions that the issue is already being dealt with adequately and smaller premises appear to need more support and follow-ups to adequately engage in the issue.

Engaging with and supporting a professional group in a new area of work may require a longer period of intervention diffusion and normalisation in order to promote full implementation. Confidence to draw on statutory powers may come as familiarity increases and responsibilities are assumed within routine practice rather than within the context of a research study. Significantly, suggestions that the intervention should utilise a multiagency approach were identified throughout the project. Collaboration with the police in intervention delivery may promote enforcement, or its threat, and promote a crucial mechanism of action for effectiveness. There are also some issues to consider in the targeting of the intervention and its assessment of effectiveness, with a need to examine alternative sources of data to identify at-risk premises and identify change over time.

Cost-effectiveness

The total cost of the SMILE intervention (training 70 EHPs and auditing 281 premises with further follow-up audit to 18 premises with higher risks of criminal incidents) was £35,196, or £125 per premises. The intervention was shown to be less effective and more costly and hence not cost-effective (dominated by controls).

Although there is considerable uncertainty on the observed significant effect of the follow-up visit on police-recorded violence, those premises receiving the follow-up audit also showed a statistically significant reduction in violent sessions and costs, suggesting that the second audit is possibly cost-effective (dominant). A sensitivity analysis capturing joint uncertainty in costs and effects indicates that the probability of a follow-up audit being cost-effective is almost 100%.

Limitations

The trial was underpowered owing to an unexpected number of premises closing between identification of premises in police data and delivery of the intervention. Premises were eligible if they had registered police-recorded violence in the 12 months leading up to the start of the project. It was expected that a number of premises would close and that, therefore, trial arms were oversampled to account for this. However, a larger number of premises were identified in police data that were open for business at the time of audit. One reason for this was the inadequacy of LA licensing data, the initial source from which premises were assessed for their economic activity. It would appear that many LAs do not hold accurate information on the premises in their catchment. Another source of this variation was duplication. Some premises were identified in the police data with more than one address. Large premises in particular would have more than one entrance or exit on more than one street. This is the nature of the data to hand in this trial, where event data were reduced to identify at-risk premises. Nevertheless, future work should consider how police information on violent crime is linked with LA licensing data and how information on premises can be improved. There is a potentially rich source of information that might inform practice if it were adequate. As a minimum, future work needs to consider the availability and use of A&E data.

A similar issue concerns measures of violence. This trial relied on police-recorded violence, a source with known recording biases. These biases are, potentially, one reason for the observed increase in violence for the intervention group, that intervention premises received greater attention. The alternative, health data, in particular A&E data on assaults, do not have the same level of coverage as police data, but are less susceptible to recording biases.

Generalisability

While this trial focused on premises licensed in Wales, it is likely that the approach adopted here is generalisable to the rest of the UK and internationally. Our approach in developing intervention materials was theoretically robust, focusing on risks and adopting theoretical models identified in the international research literature. Fundamentally, however, and as identified above, the sole reliance on police-recorded violence as an outcome measure reduces the external validity of the results presented here.

The expectation was that, should a significant effect be observed, the 12-month follow-up period would allow us to test for sustainability: whether the effect wore off because of staff turnover in premises or for other reasons. Under the circumstances this was not feasible. Nevertheless, the model presented here, one where EHPs respond to violence in licensed premises, would provide one that could be implemented in the long term. Premises that fail to heed advice would be expected to show heightened levels of violence, thereby warranting further visits from EHPs.

Finally, RCTs are generally seen as a suitable and fairly robust method for capturing meaningful evidence to inform service development. However, this approach is underused and, therefore, opportunities exist for further development. Such areas include addressing the needs of practitioners, upskilling the practitioner workforce and more fully involving robust evaluations into policy development.

Implications for practice