The clinical and cost effectiveness of diversion and aftercare programmes for offenders using Class A drugs: a systematic review and economic evaluation

Advisory panel

The project team included clinicians, commissioners/service providers and experts in the field. The full project team had input into the methods and design of the reviews. The role of this panel was to inform the choice of population, interventions and outcomes to define the initial search strategy and inclusion criteria and to help identify the relevant databases to be searched. The group was also asked to comment on the draft report.

Search strategy

The design of the systematic effectiveness review followed guidelines set out in the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement. 77 Sources included both medical and social science databases chosen in consultation with the project team and with systematic review experts. Sources were chosen to provide a balance between the health, social science and criminal justice literature and to include material derived from both mainstream and ‘grey’ literature sources.

The following databases provided a comprehensive search of the literature: MEDLINE; PsycINFO; Web of Science; Wiley Online Library; JSTOR; EMBASE; Ingenta; Cumulative Index to Nursing and Allied Health Literature; Criminal Justice Abstracts; Wilson Social Science Abstracts; Social Sciences Index; Campbell Collaboration Social, Psychological, Education, and Criminological Trials Registry; Informa Healthcare; Sage; Science Direct; HighWire; ProQuest; Applied Social Sciences Index and Abstracts; British Humanities Index; National Criminal Justice Reference Service; Social Services Abstracts, International Bibliography of the Social Sciences; Sociological Abstracts; ProQuest Dissertations and Theses; SciVerse; Metapress; Scopus; Taylor and Francis Online; System for Information on Grey Literature in Europe; Centre for Reviews and Dissemination Database; Allied and Complementary Medicine and TRIP (formerly known as Turning Research into Practice).

The project team developed a robust search string. The search strategy balanced sensitivity (to ensure that relevant material was identified) and specificity (to ensure that a reasonable proportion of the material was relevant). Medical subject heading terms and text words for inclusion in the search strategy were developed from terms relating to elements of the review question, namely: offenders, class A drugs, and diversion and aftercare programmes. Search strategy development was informed by the successive fractions approach. 78 This allows for objective testing of the productivity of main terms, restriction terms, and terms taken in combination, to optimise the balance between sensitivity and specificity.

Initial trials of the search strategy were carried out in MEDLINE and subsequently refined. Testing by proxy was used to select terms. A term was deemed sufficiently productive if at least 10% of citations were relevant to the review. Additional relevant search terms were identified by examining key papers and consulting experts. All new search terms were tested for retrieval of novel citations, i.e. those not previously identified by existing search terms. Sensitivity and specificity were tested by applying the search string to a ‘dummy database’, comprising known relevant and irrelevant citations. Subsequently, sensitivity was increased by the addition of further search terms. Specificity was improved by the addition of the Boolean term, ‘NOT’ which excluded irrelevant material. Final evaluation of the search string included testing against a control list of papers referenced by previous reviews and a cardinal list of key papers in the field. The final search strategy is listed in Appendix 3.

Databases were searched between 2 November 2011 and 6 January 2012. Electronic database searches were limited to papers published in English between January 1985 and January 2012. It was decided not to update the search prior to publication of this report. This was based on the paucity of evidence identified for the review from the original search. Given changes in the organisation and provision of the NHS and social care that also affect services for addiction it was felt that swift publication was of greater importance. Libraries of retrieved studies were exported into EndNote citation manager software (EndNote X6, Thomson Reuters, CA, USA) and deduplicated. The reference lists of all full-text articles retrieved, including grey literature, were hand-searched for additional material.

Inclusion criteria

Quality assessment

There are a large number of available instruments for the objective evaluation of study quality. However, few of these have been adequately validated79 and the vast majority are directly applicable only to randomised controlled trials (RCTs). RCTs remain a rarity in the field of criminal justice research and none of the studies meeting our inclusion criteria followed this methodology, although one study80 derived its data from a previously conducted RCT.

Objective quality assessment scales that directly address the type of ‘real-world’ approach, which is the natural methodology of drug diversion research, are virtually non-existent. One exception to this general rule is the Maryland Scale of Scientific Methods81 summarised in Appendix 6. This scale was developed specifically to address the constraints of research in the criminal justice setting and is currently the most widely utilised quality evaluation scale in this context. The scale considers both broad aspects of research design (e.g. controls and randomisation) but, more importantly in this context, considers also in greater detail threats to internal validity.

Data extraction

Data extraction (see Appendix 5) was carried out independently by at least two reviewers. Data pertaining to study design, intervention, sample demographic characteristics, relevant outcomes and any relevant statistical analyses were extracted and inputted into Statistical Product and Service Solutions (SPSS), version 20 (IBM Corporation, Armonk, NY, USA; 2011). Any disagreements were resolved by consensus and a third reviewer, if necessary.

Data synthesis

An initial narrative synthesis of the available material was used to explore and outline the extent, nature and quality of the available evidence in this area. This qualitative assessment of the available data also explored observed heterogeneity in study and participant characteristics, study designs and study outcomes. It was used to inform the structure of subsequent quantitative synthesis of the data, including the choice of comparisons to be made and the outcome measures amenable to quantitative treatments. A bivariate analysis of outcomes explored the potential associations between study and participant characteristics and outcomes identified via narrative review. Outcomes suited to meta-analysis were then converted to odds ratios (ORs) and data from individual studies combined to provide a quantitative evaluation of heterogeneity. Where studies evaluating a particular outcome (e.g. reduction in primary drug use) were identified as both statistically and conceptually amenable to combined analysis, meta-analytic models were developed to identify a pooled effect size. Meta-analysis was also used to explore publication bias.

Study flow

Database and bibliography searches identified 28,408 potentially relevant studies. Screening of titles and/or abstracts led to the exclusion of 27,110 studies, with 1328 articles proceeding to an examination of the full text. Full-text review led to the exclusion of a further 1284 articles and 14 papers, relating to 16 studies, were included in the quantitative synthesis. This process is summarised in Figure 1.

FIGURE 1.

Flow chart of included studies.

Selection of included studies

Data analysis for 15 of the 16 studies was based on secondary data drawn from available published reports; complete raw data were available for one study,82 with partial raw data available for one study in which a part of the sample met our inclusion criteria. 83 Reasons for exclusion of the studies not included in the review are listed in Figure 1 and a comparison between included and excluded studies is provided in Appendix 6.

Profile of included studies

Quality assessment

The evaluation of study quality relates to each study taken as a whole. For a number of studies (see Appendix 6) only a subgroup of participants met our inclusion criteria and subsequent analyses of outcomes are therefore based only on outcomes for these individuals. However, to evaluate the extent to which outcomes can be relied on, it is the overall study design which provides the best indicator of research ‘quality’.

Table 2 sets out the design profiles of included studies, indicating study design, sample size, allocation of participants, length of follow-up and attrition and details in line with the Maryland Scale of Scientific Methods criteria. Table 2 highlights the fact that included studies were not of a rigorously high methodological standard. Designs were, in the main, retrospective and/or correlational; sample sizes tended to be modest and there was no, or limited, follow-up beyond the intervention end point. None of the studies used random selection or random allocation of participants or was able to blind raters, where more than one group was available for analysis.

Appendix 6 sets out additional characteristics that affect the plausibility and interpretability of studies. The likely reliability and validity of outcome measures was rarely addressed. The use of intention-to-treat (ITT) analyses was restricted to retrospectively collected data. Baseline differences between comparator groups were not always controlled for. More recent studies tended to be of a higher quality when the overall evaluation of study quality was taken into consideration (rho = 0.74; p < 0.001) but not when the more restricted set of Maryland criteria only were used to judge this (rho = 0.34; p = 0.20).

Studies were grouped into three broad categories based on quality. This is a relative measure of quality between the included studies only. As stated, all included studies were not of a high methodological standard.

Participant profile

Appendix 6 provides details of the participant profiles, which are summarised below.

Details of offending behaviour

Only one study91 included a comparator group composed of non-offenders. Four studies52,90,92,93 provided specific details of the offences committed by participants and none provided details of offences committed by participants in comparator samples. Only seven studies (43.7%)52,82–84,86,87,90 provided information on the extent of CJS involvement (e.g. prior arrests, incarceration). Some additional details can be gleaned from information presented for subsamples of participants or by reference to the offender populations included. Table 8 sets out the details available.

Bivariate and meta-analyses

Not all outcomes presented by study authors are available in a format suited to further analysis. Table 9 sets out the outcome measures reported by each study which are relevant to the review parameters. These are contrasted with the outcomes which were available for bivariate and/or meta-analysis. Although no study has been entirely excluded from our statistical analyses, one or more outcomes had to be discarded for eight studies. 83,86,87,88^(Fife),88^(Glasgow),88^(combined),91,93 Ten studies had a total of nine separate outcome measures for evaluation using bivariate and meta-analysis:

• treatment completion52,80,82,84,85–87,89–91

• self-reported drug use80,82,84

• drug screening80

• drug arrests90,92

• arrests for violent offending90,92

• arrests for general offending;82,90,92 incarceration82

• scale-based outcomes (ASI scores)80,82

• other outcomes. 82,84

Outcomes presented in papers but not available for our analyses were excluded either because relevant statistical data were not presented by the authors,83,88^(Fife),88^(Glasgow),88^(combined),86,87,91,93 or, less commonly, because the data presented did not differentiate between participant groups meeting our inclusion criteria and those not doing so. 83,87

Publication bias

The funnel plot in Figure 2 is a scatterplot of effect size against standard error (expected to closely associate with sample size). A distribution which is substantially uneven is generally seen as an indication that there is a publication bias, usually in favour of studies reporting statistically significant outcomes (which tend to be studies with larger sample size). 94 To avoid duplication, only one outcome measure from each study was used in producing the funnel plot, the choice of outcome measure was made on the basis that the measure represented the main focus of the study.

FIGURE 2.

Funnel plot to assess publication bias.

While the number of studies is small and there are clear outliers in the data, the uneven distribution shown in Figure 2 is broadly indicative of publication bias. This should be borne in mind in considering the outcome of the studies reported below and throughout the text.

Meta-analyses

All included studies provided one or more metrics suited to conversion to ORs, using equations provided by Lipsey and Wilson. 95 For each meta-analysis, we present combined ORs for both fixed- and random-effects models, rather than making a priori assumptions regarding the data. We also present two measures of heterogeneity. The Q-statistic is a reliable and commonly used measure of heterogeneity, but it does not provide an estimate of the likely extent of heterogeneity. The I²-statistic provides this information. To arrive at the final models set out below, we carried out a number of exploratory analyses, each of the models given here represent the most robust model we were able to identify for a given outcome measure.

Model 1: continued primary drug use

Six studies provided data for this key outcome. 80,82,86,88^(Fife),88^(Glasgow),88^(combined) An initial exploratory meta-analysis found no significant heterogeneity, underlining the similarity in focus of included studies. We subsequently excluded data for two of the studies. Brecht and Urada86 was excluded since, in contrast with other studies it provided only a comparison between methamphetamine users and other drug users, rather than evaluating the impact of intervention per se. Eley et al. 88^(combined) was excluded because of the exceptionally large CIs associated with this study’s very small sample size (n = 10). All four of the above models were equally valid from a statistical perspective, with no significant degree of heterogeneity identified. The direction of effect across the four models was consistently in favour of a slight reduction in primary drug use as a consequence of treatment.

Table 10 sets out model characteristics for the most substantive of the four models, which is also the model focusing on the most similar interventions (varying treatment approaches, but all effected via drug court diversion). Table 10 also provides details for this model in respect of heterogeneity. The forest plot given in Figure 3 provides a visual summary of the direction of effect for each outcome measure separately and for the random-effects model used to combine outcome.

TABLE 10 - Meta-analysis: continued primary drug use

df, degrees of freedom.

Model	Study	Comparison			Outcome		Statistics for each study			z-value	p-value
							OR	95% CI
	Chun et al., 200782	SACPA vs. non-SACPA			Heroin in last 30 days (change from assessments 1 to 2)		2.326	0.972 to 5.566		1.896	0.058
	Eley et al., 200288(Glasgow)	DTTO participants			Positive tests for opiates at first test vs. at fifth test		1.795	0.532 to 6.054		0.943	0.346
	Eley et al., 200288(Fife)	DTTO participants			Positive tests for opiates at first test vs. at fifth test		1.542	0.300 to 7.919		0.519	0.604
	Marinelli-Casey et al., 200880	Methamphetamine (MTP + drug court) vs. methamphetamine (MTP only)			Reduction in methamphetamine use – 6 months after treatment		1.492	0.881 to 2.527		1.488	0.137
Fixed							1.684	1.118 to 2.535		2.493	0.013
Random							1.684	1.118 to 2.535		2.493	0.013
	Effect size (95% CI)	Test of null hypothesis (two-tail)		Heterogeneity				τ2
		z-value	p-value	Q-value	df (Q)	p-value	I 2	τ2	Standard error	Variance	τ
Fixed	1.684 (1.118 to 2.535)	2.493	0.013	0.751	3.000	0.861	0.000	0.000	0.187	0.035	0.000
Random	1.684 (1.118 to 2.535)	2.493	0.013

FIGURE 3.

Continued primary drug use: forest plot.

Model 2: continued use of other drugs

Three studies provided outcome measures suited to inclusion in a meta-analysis of the impact of treatment on reductions in use of drugs other than the participant’s primary drug of choice. One relevant outcome measure only was available for Marinelli-Casey et al. ,80 three for Newton-Taylor et al. 84 and a rather broader range for Chun et al. 82 (14 in total). Substantial heterogeneity was identified for this exploratory model (Q = 51.6, p = 0.0001, I² = 67.1). To tailor outcomes more closely to the main focus of the review, we first removed comparisons which identified reductions in alcohol use only (four outcomes from Chun et al. 82). We then explored a number of alternative models incorporating in separate meta-analyses the remaining ten contrasts taken from Chun et al. 82 (outcomes for SACPA vs. non-SACPA and therapeutic community vs. therapeutic community plus methadone were evaluated for each of the following drugs: sedatives, cocaine, amphetamines, cannabis and drug use measured via ASI drug composite scores). Since the three outcomes taken from Newton-Taylor et al. 84 were paired comparisons for the three included participants against one another, we selected from these only the most salient comparison, as identified in the study details outlined previously (graduated vs. expelled–non-engaged).

Three of the models evaluated showed significant heterogeneity between the included study outcome measures. All three were models in which outcomes for Chun et al. 82 ran in a direction counter to the anticipated impact of treatment. Two outcome measures were involved here. The first, sedative use at 30 days (Q = 8.11, p < 0.02, I² = 75.3) had increased slightly from baseline for both treated (therapeutic community plus methadone) and untreated (therapeutic community-only) groups, but more so for the latter. The second was cannabis use at 30 days, while cannabis use had decreased for all comparator groups, the decrease was less for both experimental groups compared with their comparators [SACPA vs. non-SACPA (Q = 7.33, p < 0.03, I² = 72.3) and therapeutic community plus methadone vs. therapeutic community alone (Q = 10.45, p < 0.005, I² = 80.9]. Running a comparable model with outcomes for Marinelli-Casey et al. 80 and Newton-Taylor et al. 84 no significant degree of heterogeneity was identified.

All remaining models, again favoured treatment with slightly greater combined decreases in other drug use following treatment compared to primary drug use. For Marinelli-Casey et al. ,80 this included ASI drug composite scores at 6 months subsequent to treatment. For Newton-Taylor et al. ,84 this included the proportion of court appearances during the first month at which self-reported drug use (any illicit substance) was recorded. The substances evaluated in separate analyses for Chun et al. 82 which showed slight decreases (all proportionately less than those reported by Marinelli-Casey et al. 80 and Newton-Taylor et al. 84) were cocaine use, amphetamine use and general drug use as measured by ASI drug composite scores. The relative impact of treatment on reductions in drug use was fairly similar across all models, ranging from a fixed OR of 2.14 to 2.97 (random OR 1.95 to 2.97).

Clearly, the outcome measures used in each of the three studies are, for the most part, not directly comparable, comprising a range of self-report outcomes for diverse drugs, scale-based outcomes and outcomes based on court records. To set out details for a model which incorporates measures which are as close as possible conceptually, Table 11 presents outcomes and heterogeneity statistics for three comparisons between drug court and non-drug court participants. Table 11 reports outcomes for SACPA compared with non-SACPA,82 and MTP plus drug court compared with MTP only. 80 Both of these are based on reductions in drug use measured by changes in ASI drug composite scores. Newton-Taylor et al. 84 differs from both other studies in that all participants are diverted via a drug court, but the comparators either successfully completed treatment (graduated group) or were expelled from the programme (expelled–non-engaged group). Although outcomes in this study are based on self-reported drug use, data were collected via court records not a purpose-designed scale. Nevertheless, study and outcome profiles are sufficiently similar statistically to be suited to inclusion in a combined analysis. Figure 4 provides the forest plot for this analysis.

TABLE 11 - Meta-analysis: continued use, drugs other than primary drug

TDTC, Toronto Drug Treatment Court.

Model	Study	Comparison			Outcome		Statistics for each study			z-value	p-value
							OR	95% CI
	Chun et al., 200782	SACPA vs. non-SACPA			ASI drug (change from assessment 1 to 2)		1.633	0.687 to 3.884		1.110	0.267
	Marinelli-Casey et al., 200880	Methamphetamine (MTP + drug court) vs. methamphetamine (MTP only)			ASI drug – 6 months after treatment		2.441	1.435 to 4.151		3.293	0.001
	Newton-Taylor et al., 200984	TDTC graduated vs. expelled–non-engaged			Court appearances – substance abuse reported		3.859	1.966 to 7.574		3.925	0.000
Fixed							2.609	1.792 to 3.799		5.001	0.000
Random							2.599	1.696 to 3.982		4.387	0.000
	Effect size (95% CI)	Test of null hypothesis (two-tail)		Heterogeneity				τ2
		z-value	p-value	τ2	df (Q)	p-value	I 2	τ2	Standard error	Variance	τ
Fixed	2.609 (1.792 to 3.799)	5.001	0.000	2.478	2.000	0.290	19.278	0.028	0.148	0.022	0.169
Random	2.599 (1.696 to 3.982)	4.387	0.000

FIGURE 4.

Continued other drug use: forest plot.

Model 4: treatment completion

As previously noted, treatment completion was a focus for the bulk of the included studies and 10 studies (62.5%),52,80,83–87,89,91,93 provided statistics suited to inclusion in meta-analysis. The primary focus for the majority of studies in this context was on the different completion rates achieved by users of different drug types. Three exceptions to this were:

• An exploration of treatment completion for different modes of intervention (MTP programme under drug court supervision vs. MTP programme without). 80

• A comparison of the number of treatment sessions attended during the first month of a programme (based on whether or not the participants subsequently graduated from the programme and, for those who failed to graduate, whether or not they were seen as engaging or not engaging with the programme). 84

• A comparison of completion rates for day compared with residential therapy. 93 Outcomes for Longshore et al. ,91 drew comparisons between drug types, but only in separate comparisons, with each comparison primarily focused on contrasting completion rates for different referral sources (e.g. SACPA vs. CJS).

Outcomes for all of these studies were included in an initial exploratory meta-analysis, but were subsequently excluded because of a lack of comparable focus in all other outcome measures. The very high levels of heterogeneity identified in the exploratory analysis (Q = 391.05, p < 0.0001, I² = 94.1) underlined the differences in approach to treatment completion adopted by the included studies.

Although the remaining studies52,83,85–87,89 all focused on comparing between drug types, the individual comparisons drawn were quite diverse (e.g. class A drugs vs. all other drugs; methamphetamine vs. other drugs; cocaine vs. heroin). A preliminary model including only the four studies which had a single outcome measure each was used to evaluate the extent of statistical heterogeneity these diverse outcome measures generated. These studies compared class A drugs with other drugs,83 heroin with other drugs,52 methamphetamine with other drugs86 and cocaine with cannabis. 87 Despite the difference in drug types included in the individual comparisons, no significant degree of heterogeneity was identified. Combining the study outcomes indicated a consistent pattern, with users of class A drugs showing lower completion rates overall than users of other drugs (fixed and random OR 0.918, CIs 0.881 to 0.957 and 0.853 to 0.987, respectively). The tight CIs are an indication that this fairly predictable outcome is robust.

We then explored the available outcomes for individual class A drugs. The first of these models compared treatment completion rates for heroin users with other drugs52,89 and heroin users with cannabis users. 85 Outcomes for this model were similar, with the combined OR for the studies (fixed and random OR 0.830, 95% CI 0.748 to 0.921) indicating a generally lower treatment completion rate for heroin users than for users of other drugs. The outcome for Passey et al. 52 was slightly more promising than for the other two studies, with heroin users (51%) and users of other drugs (48%) showing fairly similar completion rates.

A second model for individual types of drug compared treatment completion for cocaine users with completion rates for other drug users,89 cannabis users87 and alcohol users. 85 Again, the model showed no significant heterogeneity despite the different comparator drugs. Treatment completion for cocaine users was lower across the board than treatment completion rates for other drug users (fixed and random OR 0.731, 95% CI 0.663 to 0.806).

A final individual drug model with sufficient outcome measures for inclusion in meta-analysis focused on methamphetamine users, comparing treatment completion rates for these participants with users of other drugs86,89 and cannabis users. 85 This analysis trended in the same direction, with lower treatment completion rates identified in the two comparisons with unspecified other drugs, but slightly more positive outcomes for Anglin et al. 85 (treatment completion rates of 33% for methamphetamine users compared with 29% for cannabis users).

Given the overall lack of heterogeneity generated by combining between different drug type comparisons, we selected a single outcome measure from each study. The aim was to cover as diverse a range of class A drugs as possible within this set of studies. The choice of comparators was built up from the pragmatic base of four studies which had a single outcome measure available only – class A compared with other drugs,83 heroin compared with other drugs,52 methamphetamine compared with other drugs86 and cocaine compared with cannabis. 87 Hartley and Phillips89 provided an additional comparison between crack and other drug users. Anglin et al. 85 offered a number of outcome measures ranging across the class A drugs evaluated by the already included studies. The majority of these measures, included individually in a number of preliminary models, generated too much heterogeneity to allow valid combination. This left us with two choices of outcome measure for Anglin et al. ,85 contrasting treatment completion for cocaine users with cannabis users, the other contrasting treatment completion for heroin users with cannabis users. Of these, the OR for the heroin/cannabis comparison was more closely in line with ORs for the other included studies, so this gave us our final model. Outcomes from this model, which aimed to evaluate the strength of the evident trend for class A drug users to have lower rates of treatment completion are set out in Table 12 and Figure 5 provides a forest plot, to give a visual representation of the strength of effect. The details set out, suggest that the overall impact of primary drug type is to result in a slight but consistent reduction in treatment completion for class A drug users compared with users of other drugs.

TABLE 12 - Meta-analysis: treatment completion

MERIT, Magistrates Early Referral Into Treatment.

Model	Study	Comparison			Outcome		Statistics for each study				p-value
							OR	95% CI		z-value
	Anglin et al., 200785	SACPA (heroin/opiate vs. marijuana/hashish)			Treatment completion		0.821	0.736 to 0.915		–3.575	0.000
	Brecht and Urada, 201186	SACPA (methamphetamine vs. other drugs)			Treatment completion		0.919	0.881 to 0.958		–3.978	0.000
	Brewster, 200187	Drug court (cocaine vs. marijuana)			Survival in programme		0.931	0.512 to 1.691		–0.236	0.813
	Hartley and Phillips, 200189	Drug court (referred for crack vs. not)			Treatment completion		0.747	0.447 to 1.249		–1.111	0.266
	Passey et al., 200352	MERIT + heroin vs. MERIT + other drugs			Treatment completion		1.131	0.696 to 1.838		0.495	0.620
	Van Stelle et al., 199483	TAP (class A vs. other drug)			Treatment completion		0.519	0.251 to 1.074		–1.766	0.077
Fixed							0.904	0.870 to 0.940		–5.105	0.000
Random							0.878	0.801 to 0.961		–2.810	0.005
	Effect size (95% CI)	Test of null (2-tail)		Heterogeneity				τ2
		z-value	p-value	τ2	df (Q)	p-value	I 2	τ2	Standard error	Variance	τ
Fixed	0.904 (0.870 to 0.940)	–5.105	0.000	7.206	5.000	0.206	30.616	0.003	0.008	0.000	0.058
Random	0.878 (0.801 to 0.961)	–2.810	0.005

FIGURE 5.

Treatment completion: forest plot.

Model 5: other outcomes

Only two studies provided statistics for outcomes other than the above suited to inclusion in meta-analysis. 82,86 Brecht and Urada86 provided details for employment and training, homelessness and social support or conflict, Chun et al. 82 provided details for paid work and training and ASI composite scores for the scales medical, employment, legal, family and psychiatric. This allowed three comparisons to be drawn combining the two studies: currently employed/days paid in last 30 days; currently employed or in training/training in last 30 days and family conflict in past 30 days/ASI family change. For each of these combined analyses, Chun et al. 82 provided two outcome measures, one comparing SACPA-diverted with non-SACPA diverted participants, one comparing participants treated with methadone plus inclusion in a therapeutic community with participants receiving treatment via inclusion in a therapeutic community only. These measures were run in separate analyses.

Although statistically compatible, with no evidence of significant heterogeneity for any of the exploratory models, the outcome measures were too conceptually distinct to combine meaningfully, other than for the purposes of identifying possible trends in the data. For all three comparisons, Brecht and Urada86 contrasted outcomes for methamphetamine users with users of other drugs, while Chun et al. 82 provided comparisons for the interventions themselves (diversion via SACPA vs. diversion via probation and therapeutic community alone vs. therapeutic community plus methadone maintenance).

The analyses also failed to identify a consistent trend for these other outcomes. With regard to employment and training, Brecht and Urada86 found marginally better outcomes for methamphetamine users than users of other drugs, while both Chun et al. ’s82 experimental groups (SACPA diverted and participants treated with both therapeutic community and methadone maintenance) fared either slightly worse or were closely similar to their comparators in terms of broadly the same outcomes. With regard to family conflict, Brecht and Urada’s86 methamphetamine users showed a substantially worse profile than their other drug-using comparators, while Chun et al. ’s82 SACPA and treated groups showed outcomes very similar to their non-SACPA and therapeutic community-only comparators. In summary, the limited range of outcome measures available in the included studies and, again, their diversity of focus, prevents any meaningful conclusions being drawn from meta-analysis, despite the statistical comparability of the data presented.

Search strategy

The search strategy for the economic review combined the search terms used for the effectiveness review with economic search terms. As the economic review included those inclusion criteria used in the effectiveness review, economic search terms, plus a drug use limiter term were applied to the database of studies identified by the effectiveness review search strategy. The search strategy is listed in Appendix 7. The effectiveness review search string was also applied to further electronic databases; American Economic Association’s electronic bibliography (EconLit; via Ovid), and NHS Economic Evaluation Database (NHS EED; via Cochrane). Supplementary searches were performed on the following online sources: UK Home Office; Scottish Government; Northern Ireland Office; United Nations Office of Drugs and Crime; and a search for the author Christine Godfrey. Further efforts to identify additional studies included screening bibliographies of retrieved manuscripts, with a particular focus on review papers identified in the search.

Figure 6 sets out the process of identifying studies for inclusion in the economic review. Database searches took place from November 2011 to February 2012, retrieving studies published between January 1985 and January 2012. Libraries of retrieved studies were exported into Reference Manager (version 11; Thomson Reuters, San Francisco, CA, USA) software and duplicate entries removed.

FIGURE 6.

Flow diagram for identifying studies for the systematic review of economic evidence.

Inclusion criteria

The following inclusion criteria were applied to retrieved studies, in addition to criteria used to establish eligibility for the effectiveness review:

• focus on use of any of the following drugs: opiates; crack cocaine; powder cocaine

• inclusion of any of the following economic data: health economics analysis; details of patient costs; outcomes for inclusion in economic model

• comparison group available for economic analysis

• based on primary data collection or systematic review

• resource use and costs reported separately

• data other than charge data used

• report resource use or costs generalisable to the UK setting

• sufficient detail reported to extract costs and outcome data relevant to long-term comparisons of diversion and aftercare strategies for economic modelling.

An eligibility form, using these criteria (see Appendix 8), was applied to the titles and/or abstracts of studies identified by the search strategy. Inter-rater reliability was established with a second reviewer prior to screening. Random subsamples of those studies screened out as ineligible were subject to double-checking by a second reviewer at two further stages. Studies reported in books, conference proceedings, dissertations or theses were excluded.

Potentially relevant references were copied into a separate file and the full text of the article obtained. Two reviewers independently assessed studies, at this stage, for inclusion, with any differences resolved by discussion with a third reviewer. Only studies reporting a full economic evaluation were included in the review such as those:

• that included direct costs/savings and a measure of benefit to the participants

• where diversion/aftercare was compared with a comparison group not exposed to such an intervention

• where the incremental cost-effectiveness or cost–benefit was estimated.

Reasons for exclusion of full-text references were documented (see Figure 6).

Data extraction

An economic data extraction form was developed and piloted on five randomly selected studies (see Appendix 9).

Study flow

Database and bibliography searches identified 5443 potentially relevant studies. Screening of titles and/or abstracts led to the exclusion of 5389 studies, with 54 manuscripts proceeding to an examination of the full text. All of these studies were excluded from the review because of no focus on opiates, crack cocaine, or powder cocaine (37); no full economic evaluation (7); no usable outcomes data (1); not a study of diversion (4); or unable to obtain full text via library services (5). This process is summarised in Figure 6.

Partial economic evaluations

The partial economic evaluations51–55,83,88,96–101 were also reviewed to assess whether or not they included data to populate an economic model of the cost-effectiveness of diversion and aftercare. None of the studies included observed service use, unit cost or utility data to supplement the data sources identified for the economic model (see Chapter 5, Data sources). Reasons for this included:

1. primary or secondary resource use and costs not reported separately or in sufficient detail to extrapolate to current UK service context

2. observed resource use and costs related to services that are not relevant to the UK service context.

Population

The population for the model is opiate- and/or crack-using offenders who come into contact with the CJS. Offenders who are already in contact with treatment and aftercare services are included in this population since a proportion will have a new referral via the diversion service. The model for the primary analysis focuses on people who will receive a community-based sentence. This implies that the type of sentence is known before the decision to refer a person via the DIP. The rationale for this is based on the following factors:

• The majority of substance users report offences that would typically attract a community rather than custodial sentence, such as shoplifting, buying and selling stolen goods, other stealing. 26

• In the UK, there is no formal link between involvement with DIP and those sentencing options determined by the court, although having sought treatment may impact on the court’s decision.

• There is a paucity of data with which to estimate the proportions who would receive a community, custodial or no sentence with and without DIP.

However, the Drug Interventions Programme Operational Handbook39 is clear that the decision to use DIP should be made when a person is identified as a substance user at the point of arrest and before the sentence for the offences associated with the index arrest is known. In addition, for a proportion of people committing minor offences, the volume of repeat offending may lead to a custodial sentence. The relative cost-effectiveness of DIP for the broader population of substance user arrestees is explored in the sensitivity analysis. This broader population incorporates the possibility of a prison sentence or no sentence for the offences associated with the index arrest.

Model structure

The cost-effectiveness of diverting drug-misusing offenders into appropriate treatment at arrest, was compared with the usual pathway of such offenders through the CJS was assessed using a decision analytic model. The structure and design of the model was agreed with the wider project group which included experts in offending and drug use research and forensic practitioners.

The model for the primary analysis (community-based sentences) starts with a simple decision tree. This represents the sequence of possible events following an index arrest and decision to manage an offender suspected of substance use by DIP with or without a drug test over a 1-year time horizon. A simplified version of the decision tree is shown diagrammatically in Figures 7 (DIP and treatment pathways) and 8 (subsequent offending pathways) for the 1-year time horizon used in the primary analysis. Appendix 10 presents the full decision tree in tabular format. For the longer time horizons in the sensitivity analyses, subsequent cycles are included using a Markov approach.

FIGURE 7.

Decision tree, year 1. a, DIP and treatment pathways; and b, subsequent offending pathways.

FIGURE 8.

Decision tree, year 1 (continued): subsequent offending pathway.

The tree begins with a square decision node, where a choice is made between the alternative strategies of DIP with or without a drug test or no DIP with or without a drug test. Circular chance nodes indicate a point where a number of subsequent events are possible; these events are assigned a likelihood that they will occur (probability). The design of the tree sets out possible progression through a number of chance nodes from the starting point of an index arrest to the end point of further offence/no offence (within a 1-year time horizon). This design leads to 112 mutually exclusive sequences of events, or pathways (see Appendix 10). Each pathway probability reflects a joint probability, estimated by multiplying probabilities along the pathway. 110 The model does not explicitly include whether or not those entering treatment, or not, achieve abstinence or complete treatment, or not. This is because of a lack of clear indicators of what constitutes a completed treatment episode or what constitutes abstinence or a successful treatment outcome. However, there is evidence that longer times in treatment are associated with reduced substance use, lower levels of offending, lower overall costs and improved health status. Accordingly, the model explicitly includes the impact of treatment entry on new recorded offences. The model also includes time in treatment for the different branches of the pathway for those receiving treatment. The time in treatment is used to estimate the costs of treatment and the health status utilities associated with the different treatment pathways.

For the longer time horizons in the sensitivity analyses, at the start of the second and subsequent cycles, people can be in one of two states: alive or dead. Those alive will fall into one the following categories (Figure 9):

• substance user

• not dependent on class A drugs.

FIGURE 9.

Markov states in subsequent cycles for longer time horizons.

The series of events that occur within a cycle for people with a recorded offence include a chance that the person will be subject to DIP with or without a drug test, or not, and then incur the costs and QALYs associated with the treatment and reoffence pathways in the initial decision tree (see Figures 7 and 8).

Likelihood of events

Probability distributions

Treatment probabilities and offending probabilities were assigned beta distributions, with parameter approximation utilising estimates of the mean and standard deviation (SD). A series of logistic regressions were used to predict progress through the model based on available covariates in the probation/prison sample. Where low pathway numbers precluded distribution estimation, similar pathways were pooled to obtain probability distributions.

Resource use and costs

Pathway costs include the following:

• cost of DIP with or without drug test

• cost of drug test

• cost of drug treatment

• costs associated with arrest

• costs associated with prison

• costs associated with remaining in the community after arrest and conviction

• costs associated with subsequent recorded offence.

The offences associated with the index arrest are assumed to be the same whether a person follows the DIP or no DIP pathway. This implies that there will be no difference in the type and costs of community sentences between DIP and no DIP. Accordingly, the costs of the initial sentences are excluded from the primary analysis. The impact of assuming that DIP also has an impact on the type and cost of community sentence at this stage is explored in sensitivity analysis. All unit costs were adjusted to 2012 values using the annual percentage increase in the Retail Price Index, accessed via the ONS. 109 The cost of DIP was derived from unit costs provided by the National Treatment Agency for Substance Misuse (now part of Public Health England). This analysis costed contact per treatment entry via DIP at a range of values from £85 to £213, at 2004/5 prices.

The cost of a drug test was derived from the Home Office report, Evaluation of Drug Testing in the Criminal Justice System. 106 Cost per on-charge drug test across nine UK sites ranged from £44 to £168, with a mean unit cost of £79 (2001 prices). Not all people who receive DIP have a drug test; the unit cost of the drug test was weighted by the probability of having the test, based on actual proportions receiving a drug test in the DDW prison/probation sample.

The average (mean) cost per day of drug treatment was derived from the DTORS cost-effectiveness analysis. 42 The total cost of treatment from triage to second follow-up was divided by the number of days of treatment received from triage to second follow-up, for the DTORS participants who received treatment. This cost per day was then multiplied by the number of days of treatment recorded for the DDW sample.

The costs associated with offending behaviour included:

• cost of next arrest

• cost of prison

• cost of probation

• cost of subsequent offending.

Unit costs of offending were derived from a Home Office report,107 supplemented by the costs of offences produced for the DTORS study. 42 The unit costs of prison and probation are the average costs per offender, rather than the cost per offender receiving a prison or probation sentence, which is the cost required for the economic model. Published statistics111 reporting the proportion of offenders in prison and serving community-based sentences (community orders and suspended sentences) were applied to estimate the cost per offender sentenced to prison and the cost per offender having probation.

These costs were used to estimate the costs for each individual in the DDW prison/probation sample with an arrest. The arrests were costed by type of offence(s) occurring within a 1-year time horizon of the index arrest. A proportion (27%) of individuals arrested in the DDW prison/probation sample were charged with committing more than one offence. For those people in the DDW prison/probation sample with only one offence, the cost of arrest included the CJS cost of proceedings for that offence. In addition, the wider non-CJS costs of offence were added to the costs of proceedings.

The costs associated with a number of crime types were not available (drugs offences – misuse; drugs offences – supply; fraud and forgery; other indictable offences; and other summary offences). Costs assigned to these categories were estimated using the average CJS unit costs associated with minor crime (theft; theft of vehicle; theft from vehicle; attempted vehicle theft; criminal damage).

For the primary analysis, the costs of subsequent offending were estimated from the model to take into account the uncertainty in the data. For the sensitivity analysis the costs associated with subsequent offending were estimated using the mean total offending costs of arrested individuals in the sample:

Costs associated with the model were assigned gamma (mean and SD) or triangular distributions (best estimate and range).

Outcomes, utility values and quality-adjusted life-years

Utilities assigned to each pathway were derived from the DTORS cost-effectiveness analysis. 42 DTORS participants completed a self-report measure of health (Short Form questionnaire-12 items). 112 This assesses physical function, limitations in role as a result of physical or emotional problems, the effect of pain on normal work and activities, general health, vitality, and the impact of physical or emotional problems on social activities and mental health. 42 Utility weights derived from a general population sample were used to aggregate question responses into a composite measure of health-related quality of life. 113

The data from DTORS were reanalysed to estimate the mean utilities of the index arrest and utility while in treatment. SDs were calculated to estimate distributions for the model.

The DTORS participants were referred from a variety of CJS and non-CJS, not all of which were formal DIP schemes. However, at baseline the reported health status and utility values did not vary between the different forms of CJS referral and referral from other sources (linear regression, p > 0.20). Accordingly, the full DTORS sample who reported offending in the 4 weeks prior to baseline was used to estimate the starting utilities for the DIP with or without drug test pathways. There were differences in the utility values at follow-up and by whether or not the person started treatment. For those who started treatment, utility increased as time in treatment increased. Accordingly, separate utility values were estimated for those who entered treatment and those who did not, which were applied to the treatment and no treatment events in the model. The utility values were multiplied by time spent in the treatment pathways to estimate QALYs for this section of the model.

The DTORS sample was characterised by differences in the utility values of those who reported offending in the 4 weeks prior to follow-up and those who did not. For pathways resulting in treatment and no recorded offence, the utility value was estimated from the DTORS sample completing a follow-up assessment, receiving treatment and not reporting an offence in the previous 4 weeks. QALYs for this section of the model were estimated as 365 days minus time in treatment.

For pathways resulting in treatment and a recorded offence, the utility value was estimated from the DTORS sample who completing a follow-up assessment, had treatment and reported an offence in the previous 4 weeks. It was not possible to estimate utility values from the DTORS data for arrest and subsequent imprisonment or community sentence. For the primary analysis, it was assumed that arrest and imprisonment would reduce health status and utilities to baseline values, but that arrest and community sentence would not. The DTORS sample who reported offending at follow-up were used to estimate the utility values to apply to the model pathways for community sentences. These utility values were applied to the time remaining following treatment, for those who received treatment and had a recorded offence.

Key assumptions

1. The offences associated with the index arrest are assumed to be the same whether a person follows the DIP or no DIP pathway. This suggests that there will be no difference in the type and costs of community sentences between DIP and no DIP. Accordingly, the costs of the initial sentences are excluded from the primary analysis.

2. The primary analysis focuses on the population of offenders who will receive a community sentence for the offences associated with the index arrest.

3. The primary analysis includes people who are already attending treatment at the index arrest.

4. The primary analysis assigns all DIP and treatment costs at the start of the 1-year time horizon. This assumes that there will be no additional DIP or treatment costs following subsequent offences. The approach gives an accurate estimate of the total DIP costs and community-based treatment for the first year (estimated from the DDW). However, it may underestimate the costs of prison-based treatment services for those who reoffend and receive a prison sentence.

5. For the primary analysis, the probability and costs of recorded offences, arrests, type of offence and whether a person received a prison or community sentence was assumed to be the same for the DIP and no DIP arms of the model. This was based on the assumption that the main effect of a CJS treatment referral was on entry into treatment. A related assumption is that subsequent offending behaviour depended on engagement with treatment and the effectiveness of treatment services. It is also supported by the analysis of DDW data used to estimate parameter values. However, it is important to note that the DDW data may reflect selection biases because of differences in the characteristics of people who received CJS treatment referral via the DIP and those who did not. In addition, it is possible CJS treatment referral also affected the severity of offending behaviour and the CJS and non-CJS costs of offences.

6. Health status and utility was higher after treatment than before. It was also assumed that any gains in utility were maintained after treatment, for those people who did not reoffend. This implies either that there was no relapse or that relapse did not affect health status and utility.

7. People who entered treatment and had a recorded offence after the index arrest did so because treatment was less effective or ineffective at dealing with the drug use problem and, therefore, offending behaviour. Accordingly, the health status and utility of people with a recorded offence following the index arrest was assumed to be lower than during treatment, resulting in lower overall QALYs. In addition, it was assumed that health status after treatment remained higher than that at baseline, for people with a recorded offence.

Primary analysis

Table 21 shows the overall expected 1-year cost and QALYs for the DIP and no DIP options considered in the model and the incremental or net costs and QALYs associated with DIP. The cost-effectiveness plane is shown in Figure 10, and Figure 11 presents the CEAC.

FIGURE 10.

Cost-effectiveness plane, primary analysis.

FIGURE 11.

Cost-effectiveness acceptability curve of DIP, primary analysis.

Overall, DIP is associated with net additional savings and a small net QALY. However, the CIs for each of these are wide and cross zero. This suggests that any differences are not statistically significant and could be due to chance. The cost-effectiveness plane shown in Figure 10 indicates that DIP is likely to be associated with similar costs and similar QALYs than no DIP in the majority of the 10,000 simulations run for the analysis.

The CEAC (see Figure 11) suggests that DIP may be cost-effective in around 50% of the simulations. This means that if decision-makers were willing to pay up to £30,000 to gain one additional QALY for arrested drug users to receive DIP, there may be a 50% chance that DIP would be cost-effective. However, this result needs to be treated with caution, as the primary analysis indicates that there is no evidence of a difference in costs or QALYs between DIP and no DIP. Furthermore, limitations with the data and uncertainty about structural aspects of the evaluation mean that it is not clear whether DIP has no impact compared with no DIP or whether or not the evidence is insufficient to identify a difference. The sensitivity analyses explore whether or not changes in key parameters, design choices or the model structure would change the conclusions of the analysis and whether or not there are specific parameters to focus on in further research.

Sensitivity analyses

Table 22 shows the results of key sensitivity analyses to address uncertainty in the sources and methods used to estimate data. Table 23 reports the results of key sensitivity analyses to assess the impact of changes in the model structure and design choices. Only the results where the probability that diversion is cost-effective is < 48% or > 52% are shown. The parameters that were changed for the sensitivity analyses are shown in Appendix 10 together with the results of the sensitivity analyses conducted.

Overall, the sensitivity analysis indicates that there is substantial uncertainty. Although the probability that diversion is cost-effective is > 52% in some cases, the 95% CIs for the net costs, net outcomes and NB statistic are wide and all cross zero.

Summary of economic model analyses

The primary and sensitivity analyses indicate a high level of uncertainty with a high level of variance around the net costs, outcomes and NB statistics. All the analyses demonstrated wide 95th percentiles around the net costs, QALYs and NB statistics. This means that there is a lack of evidence with which to draw conclusions about the cost-effectiveness or otherwise of diversion. This is supported by the use of multiple-way sensitivity analyses to reflect changes in more than one parameter at a time.

Scope of available effectiveness evidence

The scope of the literature initially identified for inclusion in the review covered nearly 1300 studies that potentially met the inclusion criteria. However, only 16 provided data that were relevant to the population, intervention, control and outcomes specified for the review.

Nearly one-third of the studies identified for possible inclusion (n = 400, 30.8%) were excluded from analysis. This was because it was not possible to identify outcomes specifically for class A drug users, the population of interest for this review. The outcomes evaluated in the review indicated that the type of drug use prior to or at the start of a diversion intervention is important in determining the likely impact of an intervention. The analysis of studies included in the review also highlights the importance of tailoring interventions to the needs of particular user groups. These factors mean that the under-reporting of the characteristics of study participants found in the studies screened is particularly unfortunate. One of the key requirements of effective clinical practice is to be able to identify whether or not a given intervention is appropriate for a particular user group. The current research literature does not facilitate this.

In common with other similar literatures119 an additional problem is the lack of empirical evidence about the impact of the intervention on participants. Nearly half of the available, potentially relevant, studies (n = 627, 48.2%) presented little or no evidence at all of the outcomes of the interventions considered. These 627 studies focused instead on describing an intervention; a narrative evaluation of process, procedures or implementation; or prioritising a discussion of policy-related issues. All of these are, of course, valid contributions to the literature. However, in the absence of an evidence base, they achieve little in terms of the core goal of improving clinical practice and promoting a reduction in drug use and offending.

Evidence of effectiveness

Included studies were of poor methodological quality and characterised by modest sample size, high attrition rate, retrospective data collection, limited follow-up and no random allocation. The majority of studies comprised US methamphetamine users and there was evidence of publication bias in the available literature. Meta-analysis indicated a consistent but small impact of diversion interventions reducing drug use [an OR of 1.68 (95% CI 1.12 to 2.53) for reduced primary drug use and an OR of 2.60 (95% CI 1.70 to 3.98) for reduced use of other (secondary) drugs].

Evidence of cost-effectiveness

A systematic review of the cost-effectiveness of diversion and aftercare for opiate-, crack-, or cocaine-using adult offenders did not identify any relevant studies. The majority of studies identified by the search and screening of abstracts and titles were excluded because they did not meet the criteria for a full economic evaluation. These criteria included a comparison of the diversion intervention with an alternative and a formal comparison of the net costs and benefits (i.e. effectiveness, QALY or monetary measure of participant health and well-being) of diversion in a cost-effectiveness or cost–benefit ratio.

The analyses of the economic model highlighted a lack of evidence for the cost-effectiveness or otherwise of diversion in the UK setting. This was primarily because of high uncertainty in the data estimates. The primary analysis indicated the 95th percentiles for both net cost and net QALYs cross zero, suggesting no statistically significant differences in cost and outcome (net cost –£147, 95th percentiles –£17,573 to £16,317; net QALY 0.005, 95th percentiles –0.057 to 0.065). The wide 95th percentiles of net cost, net outcome and NB for the primary and sensitivity analyses indicate a high level of uncertainty about the parameter estimates used. Furthermore, limitations with the data and uncertainty about structural aspects of the evaluation mean that it is not clear whether or not DIP has no impact compared to no DIP or whether or not the evidence is insufficient to identify a difference.

Effectiveness

Cost-effectiveness

Economic model

Development and standardisation of interventions

The systematic review highlighted that there are a number of inter-related components in the way in which drug-using offenders are identified, diverted into treatment, as well as in the treatment provided. In particular, the effectiveness of any diversion scheme in reducing drug use and/or offending is reliant on the effectiveness of the treatment package a person is diverted into. Additionally, there is variation in drug-using offenders in terms of the level and duration of offending, type of drug use and reasons for drug use and offending that indicate potential differences in behaviour and the effectiveness of both diversion and subsequent treatment.

The studies included in the review were typically ad-hoc evaluations of existing interventions, which were inadequately described. While this has many benefits in terms of evaluating real-life practice, the rationale for the intervention (e.g. theoretical models of causation) and understanding of the mode of action is unclear.

An important finding of this review was that the included evaluations did not include assessment of the acceptability and feasibility of the interventions. In addition, it is not clear what proportion of drug-using offenders will receive a DIP assessment/drug test. The estimate used for the sensitivity analysis was 14%, but this was based on expert opinion. National statistics suggest that 57,000 people arrested had a formal DIP assessment in 2009. However, it is unclear how many people arrested were eligible for a DIP assessment, with or without drug testing. In addition, the proportion of treatment referrals from DIP is approximately 13% of all referrals and 30% of new referrals. There may be a number of reasons why DIP is, or is not, used and evidence about these issues is required.

Key methods to address research priorities 1 and 2 include best evidence reviews of the qualitative, quantitative and mixed-methods literature about the development and process of diversion (this systematic review excluded over 600 papers concerned with a narrative evaluation of process, procedures or implementation; or prioritising a discussion of policy-related issues). This may also need to include (1) elements of realist review methods to map relevant theories and models underlying diversion programmes to evidence of their effectiveness; (2) interviews and focus groups with stakeholders (e.g. drug-using offenders and staff in the CJS and drug treatment services as well as staff in related health and social care services); and (3) review and analysis of documentary evidence of local and national policies and procedures, existing diversion and national treatment databases to identify the processes and mechanisms which facilitate or hinder the implementation of diversion programmes. For example, the NDTMS now provides the capacity for long-term follow-up of treatment access and outcomes among those contacted by diversionary schemes.

Development of core outcome set

This review highlighted the diversity of outcomes and measurement methods used to evaluate the effectiveness of diversion and the difficulties this raises for meaningful meta-analysis of study outcomes. Another key finding is that existing studies typically focus on a limited range of outcomes of diversion programmes, mainly centred around treatment completion. The impact of diversion on offending (drug and non-drug offences), changes in drug-using behaviour or abstinence was not consistently evaluated, and the impact on participant health and well-being or broader social benefits was not included in the studies reviewed. The economic evaluation highlighted a lack of evaluation of health status and use of measures to estimate health-related utility and QALYs. The latter are currently the measure preferred by NICE for health technology assessments. However, whether or not they are appropriate measures of outcome for evaluations of the effectiveness and cost-effectiveness of diversion is an area where further research is required. 131

Key stages and methods include:

1. Qualitative research with stakeholders to identify the range of relevant outcomes. The stakeholders include drug-using offenders and family/carers; professionals in relevant CJS, treatment health and social care services; researchers/experts; and policy-makers.

2. Review of existing literature to identify any additional outcomes and supplement the list developed from the qualitative research.

3. Formal methods such as Delphi surveys and consensus meetings with stakeholders to agree and validate the content of the core outcome set.

4. A systematic literature review to identify outcome measures and assess the properties of those instruments for inclusion in the core outcome set.

5. Stated preference surveys to assess the preferences and priorities of key decision-makers (service users, carers, health care professionals and policy-makers) for the core outcomes identified in stages 1–3.

Evaluation of specific interventions

The systematic review and economic evaluation indicated that the evidence from published evaluations of effectiveness, cost-effectiveness and existing secondary data in the UK is uncertain. This is as a result of the low methodological quality of the studies included in the review. There were also differences between the US and UK settings, which means that the relevance of much of the evidence to the UK was unclear. The available data from secondary sources in the UK were collected for different purposes. This means that they do not include key measures to estimate the service use and outcomes of events related to diversion or to control for biases in the data. An implication is that further research may be needed to evaluate the effectiveness and cost-effectiveness of diversion programmes in the UK CJS and treatment settings, and in the UK drug-using population. Whether or not further evaluation is required, and what interventions are assessed, will depend to some extent on the outcomes of the research outlined above. This will also determine the extent of the feasibility and pilot work (described in research priorities 4 and 5 below) required. In general, it is important that the work below includes accurate recording of the drug-use profile of participants; the intervention(s) and its component parts and the characteristics of participants. Important aspects of the latter include ethnicity, sociodemographic profile, educational level, mental health status and prior and current offending behaviour.

The review highlighted the fact that evidence-based research in the criminal justice context remains in its infancy and many studies cited ethical and practical constraints on the use of RCTs in criminal justice settings. However, others in the field have argued that the current lack of high-quality studies capable of providing robust evidence of effectiveness is no longer justifiable. 120 The evidence considered in this review and economic evaluation suggests that a RCT is likely to be necessary to identify potentially small, but important and relevant, effects of the intervention that may take some time to occur following diversion. In addition, there are likely to be important effects of selection, allocation and unobservable biases.

Limitations of the existing evidence may in part be due to the fact that they were ad hoc evaluations of existing interventions, which in some cases made use of existing higher-level data. There was inadequate description of the following:

• participant samples and the target population

• recruitment and selection procedures

• the components of the diversion interventions and any comparators and the rationale for their selection

• whether or not there was sufficient sample size to give the power to detect statistically significant and important differences in effectiveness.

In addition, it was unclear whether or not there was adequate follow-up post intervention to identify the impact of diversion on subsequent treatment, reoffending and substance use. These factors were identified in the economic evaluation as potentially important influences on the costs and outcomes of DIP. Finally, triangulation of key outcome measures against corroborating data is needed to assess the reliability and validity of outcome measures and process evaluation to assess procedures and explore the experience of participants.

Efficacy of diversion and aftercare programmes for offenders using class A drugs

Methods

Applicants should demonstrate knowledge of current research in the field and of systematic review methods and state how these would apply to the question posed. Valid and reliable methods should be proposed for identifying and selecting relevant material, assessing its quality and synthesising the results. Guidance on choice of appropriate methods is contained in NHS CRD Report 4 Undertaking systematic reviews of research on effectiveness (www.york.ac.uk/inst/crd/report4.htm). Where policy implications are considered, the emphasis should be on assessing the likely effects of a range of policy options open to decision-makers rather than a judgement on any single strategy. Where epidemiological modelling or economic evaluation is required, the range of uncertainty associated with the results should be assessed. In the assessment of cost-effectiveness, further data collection may be required to estimate resource use and costs. If there is evidence that the ratio of costs and benefits may differ between readily identifiable groups, applicants are encouraged to state how they will identify these differences.

Cochrane

Applicants wishing to produce and maintain a Cochrane systematic review from this HTA commissioned systematic review should make the case in their proposal. This will need to include the approval of the relevant Cochrane Review Group (www.cochrane.org). Any additional costs associated with the initial preparation of a Cochrane review should be included in your project proposal. Maintenance costs cannot be met.

Public involvement in research

The HTA programme recognises the benefit of increasing active involvement of members of the public in research and would like to support research projects appropriately. The HTA programme encourages applicants to consider how the scientific quality, feasibility or practicality of their proposal could be improved by involving members of the public. Examples of how this has been done for health technology assessment projects can be found at http://www.hta.ac.uk/PPIguidance/. Research teams wishing to involve members of the public should include in their application: the aims of active involvement in this project; a description of the members of the public (to be) involved; a description of the methods of involvement; and an appropriate budget. Applications that involve members of the public will not, for that reason alone, be favoured over proposals that do not but it is hoped that the involvement of members of the public will improve the quality of the application.

Updating

It is the policy of NETSCC, HTA that all search strategies undertaken as part of evidence synthesis/secondary research projects must not be more than 12 months out of date when the draft final report is submitted. We expect that most projects will manage to bring their searches up to date prior to analysis and writing up. As research funders we are aware that exceptional circumstances can apply that would not allow this to be case but this must be the exception rather than the rule and will be assessed on a case-by-case basis. The expectation is that projects funded by the HTA programme will deliver information that is both relevant and timely.

In addition, in order to inform decisions on whether and when to update the review, researchers will be expected to give some indication of how fast the evidence base is changing in the field concerned, based on the nature and volume of on-going work known at the time the review is completed. Applicants should note that they will not be expected to carry out any future updating as part of the contract to complete the review.

Communication

Communication of the results of research to decision-makers in the NHS is central to the HTA Programme. Successful applicants will be required to submit a single final report for publication by the HTA programme. They are also required to communicate their work through peer-reviewed journals and may also be asked to support NETSCC, HTA in further efforts to ensure that results are readily available to all relevant parties in the NHS. Where findings demonstrate continuing uncertainty, these should be highlighted as areas for further research.