Screening for psychological and mental health difficulties in young people who offend: a systematic review and decision model

Article history

The research reported in this issue of the journal was funded by the HTA programme as project number 10/35/01. The contractual start date was in August 2011. The draft report began editorial review in May 2013 and was accepted for publication in January 2014. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

Simon Gilbody is a member of the National Institute for Health Research Health Technology Assessment Clinical Evaluation and Trials Board.

Copyright statement

© Queen’s Printer and Controller of HMSO 2015. This work was produced by Richardson et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

Mental health difficulties in young people who offend

In England and Wales young people between the ages of 10 and 17 years committed 201,800 offences in 2009–10 and were responsible for 17% of all proven offending. 1 Problems linked to offending behaviour include educational underachievement, substance abuse and mental illness. With regard to mental illness, there is a lack of precise estimates of the prevalence and types of mental health difficulties experienced by young people who offend, but what evidence there is suggests prevalence figures substantially in excess of age-equivalent, general population rates. 2,3

The types of difficulties for which these rates are elevated cover a wide range of mental health problems, including depression, anxiety [particularly post-traumatic stress disorder (PTSD)], attention deficit hyperactivity disorder (ADHD), psychotic-like symptoms and self-harm. 3,4 There is also some evidence that rates of learning disability and special educational needs may be high among this group. 5 The presence of mental health difficulties among young people who offend may increase the risk of a range of negative outcomes for both the young person and the wider community. The presence of mental health problems such as depression among this group may act as a risk factor for the persistence of offending behaviour into adulthood. 6 Additionally, conduct disorder in young people leads to a range of difficulties. For example, adults who had conduct disorder in adolescence are 70 times more likely to be imprisoned before the age of 25 years. 7 The cost of crime in England and Wales committed by adults who had conduct disorder as a child has been estimated at £2.25B. 8

Screening for mental health difficulties in young people who offend

Despite their prevalence and potential to increase the risk of negative outcomes, these difficulties remain under-identified and under-treated. 4,9 A number of screening methods have been used to identify young people with mental health problems, including those specifically tailored to young people who offend [e.g. Massachusetts Youth Screening Instrument – version 2 (MAYSI-2)10] and those developed in general settings.

Interventions for mental health difficulties in young people who offend

Screening, however, can be justified only if it results in a more effective treatment than would otherwise be the case and does so with a favourable ratio of costs to benefits. 11 There is substantial evidence from studies of young non-offender samples that effective psychological and pharmacological treatments exist for many of the mental health problems that are common in young people who offend, including treatments for depression,12 anxiety problems,13–15 ADHD16 and psychotic-like symptoms. 17 Evidence on the effectiveness of mental health interventions specifically for young people who offend appears to be very limited. One previous systematic review in this area cautiously concluded that treatments may be effective, although it suggested that larger, high-quality trials were needed. 18

Current UK policy and practice

In the UK, all young people who have offended are the responsibility of youth offending teams. The majority of young people are supervised in the community, but a smaller proportion is given a sentence that includes a custodial component, which is then followed by subsequent supervision in the community. A number of factors determine whether a young person is given a custodial sentence or is supervised in the community; these include the severity of the crime and the extent of any previous offending behaviour.

Those young people supervised in the community will typically receive either a referral order, if a first offence, or a youth rehabilitation order. These orders can vary in terms of both the level of supervision required and the additional conditions placed on the young person. For those people given a custodial sentence, the setting in which the person is placed will be determined by age and level of maturity. Secure children’s homes are typically for young people aged 10–15 years and young offender institutions are for those aged ≥ 16 years. There is also the option of a secure training centre for those aged 15 or 16 years.

There has been a focus in the UK in recent years on the prevention of offending by young people and the treatment of these young people. 19 As a result, there is substantial policy interest in screening for mental health problems in people who offend and access to appropriate mental health services for such people. The Criminal Justice Bill 2007 and Lord Bradley’s report20 set out a range of strategies to improve the situation, including youth rehabilitation orders. Department of Health guidelines21 have also made it clear that young people in secure settings should have appropriate access to mental health services. Such policy documents are supported by the recent national framework to improve mental health and well-being,22 which organises six high-level objectives of mental health strategy across all sectors of society.

Available evidence on practice suggests that the provision of adequate mental health screening and intervention remains patchy, with demand outstripping supply. 4 A joint initiative between the Youth Justice Board and the Department of Health has sought to improve the identification and assessment of health-related needs in children and young people in contact with the youth justice system, including mental health needs. This initiative has led to the Comprehensive Health Assessment Tool (CHAT),23 a bespoke, strategic toolkit that aims to identify and assess health-related needs of children and young people in contact with any part of the youth justice system. Within CHAT there are five areas of assessment:

• part 1 assesses any immediate risk associated with physical health, mental health, substance misuse and safety

• part 2 assesses physical health

• part 3 assesses substance misuse

• part 4 assesses mental health

• part 5 assesses neurodevelopment disorders such as learning disability, autistic spectrum disorders and speech and language impairment, as well as any traumatic brain injury.

Versions of CHAT are available for the secure estate and community settings. For the secure estate, a reception health screen needs to be completed within 2 hours or before the first night of admission to identify any immediate risks or concerns, which may lead to the fast-tracking of a more detailed CHAT evaluation for any areas identified as important. All areas of the CHAT assessment should then be completed in the first 10 days of intake, with the mental health assessment being completed within the first 3 days. For community settings the health reception screen is not used.

It is intended that the information from an updated version of Asset,24 termed AssetPlus, will be made available to the professionals conducting the CHAT assessment in both incarcerated and community settings. Asset is an assessment tool that aims to identify those risk factors for the young person’s offending, including mental health difficulties. 24 It can be used, therefore, to identify potential mental health needs that may require further assessment and intervention. CHAT will replace previous mental health screening pathways, which, following a red flag on the Asset tool, involved further structured assessment with measures such as the Screening Interview for Adolescents (SIfA). 25

The Comprehensive Health Assessment Tool began to be rolled out in 2012. The clinical effectiveness and cost-effectiveness of this screening strategy has not yet been determined.

Summary

A wide range of mental health problems are common in young people who offend, and their presence is linked to a range of negative consequences both for the young person and for the wider society. There is currently substantial policy interest in screening for mental health problems in young people who offend, but the value of such screening is currently unknown.

Screening

One option for identifying mental health problems in young people who offend would be to offer this entire group a detailed diagnostic mental health assessment in the form of a gold standard interview conducted to internationally recognised criteria. 26,27 There are advantages to this: all who were offered treatment would be in need of it and all of those not given treatment would not require it. Although such an approach would give perfect precision, it may not be feasible because it may require substantial resources to implement.

The use of screening instruments, which trade a saving in resources for a reduction in precision, is the typical alternative to such a strategy. 11 Screening measures that have been used with young offenders can be divided into a number of broad categories: those that are designed to detect a specific mental health problem, such as major depression, and those that are designed to detect a general mental health problem or need. 28 Often this maps onto a division in young offender measures between those instruments that provide diagnostic test accuracy data and those that identify a mental health need but do not establish the accuracy against a gold standard diagnostic interview.

A further division is into those measures that are specifically designed for use with a young offender population (e.g. MAYSI-2,10 CHAT mental health screen23) and those that are used with young offenders but which were originally developed for use in the wider population. A potential advantage of measures designed specifically for young offenders is that they may consider expected characteristics of the population (e.g. limited literacy) and may be designed for use by youth justice personnel with no formal mental health training. However, a potential disadvantage is that they may not have received the same level of psychometric evaluation as some of the more widely used measures.

Each screening instrument from these broad categories could be used in a number of ways to make a decision about a person’s mental health needs, including the need for treatment. Scores on a screening measure could be considered alone in making that decision, in combination with each other (e.g. a general screen for any mental health problem followed by a disorder-specific screen) or in combination with a gold standard (e.g. a general screen followed by gold standard interview for all those scoring positively on the screen).

Currently, there is uncertainty around which broad category of instrument is likely to be most effective (e.g. bespoke measures for young offenders vs. measures originally designed for use in the wider population) and within a category it is unclear if particular screening instruments are more accurate than others in identifying mental health problems. In addition, there is further uncertainty around whether a decision should be made on the basis of a single instrument or whether a combination should be used in a screening pathway.

Many screening instruments have a range of possible scores and so it is possible to identify different points along that range above which a person could be predicted by the screening instrument to have a mental health difficulty. As this cut-off point is varied, sensitivity and specificity will also change in a consistent way: as sensitivity increases, specificity will decrease (and vice versa). 11 (For an introduction to methods of quantifying diagnostic test accuracy, including concepts such as sensitivity and specificity, see Appendix 1.) There is, then, always a balance to be struck: if sensitivity is high, specificity is likely to be low; if specificity is high, sensitivity is likely to be low. A decision needs to be made about what balance between sensitivity and specificity is likely to be appropriate in a particular decision context. There are no definitive guidelines but, as a general rule, when the clinical context involves screening, high sensitivity is usually valued over high specificity. If sensitivity is high, this means that few people who have a condition will be missed, even if this is at the expense of somewhat lower specificity. Ensuring that few people with the condition are missed is often an aim of a screening strategy. However, in many decision contexts – including screening for mental health problems in young people who offend – it may not be possible to ensure very high sensitivity. Screening measures for mental health problems can have substantial inaccuracies when assessed against a gold standard, which means that very high sensitivity on such instruments is likely to be associated with low specificity. A consequence of low specificity is a high false-positive rate, which can be problematic in a number of ways. For example, if screening is used in the absence of a confirmatory gold standard diagnosis, treatment may be offered to many people who do not in fact require it. This may be potentially damaging to the recipients and can have substantial costs attached to it for services. Even if a screening measure is followed by a confirmatory diagnostic assessment, it may be inefficient and prohibitively costly to refer on for that further assessment all people who score positive to a screen if that number contains a large number of false positives. As a very broad guideline, then, a cut-off on a screening instrument may be required that gives sufficiently high sensitivity while retaining moderate specificity.

Studies of diagnostic test accuracy typically evaluate the screening measure against a gold standard categorisation of those with and without the mental health diagnosis, regardless of whether or not the true cases are already known to services or are previously unidentified cases. In this particular decision context, the screening for mental health problems in young people who offend, screening may be of value only for the identification of previously unidentified cases, because known cases may already be receiving treatment. There are a number of uncertainties related to this distinction between known and unidentified cases. It is unclear if the diagnostic performance of the test may differ if restricted to the identification of previously unknown cases. It is also unclear if the characteristics of the previously unidentified cases and the already identified cases differ, and this may be of relevance to understanding the likely performance characteristics of a test when restricted to the identification of new cases. For example, it is possible that already known cases will be more severe and therefore easily identifiable in the absence of screening, whereas unidentified cases may be less severe. This may have consequences for the need to offer treatment or the type of treatment offered. The prevalence of unidentified cases is also unclear, and this may have consequences for the balance between true positives and false positives at a particular cut-off point on an instrument. This in turn may affect the selection of an optimal cut-off point and the balance it offers between sensitivity and specificity.

Additional features of the decision problem relate to uncertainties about the behaviour of professionals in terms of screening. For example, it is unknown whether or not professionals find particular instruments acceptable and whether or not the results from a screening measure have an impact on professionals’ behaviour, such as making a referral for a particular type of treatment.

Clinical effectiveness and cost-effectiveness

On the assumption that professional behaviour is altered by the results of a screening test, screening and referring are of use only if there is an effective and cost-effective treatment for the particular mental health problem. In terms of effectiveness there are a large number of uncertainties. These include whether or not interventions for mental health problems in young people who offend are clinically effective and cost-effective, whether or not improvements in mental health symptoms are related to changes in other outcomes, such as the likelihood of reoffending, whether or not the interventions are acceptable to this population and whether or not potentially effective interventions can be feasibly delivered in UK settings.

Setting

Young people who have offended may be in the community or incarcerated. In terms of the decision problem outlined above, each of the considerations applies separately to these two settings. It is possible, for example, that a distinct screening pathway may be more appropriate in one setting than in another.

Objectives

On the basis of this decision problem we developed five objectives related to diagnostic test accuracy, the clinical effectiveness and cost-effectiveness of screening and (more broadly) the clinical effectiveness and cost-effectiveness of interventions for mental health problems in young people who offend. These five objectives are to:

1. conduct a systematic review and evidence synthesis of the diagnostic properties and validity of existing screening measures for mental health problems in young people who offend

2. assess the clinical effectiveness of screening strategies in this population and (more broadly) the clinical effectiveness of interventions for mental health problems

3. assess the cost-effectiveness of screening strategies in this population and (more broadly) the cost-effectiveness of interventions for mental health problems, with specific reference to identifying in which groups they may be cost-effective

4. assess whether or not current screening strategies meet minimum criteria laid down by the UK National Screening Committee (NSC) in the light of this evidence synthesis

5. identify research priorities and the value of developing future research into screening strategies for young offenders with mental health problems.

Structure of the report

We carried out a single comprehensive search to identify the evidence needed for this research. This search is described in Chapter 3. We then conducted the research in a number of interlinked phases in which we summarised the available literature on screening assessments for treatable psychological and mental health conditions in young offenders.

At each stage of the review and in the production of the final report we adhered to the relevant guidelines for the conduct and reporting of systematic reviews. 29,30 The research is registered on the PROSPERO database (registration number CRD42011001466). A copy of the original protocol for the review is available alongside copies of this report on the National Institute for Health Research (NIHR) website (www.journalslibrary.nihr.ac.uk/).

Stakeholder involvement

We established an expert advisory group and two stakeholder groups. The expert advisory group consisted of academics with methodological expertise in the conduct of systematic reviews and content expertise in the criminal justice system. Members of this group were approached at various stages of the project to offer advice on specific questions.

One stakeholder group consisted of professionals working within the justice system. We sought to include professionals working in both community settings and the secure estate. We met with members of this stakeholder group at various stages of the project. A specific role of this group was to help establish current UK practice in the screening and treatment of mental health problems in young people who offend and more generally to clarify the nature of the decision problem.

A second stakeholder group consisted of young people (age range 10–15 years) from the National Association for the Care and Resettlement of Offenders (Nacro). We held two meetings with these young people to gather their views on a range of subjects relevant to the review, including the acceptability of different potential screening pathways and different types of interventions. The older members of this group were asked to comment and help draft the plain English summary.

Appendix 2 provides a list of the stakeholders and professionals who provided advice during the review process.

Search terms

The search strategies were devised using a combination of subject indexing terms, such as medical subject heading (MeSH) in MEDLINE, and free-text search terms in the title and abstract. The search terms were identified through discussion among the research team, through contact with members of the advisory group, by scanning background literature and by browsing database thesauri.

We considered two main approaches to searching the literature: a single comprehensive search to identify studies of relevance to each phase of the review (e.g. screening, clinical effectiveness, cost-effectiveness) and an alternative strategy of developing separate searches for each phase. We chose to use a single comprehensive search as the most effective and efficient means of identifying the relevant literature for each phase.

The search terms for each database covered three broad constructs:

• age: terms to identify adolescents or young people

• offenders: terms to identify people who had offended or who were in contact with the criminal justice system

• mental health: terms to identify the range of mental health outcomes examined in the review.

Search terms for these three constructs were combined using the Boolean ‘AND’.

Our decision to use a single comprehensive search based on these three broad constructs had the advantage that the search was not reliant on specific terms for a particular phase, which may have limited sensitivity. For example, an alternative strategy for the diagnostic test accuracy phase would be to use methodological filters to identify test accuracy studies using terms such as ‘sensitivity’ and ‘specificity’. However, there is evidence that the inclusion of such filters in searches for such studies can lead to relevant studies being missed. 31 Another strategy would have been to list as search terms some of the more commonly used screening measures in practice and research in this area. However, this would have predetermined the type of screening measures that would be identified by the review and may have missed studies of other relevant screening measures.

The final set of search terms was developed through an iterative process. A series of pilot searches were run and the results examined and discussed by members of the research team. We considered the likely sensitivity of the search terms by establishing whether or not key citations that we knew were likely to meet inclusion criteria were retrieved by the search.

The searches were not limited by date range or language.

Databases and resources

A range of databases and resources was searched, including standard databases of predominantly peer-reviewed publications as well as resources for the identification of grey literature. The focus of the review spans the mental health literature and the criminal justice literature. We therefore specifically sought to examine databases that covered health and mental health as well as crime and social care. The following databases and resources were searched:

• PsycINFO

• MEDLINE

• EMBASE

• Cochrane Database of Systematic Reviews (CDSR)

• Database of Abstracts of Reviews of Effects (DARE)

• Cochrane Central Register of Controlled Trials (CENTRAL)

• Health Technology Assessment (HTA) database

• NHS Economic Evaluation Database (NHS EED)

• Applied Social Sciences Index and Abstracts (ASSIA)

• Criminal Justice Abstracts

• National Criminal Justice Reference Service (NCJRS)

• Social Policy & Practice

• Social Services Abstracts

• Public Affairs Information Service (PAIS) International

• Science Citation Index (SCI)

• Social Science Citation Index (SSCI)

• Conference Proceedings Citation Index – Science (CPCI-S)

• Conference Proceedings Citation Index – Social Science & Humanities (CPCI-SSH)

• Social Care Online

• The Campbell Library

• Health Economic Evaluations Database (HEED)

• OAIster

• Index to THESES

• Zetoc

• Research Papers in Economics (RePEc).

The following organisation websites and conference proceedings were also searched:

• Department of Health (www.dh.gov.uk/)

• Department for Education (www.education.gov.uk/)

• Home Office (www.homeoffice.gov.uk/)

• Joseph Rowntree Foundation (www.jrf.org.uk/)

• Royal College of Psychiatrists (www.rcpsych.ac.uk/)

• Youth Justice Board (www.yjb.gov.uk/)

• Policy Studies Institute (www.psi.org.uk/)

• Mental Health Foundation (www.mentalhealth.org.uk/)

• Young Minds (www.youngminds.org.uk/)

• Nacro (www.nacro.org.uk/)

• Revolving Doors (www.revolving-doors.org.uk/home/)

• Prison Reform Trust (www.prisonreformtrust.org.uk)

• Centre for Mental Health (www.centreformentalhealth.org.uk/index.aspx)

• British Society of Criminology (www.britsoccrim.org/)

• American Society of Criminology (www.asc41.com/).

Searches were conducted in April 2011. Full details of the specific search strategies for PsycINFO, MEDLINE and EMBASE are given in Appendix 3.

Additional search strategies

In addition to the searches of databases and other resources, we used three additional methods to identify relevant citations:

• Reverse citation search: we undertook reverse citation searches on all included papers using the Web of Science (WoS) Institute of Scientific Information (ISI) citation database.

• Manual check of reference lists: we conducted a manual check of the reference list of all included studies and previous major relevant reviews.

• Contact with experts: we contacted experts in the field to identify other potentially relevant papers and to request further information about included studies when necessary.

Deduplication

The number of databases searched and the use of several search strategies meant that some degree of duplication occurred. To manage this, the titles and abstracts of bibliographic records were downloaded and imported into EndNote X5 bibliographic management software (Thomson Reuters, CA, USA) and duplicate records were removed.

Screening of citations

Two reviewers screened the titles and abstracts identified in the literature search for studies that were potentially eligible for any phase of the review. Full papers of potentially eligible studies were obtained and assessed for inclusion independently by two reviewers. At both stages (first sift – titles and abstracts; second sift – full papers), disagreements were resolved by consensus or deferred to a third party if necessary.

Inclusion and exclusion criteria

We developed detailed separate PICO criteria (population/patient problem, intervention, comparison, outcome) for the different phases of the review; these are summarised in each of the relevant review chapters. Guidance was given to coders to be inclusive at the first sift (titles and abstracts) if there was any uncertainty about a citation but to apply the PICO criteria rigorously at the second sift (full papers).

Overview of the literature search

Figure 1 summarises the literature searching process. The figure given for papers identified outside of the database searches includes papers identified by website searching as well as papers identified from other sources (e.g. contact with experts). The reasons for exclusion for the studies that passed the first sift (title and abstracts) but which did not meet second sift (full paper) inclusion criteria are given in Appendix 4.

FIGURE 1.

Summary of the literature search.

Methods to assess diagnostic test accuracy

As described in Chapter 2, sensitivity and specificity are central concepts in understanding diagnostic test accuracy and are described in detail in Appendix 1, along with further information on methods of quantifying diagnostic performance.

Assessing the validity of mental health needs measures

In recognition of the argument that the presence of a diagnosis does not necessarily equate with the level of need in young people who offend,28 we reviewed studies of screening measures designed to establish the presence of a mental health need. For these types of studies it is not possible to apply the standard strategies of assessing diagnostic accuracy because there is no gold standard of ‘mental health need’ against which the identification of the mental health need screening instrument can be assessed. It is impossible, therefore, to create a 2 × 2 table and summarise the performance of the screening instrument in terms of characteristics such as sensitivity and specificity.

For these studies we assessed the extent to which the assessments of mental health need had established criterion-related validity. Validity refers to the extent to which an instrument measures what it is intended to measure. 32 As applied to the question here, validity refers to whether or not a measure of mental health need in young people who offend does in fact measure the mental health needs of this group. Criterion-related validity assesses the validity of a measure by examining the extent to which it relates in ways we would expect it to relate to other measures of the same or different constructs. For example, if a mental health needs assessment is in fact a valid measure, we would expect it to relate to other indicators of mental health need, such as subsequent use of mental health services.

Rather than exclude all studies of mental health needs assessments that did not report the agreement of the measure against a gold standard diagnosis, for instruments for which we could not identify diagnostic test accuracy data we sought to include reports of validation studies that established the criterion-related validity of the mental health needs assessments.

Methods

This first phase of the review sought to answer two main questions:

1. For those screening measures reporting diagnostic status, what is their diagnostic accuracy?

2. For those screening measures identifying level of need, what evidence is there that these measures are valid indicators of mental health need?

In addition, we summarised the prevalence of mental health problems as identified by the screening instruments in these studies.

Results

A total of nine studies including eight independent samples met our inclusion criteria. 35–43 Two of the included studies38,43 reported data on samples that had some although not complete overlap with each other. The smaller of the two studies38 reported additional details of diagnostic accuracy not reported in the larger study. 43 Specifically, Hayes et al. 38 reported data on the performance of a voice-administered MAYSI-2, whereas the larger study by Wasserman et al. 43 study reported data on a paper and pencil version alone. We therefore report the results of both studies, the larger study because of its greater size and the smaller study because of the additional information it contains on the performance of the voice-administered version of the MAYSI-2. An additional citation44 provided a summary of the results of the included Wasserman et al. study. 43 All of the information contained in it was also included in the original report and so this citation was excluded. A further citation45 reported data on a subset of a sample reported in the included Kerig et al. study. 39 It did not contain additional information on diagnostic test accuracy and so was also excluded in favour of the larger data set reported in Kerig et al. 39

Eight of the nine studies reported data on the diagnostic test accuracy of one or more screening instrument. 35,36,38–43 The remaining study reported data on the validity of a mental health needs assessment, which will be discussed separately. 37

Diagnostic test accuracy results

Quality assessment of the included studies

Table 2 summarises the risk of bias individually for the eight included studies according to QUADAS-2 criteria and Table 3 summarises the applicability criteria individually for the eight studies. Figures 2 and 3 provide an overall summary of the risk of bias and applicability respectively.

TABLE 2 - Quality assessment of the included diagnostic test accuracy studies: risk of bias

✓, criterion met; ✗, criterion not met; ?, unclear if criterion met.

Study	Patient selection: consecutive or random sample	Patient selection: avoided case–control	Patient selection: avoided inappropriate exclusions	Patient selection: overall risk of bias	Index test: index test interpreted blind to reference test	Index test: threshold prespecified	Index test: overall risk of bias	Reference test: reference test correctly classifies target condition	Reference test: reference test interpreted blind to index test	Reference test: overall risk of bias	Flow/timing: interval of ≤ 2 weeks	Flow/timing: all participants receive a reference standard	Flow/timing: all participants receive same reference test	Flow/timing: all participants included in analysis	Flow/timing: overall risk of bias
Cashel 199835	?	✓	✓	Unclear	✓	?	Unclear	✓	✓	Low	?	✓	✓	✓	Unclear
Grubin 200236	?	✓	✓	Unclear	✓	?	Unclear	✓	✓	Low	✓	✗	✓	✓	High
Hayes 200538	✗	✓	✓	High	✓	✓	Low	✓	✓	Low	?	✓	✓	✗	High
Kerig 201139	✓	✓	✓	Low	?	✗	High	✓	?	Unclear	✓	✓	✓	✓	Low
Kuo 200540	✗	✓	✗	High	?	✗	High	✓	?	Unclear	?	✗	✓	✓	High
McReynolds 200741	✓	✓	✓	Low	✓	✓	Low	✓	✓	Low	?	✓	✓	✗	High
Vreugdenhil 200642	?	✓	✗	High	?	✓	Unclear	✓	?	Unclear	?	✗	✓	✗	High
Wasserman 200443	✗	✓	✗	High	✓	✓	Low	✓	✓	Low	?	✓	✓	✗	High

TABLE 3 - Quality assessment of the included diagnostic test accuracy studies: applicability criteria

High, high level of concern about applicability; low, low level of concern about applicability.

Study	Patient selection: applicability	Index test: applicability	Reference test: applicability
Cashel 199835	Low	High	Low
Grubin 200236	High	Low	Low
Hayes 200538	Low	Low	Low
Kerig 201139	Low	Low	Low
Kuo 200540	Low	Low	Low
McReynolds 200741	Low	Low	Low
Vreugdenhil 200642	Low	Low	Low
Wasserman 200443	Low	Low	Low

FIGURE 2.

Overall risk of bias across QUADAS-2 domains for the included diagnostic test accuracy studies (n = 8).

FIGURE 3.

The QUADAS-2 applicability criteria for the included diagnostic test accuracy studies (n = 8).

Patient selection

The patient selection domain assesses if the way in which participants were selected may have introduced a bias. Four studies, consisting of three independent samples, were rated as being at high risk of bias for this domain;38,40,42,43 the risk of bias was rated as low for two studies39,41 and unclear for the remaining two studies. 35,36

Although all studies avoided a case–control design, some studies did not use random or consecutive sampling for recruiting participants and others were judged to have a high number of inappropriate exclusions. The absence of random or consecutive sampling could artificially either increase or decrease the observed performance of a screening instrument against a gold standard; the direction of the influence would be determined by the exact nature of the sampling procedure used. The same is true of the high number of inappropriate exclusions from the sample. Therefore, although there is some evidence of bias for the patient selection domain, it is unclear what effect this had on the observed diagnostic accuracy in the included studies.

Index test

The index test domain asks whether the conduct or interpretation of the screening test may have introduced a bias. The overall risk of bias for this domain was rated as high for two studies,39,40 unclear for three studies,35,36,42 and low for three studies, consisting of two independent samples. 38,41,43

For some studies it was unclear if the index test was interpreted blind to the reference standard. Blinding is essential to ensure that knowledge of the results does not influence the scoring of the reference standard, which may artificially inflate the observed diagnostic test accuracy of the screening test. Some studies also failed to use a prespecified cut-off point on the index test. The post hoc selection of the cut-off point can capitalise on a chance finding and artificially inflate observed diagnostic test accuracy.

Reference standard

The reference standard domain assesses whether the gold standard used or the conduct or interpretation of the gold standard test may have introduced bias. The overall risk of bias for the reference standard domain was considered low for five studies,35,36,38,41,43 consisting of four independent samples, and unclear for three studies. 39,40,42 The unclear ratings resulted from a lack of clear evidence that the diagnostic gold standard was conducted blind to the results of the index test. As with lack of blinding for the index test, this can also distort the observed diagnostic performance of the screening test.

Flow and timing

Six out of the eight studies,36,38,40–43 consisting of five independent samples, were rated as being at high risk of bias in terms of the flow and timing domain, which assesses whether or not the participant flow through a study and the timing of measurement may have introduced bias. The reasons for the rating of high risk for many of the studies were that not all participants received the reference standard and not all participants were included in the analysis. Participants included in the diagnostic test accuracy analysis may have differed in systematic ways from participants who were not included and this may distort the test accuracy.

Applicability criteria

Table 3 summarises by individual study the extent to which the QUADAS-2 applicability criteria are met and Figure 3 provides an overall summary. The applicability criteria were broadly met for the patient selection and index test domains and entirely met for the reference standard domain. One study did not meet the criterion for index test applicability. 35 As we describe earlier, this was because the study used the MMPI-A, a 458-item measure taking approximately 90 minutes to complete, which makes it unsuitable as a screening measure, although one or more subscales could feasibly be used to screen. One study did not meet the applicability criterion for patient selection. 36 This was because the study recruited from a variety of adult and young offender institutions, although it was possible to extract some data for the young offender population and the results discussed later for that study are based solely on the young offender subgroup.

Summary

With the exception of the reference standard domain, for which a majority of studies had a low risk of bias, the risk of bias was either high or unclear for the majority of studies in the other three QUADAS-2 domains. No study was rated as being at low risk for all four domains. In contrast, applicability criteria were broadly met across all studies. The identified studies are therefore largely relevant to the diagnostic test accuracy question that this review seeks to answer but some caution is needed in relying on the diagnostic accuracy data reported by these studies because the level of potential bias across many QUADAS-2 domains was often unclear or high.

Validity of the mental health needs assessment results

Some screening measures for mental health problems in young people who offend do not provide sensitivity and specificity against a gold standard diagnosis; instead, they aim to identify a ‘mental health need’. For any screening measure described as a mental health needs assessment and for which there were no diagnostic test accuracy data available, we sought validation studies of that screen as a measure of mental health need. As described in more detail earlier, for inclusion studies had to provide evidence of some form of criterion-related validity for a mental health need.

Although there are a number of screening measures that are designed to identify mental health needs in young people who offend, including a number developed in the UK, we identified only one study that met inclusion criteria. 37

Haapanen and Steiner37 assessed the performance of a battery of tools termed the Mental Health and Substance Abuse Treatment Needs Assessment. Although this battery of tests includes some measures for which there exist diagnostic test accuracy data, such as the MAYSI-2, we included this study because it was the entire battery of tests that was designed to assess mental health needs. The full battery consisted of the Achenbach Child Behavior Checklist – YSR, the MAYSI-2, the Weinberger Adjustment Inventory and the Drug Experience Questionnaire. Paper and pencil versions of the test were used.

The sample consisted of 836 young people who were all committed to the California Youth Authority, which deals with young people who have committed very serious crimes, who have a substantial criminal history or who have failed at local interventions. In total, 79.4% of the sample was male. The average age of the sample was described as 16–17 years and the ethnicity of the sample was described as predominantly Hispanic or African American. Further details about ethnicity were not given. Validation of this combined mental health assessment was based on a case-note review, which was used to establish whether the young people were offered mental health treatment, were offered psychopharmacological treatments or were identified as requiring treatment but treatment was not provided. The level of reporting of the results is limited and the results are largely descriptive. In general, however, the authors state that elevated scores on instruments such as the MAYSI-2 and YSR were related to an increased use of mental health services or need for such services, at least for the male sample.

Although the search identified a number of studies reporting data on measures used as part of the Asset screening pathway in the UK [e.g. Asset, Screening Questionnaire Interview for Adolescents (SQiFA), SIfA], none of the studies met our inclusion criteria for establishing the criterion validity of the screening measure against other measures of mental health need. For example, validity data are reported for the Asset instrument54 but, as this instrument is designed primarily to identify the factors contributing to a young person’s offending, the validation was against indices of reoffending rather than mental health need.

Another report described a number of studies, one of which examined the use of the Salford Needs Assessment Scale for Adolescents (SNASA) in 301 young people who had offended and also interviewed the case managers of the sample to enquire about needs. 4 However, after discussion we concluded that this comparator – the views of case managers – did not constitute adequate evidence of criterion-related validity. A further citation reported data on the reliability of the SNASA but did not report data directly relevant to assessing criterion-related validity. 55

It should be recognised, however, that our search strategy may have missed important studies in this respect. For a study to be identified, it had to mention the measure in the title or abstract along with information about validity. It is possible that studies may contain information relevant to establishing the validity of a measure without the measure being referred to in the title and abstract. For example, studies that used one of the mental health needs assessments as the outcome measure in a trial would provide evidence relevant to establishing criterion-related validity, but such studies would not necessarily be identified as part of the search if the measure was not referred to in the abstract. For example, the SNASA was used as an outcome measure in a trial identified as part of the clinical effectiveness review. 56 Full details of this study are given in Chapter 6. In brief, the SNASA was used in a trial of cognitive–behavioural therapy (CBT) compared with treatment as usual, with the intervention designed to improve a range of mental health outcomes, including depression and anxiety symptoms. Outcome was assessed at 11 months’ follow-up. The two groups appeared to be broadly comparable at follow-up on the SNASA [CBT group (n = 18): mean 10.5, standard deviation (SD) 3.54)]; treatment as usual group (n = 20): mean 10.75, SD 4.0]. Evidence for criterion-related validity would have required lower scores at follow-up in the CBT group. However, the absence of such a relationship and its implications for understanding the validity of the SNASA should be treated with caution given the small sample size.

Summary

There were too few studies to make any firm conclusions about the diagnostic test accuracy of any of the screening measures examined in this review. Ideally, a conclusion about a particular screening measure would require a large number of studies reporting diagnostic performance at a range of cut-off points. This would allow the calculation of pooled estimates of sensitivity, specificity and other indices of test accuracy. It would also allow the examination of potential sources of heterogeneity in observed test accuracy across studies.

It is also difficult to draw conclusions about the comparative performance of different screening methods, even though some studies examined the performance of more than one measure in the same sample. Studies did not typically compare the performance of the measures across the full range of potential cut-off points. In some cases the reported performance of one measure had a particular balance between sensitivity and specificity whereas another measure had a different balance. It is unclear how the two measures would compare at cut-off points that attempted to give broadly the same balance (e.g. reasonably high sensitivity combined with acceptable specificity).

As discussed in Chapter 2, in which the decision problem was outlined, one of the aims of screening may be to detect previously unidentified cases. The diagnostic test accuracy data reported by the studies included in this review did not appear to differentiate between previously identified and unidentified cases. It is unclear, therefore, how the reported accuracy would be altered if the analysis was restricted to previously unidentified cases.

Any conclusions about the diagnostic performance of screening measures in populations of young people who have offended are, therefore, necessarily tentative. One potential conclusion is that there is no clear evidence that the MAYSI-2, a test specifically designed for use in this population, has superior operating characteristics to those of other general measures. More generally, the reported literature standard cut-off points for the MAYSI-2 and other instruments suggest a combination of both moderate sensitivity and moderate specificity, and so altering the cut-off point to increase sensitivity may lead to unacceptably low specificity.

One of the objectives of the review was to establish in which groups of young offenders screening may be of use. There were too few studies to make any firm conclusions about such groups, for example community compared with incarcerated settings or particular diagnostic subgroups. However, data were identified on screening accuracy for some mental health problems, including depression, PTSD, ADHD, conduct disorder and ODD. These disorders are therefore candidates for use as exemplars in the decision model.

Reflections on policy and practice

Although current UK policy recommends screening for mental health problems in young people who offend, there is currently a limited evidence base examining the diagnostic test accuracy of available screening measures.

There also appears to be limited validation of mental health needs assessments. Although this information would be useful, it is not immediately clear how it could be used to inform decision-making in terms of the clinical effectiveness and cost-effectiveness of a screening strategy, because this typically requires sufficient data to calculate sensitivity and specificity against a gold standard.

The majority of the diagnostic test accuracy studies were conducted in the USA; only one was conducted in the UK. Furthermore, many of the studies were conducted in incarcerated settings. In the UK, in contrast, most young people who offend are managed in the community. The existing literature – already limited in terms of the number and quality of studies – may therefore be further limited by the extent to which the findings can be generalised to UK settings.

Overview of screening study designs

There are a number of potentially relevant screening designs. In the simplest design, participants are allocated to either a screening condition or a no-screening condition, in which a formal screening method is used to establish whether or not someone meets predetermined criteria for having a particular mental health problem. The impact of the screening intervention on relevant mental health outcomes is then assessed. Other designs link the results of the screening to some form of enhancement of care. For example, participants could be assigned to a screening or a no-screening condition and those people in the screening condition who score above a clinical cut-off point could then be offered some form of enhanced care (e.g. further assessment followed, if necessary, by a pharmacological or psychological treatment). All designs were considered relevant and we sought to identify any trials that had used these approaches in a young offender population.

Method

In this phase of the review we sought to answer the following question: ‘What is the clinical effectiveness of screening strategies for mental health problems in young people who offend?’

The search strategy outlined in Chapter 3 was used to identify relevant studies.

Results

Despite an extensive search involving a large number of databases and additional searches of other resources, resulting in the examination of > 13,000 citations, we identified no studies that met the PICO criteria.

Summary

In the absence of evidence of the clinical effectiveness of screening strategies for mental health outcomes we undertook a further review, described in the next chapter, in which we sought to identify any evidence of the clinical effectiveness of interventions for mental health difficulties.

Reflections on policy and practice

Screening for mental health problems is currently recommended for young people who have offended. However, to date there are no trials examining the effectiveness of such a strategy.

Method

In this phase of the review we sought to answer the following research question: ‘What is the clinical effectiveness of interventions for mental health problems for young people who have offended?’

Results

Summary

There were too few studies to make firm conclusions about the clinical effectiveness of interventions for mental health problems in young people who offend. No conclusions can be made about the effectiveness of psychopharmacological interventions because no studies were identified that met our inclusion criteria. Any conclusions about psychological interventions are also tentative given the limited number of studies identified. Those studies that did meet the inclusion criteria tended to be small and many had methodological problems or key methodological features were poorly reported. There was an insufficient number of studies using broadly comparable interventions for broadly comparable psychological difficulties to conduct a meta-analysis. This would have compensated for the small size of many of the studies and would have allowed an assessment of the extent to which variations in effect sizes were associated in a systematic way with variations in the methodological quality of the primary studies.

Many of the interventions used in the studies had a broad focus and a clearly identified primary outcome was not specified. Instead, results for a range of outcomes were reported. Clear specification of the primary outcome is important to protect against post hoc selective reporting and may be particularly important for studies in this area with a general treatment focus, such as improving coping, for which there may not be an obvious primary outcome.

Other methodological limitations or poor reporting of methodological features were also present in a number of studies (e.g. lack of allocation concealment, lack of blinding of outcome assessment). The general effect of these would be to artificially inflate the observed clinical effectiveness. Although some studies did report broadly encouraging findings, caution is therefore needed. This conclusion is in keeping with that of an early systematic review in this area. 18 The current and previous reviews used somewhat different inclusion criteria. Each review identified 10 studies, of which six were common to both. Although Townsend et al. 18 performed a meta-analysis of three group CBT studies that had depression as an outcome, we chose not to conduct a meta-analysis. The meta-analysis of Townsend et al. 18 combined studies that had depression as a primary treatment focus64 with other studies that had a distinct focus, such as the Ahrens and Rexford59 trial in which the treatment focus was for PTSD symptoms. We judged that it was not meaningful to combine studies in which depression was a primary focus with those in which it was a secondary outcome.

One of the objectives of the review was to identify in which groups it may be of use to screen and offer interventions. There were too few studies to establish the presentations for which screening and offering interventions may be of use and too few studies of both community and incarcerated settings to make conclusions in terms of these populations. However, the review did identify trials of depression, PTSD and conduct disorder. Studies of diagnostic test accuracy were also identified for these three presentations. This suggests that these may be potential candidates for the development of an exemplar decision model, as discussed in Chapter 8. The clinical effectiveness data, however, suggest that depression may be the most suitable exemplar. The one depression trial,64 a study of a CBT package for depression, was one of only two studies to be rated as being at low risk of bias across a number of the quality assessment domains. In contrast, the anxiety study59 was rated as being at either unclear or high risk of bias across a number of domains. Of the two studies examining conduct disorder as an outcome, one61 was rated as being at high or unclear risk of bias across all quality assessment domains. In addition, this two-arm trial compared two forms of CBT against each; it did not include a usual care comparator, which would be of use in the modelling phase. For the second study it was not possible to extract relevant estimates of effect sizes. 66 As will be discussed in more detail in Chapter 8, any decision model based on the available data will be at best an illustrative example of the type of modelling that could be undertaken in the area in future when a larger number of higher-quality studies is available.

Reflections on policy and practice

As with the results for the diagnostic test accuracy studies, there may be concerns about the extent to which the findings on clinical effectiveness can be generalised to a UK setting. Many of the studies were conduced in the USA and a number were conduced in incarcerated settings. As described in Chapter 1, the majority of young people who offend in the UK are managed in the community. Two UK studies were included in the review; however, these were both conducted in custodial settings.

Although screening is currently recommended for the identification of mental health problems in young people who offend, screening strategies assume that there are clinically effective treatments that can be offered to people who are identified as requiring an intervention. Although some positive results were reported in the identified trials, there remains substantial uncertainty around the clinical effectiveness of interventions for mental health problems in this population.

Methods

In this phase of the review we sought to answer the following questions:

• What is the cost-effectiveness of screening strategies for mental health difficulties among young people who have offended?

• What is the cost-effectiveness of interventions for mental health difficulties in young people who have offended?

The search strategy outlined in Chapter 3, which included searches of NHS EED and HEED, was used to identify relevant studies.

Results

No studies were identified that met the inclusion criteria for this stage of the review.

Summary

Despite significant discussion about and policy recommendations on strategies to detect mental health needs among young people who offend, the review identified no studies that were full economic evaluations of identification strategies or clinical interventions for mental health problems among young people who have offended.

Reflections on policy and practice

Although there are current policy guidelines recommending screening and on the basis of this intervening for mental health problems in young people who offend, no studies addressing the cost-effectiveness of these policies were identified.

Setting the decision context

Detection and treatment of mental health conditions in young offenders is, by definition, an intersectoral issue. When developing a decision-analytic model it is important to establish the context for informing resource allocation decisions.

The decision analysis primarily considers costs and health outcomes [expressed in quality-adjusted life-years (QALYs)] from the perspective of the UK health services. Current policy stipulates that screening occurs at the initial contact with the criminal justice system or at the first available opportunity, whereas the treatment pathway is expected to follow based on the outcome of screening.

Adopting this conventional health service-only perspective could infer certain limitations as the costs and benefits of treating mental health issues in young offenders may go beyond the health-care system. To capture further intersectoral effects, supplemental analysis extends the perspective to consider costs and benefits that may be realised by the youth justice system in terms of future crimes averted as a result of treating depression in young offenders.

Methods

The objective of the exemplar model was to estimate, based on best available data, the costs and health outcomes for a range of feasible identification strategies. As already explained, the analysis is primarily set to the conventional health services perspective and this initial perspective is extended to also consider the youth justice system perspective. All costs are expressed in present-day values (2013) and health outcomes are expressed in QALYs. The time horizon for the analysis of depression is 1 year; hence, no discounting of costs or benefits was applied.

The model was made up of two parts: (1) an identification model, reflecting the diagnostic performance and administration costs of the alternative identification strategies, and (2) a treatment model that evaluated the subsequent costs and outcomes (expressed in QALYs). To consider intersectoral implications, the effect of treatment on recidivism rates was incorporated as a cost offset against the cost of the identification strategy.

For an identification strategy to be cost-effective, it is important that a cost-effective treatment strategy is available. Without a known cost-effective treatment, identification of an effective screening strategy would not be useful for the decision-maker because the identified patients would not be offered a cost-effective treatment. Hence, a treatment model is required to evaluate the cost-effectiveness of identification strategies in terms of the health benefits of identifying mental health conditions. Given the importance of the subsequent treatment, the treatment model for young offenders with depression is considered first.

Table 20 summarises the stages of the analysis, detailing sequentially the screening strategies evaluated and the perspectives adopted.

The identification model

The identification model assumed a decision tree structure to capture the outcomes of varying the sensitivity and specificity of the diagnostic strategies. Four possible outcomes are considered for each diagnostic strategy and relevant costs and outcomes evaluated for: (1) true positive, (2) false negative, (3) true negative and (4) false positive. The identification model is driven by the prevalence of depression, the sensitivity and specificity of specific diagnostic tools and the cost associated with each strategy.

Strategies evaluated

The decision problem would ideally compare all potential identification strategies that are feasible to implement within the youth justice system. However, in reality, evaluation of these options has been constrained by the availability of evidence. Chapter 4 was used to inform the identification strategies considered in the economic analysis and only studies containing sufficient data were used to form the basis of the parameters included.

The base-case analysis considered single-stage screening followed by treatment or no treatment for depression based on the outcome of screening. In addition to the base-case analysis, separate scenarios were considered that explored a range of alternative strategies (discussed in more detail subsequently). The alternative approaches considered included (1) the effects of two-stage screening in which the second stage uses a gold standard confirmatory interview; (2) the impact of diagnostic accuracy on recidivism; and (3) varying estimates of input parameters in the model (such as prevalence).

The diagnostic component

Model structure and key assumptions

To consider the implications of screening populations entering the youth justice system and the potential gains for those specifically with depression, Figure 4 illustrates the outcomes of screening and treatment and describes the associated costs and outcomes considered.

FIGURE 4.

Schematic of the identification model and the implied treatment outcomes.

Within this screening framework, all young offenders (depressed or not) entering the criminal justice system are screened on first contact with the system (as per guidelines under CHAT). For the purpose of the current analysis it is assumed that mental health need will be unknown up until the point of screening.

Each individual entering the system would receive an intervention strategy incurring the cost of the related screen. Here we consider the time taken by the health-care professional (within the criminal justice system) to conduct the screening as the main cost element.

The Offender Health Research Network indicates that reception health screening is typically carried out by nurses. 91 To apply the specific unit cost to the required time for screening, a value of £24 per hour for ‘Prisons: Nurse (mental health)’ taken from Brookes et al. 87 was used. The specific cost parameter for each individual diagnostic tool evaluated is presented in Table 21.

TABLE 21 - Screening tools, administration time and associated costs

Screening tool	Administration time (minutes)	Cost (£)	Source
MAYSI-2 (paper)	10	4	Wasserman et al.43
MMPI-A	90	36	Cashel et al.35
MAYSI-2 (paper)	10	4	Hayes et al.38
MAYSI-2 (voice)	10	4
DPS	15	6	McReynolds et al.41
MFQ	6	2.4	Kuo et al.40
Short MFQ	2.5	1
MAYSI-2 (paper)	10	4

Of the total screened, those expected to screen positive (either true positive or false positive) receive the intervention for depression (e.g. group CBT). Dependent on the distribution across the four diagnostic outcomes, the expected number of QALYs and the cost were calculated. For individuals who screened as false negative (i.e. screened negative despite depression), the mental health need is unmet and the risk of reoffending remains high.

To inform the expected prevalence of depression within the youth justice system, we used data from Fazel et al. ,2 who determined prevalence rates of mental disorders (including depression) among adolescents in juvenile detention using a summary of 25 surveys. The results from this review were extracted and presented by gender, indicating that the depression rate in males is 11% (95% CI 7% to 14%) and that in females is 29% (95% CI 22% to 37%).

Sensitivity analysis

Sensitivity analysis explored the impact of considering alternative values for three key drivers in the model: (1) prevalence of depression under usual care; (2) utility weights assigned to DFDs; and (3) the effect of CBT on recidivism.

Results

The results are presented in two parts: (1) the primary (base-case) results including the outputs of the base-case model (as outlined above) and (2) sensitivity analysis evaluating the impact of varying input parameters on the cost-effectiveness analysis. Both sets of results are presented using two perspectives, that is, the health services perspective and an intersectoral perspective. The intersectoral analysis estimates the cost offset to the youth justice system through reduced criminal activity as a result of identifying and treating depressed offenders.

Primary results

The analysis was conducted for single-stage screening and two-stage screening strategies. Single-stage screening may use the gold standard approach (i.e. DISC with a sensitivity and specificity of 1) or may use a relatively imperfect but less resource-intensive screening tool with a sensitivity and specificity of < 1 (such as MAYSI-2). Single-stage screening is followed by a treatment decision based on the outcome of screening. The two-stage screening strategy involves the administration of a gold standard instrument on individuals who screened positive (both true and false positives) on the single-stage screening measure (as expected, two-stage screening does not apply to the case when the gold standard tool is used as the first screening tool).

Cost-effectiveness of using single-stage screening tools from a health-care perspective

Table 22 presents the cost-effectiveness estimates of single-stage screening compared with usual practice. Usual practice for the purposes of the model is initially simulated as no active detection (i.e. no formal screening employed). Note that, in terms of treatment costs, the primary analysis assumed that one therapist would be present during the group CBT sessions.

TABLE 22 - Cost-effectiveness of single-stage detection strategies to inform the treatment decision (health-care perspective)

NA, not applicable.

	Gold standard	MAYSI-2				MFQ	Short MFQ	DPS	MMPI-A
		Paper version	Paper version	Voice version	Paper version
Study (no. of screening tests in the study)	e.g. DISC (NA)	Wasserman 200443	Hayes 200538 (1)	Hayes 200538 (2)	Kuo 200540 (3)	Kuo 200540 (1)	Kuo 200540 (2)	McReynolds 200741	Cashel 199835
Cost per QALY (single screening tools) (£)	43,195	77,378	83,156	61,844	71,804	42,839	52,174	52,322	78,662

In general, Table 22 shows that, compared with current practice (i.e. no active screening), single-screening strategies identified in the systematic review are not likely to be cost-effective based on the commonly used WTP threshold in the UK for an additional unit of health outcome (i.e. £20,000–30,000 per QALY).

As emphasised earlier, there is limited evidence on the effectiveness and cost-effectiveness of treatments for depression in young offenders, which suggests that, even if screening had no cost and had a sensitivity and specificity of 1 (which is unrealistic), treating the true positives with group CBT would cost £17,542 per QALY (based on a health services perspective). Therefore, introducing screening costs would only increase the cost per QALY because of the additional costs of screening and the loss of false-negative cases who would have benefited from treatment. Hence, it is not surprising that none of the single screening strategies was found to be cost-effective. This further reinforces that, for screening to be cost-effective, it is important to have a treatment strategy that produces a significant improvement in outcomes at a relatively low cost.

Cost-effectiveness of two-stage screening strategies from a health-care perspective

Use of single screening tools alone is not cost-effective as the imperfect accuracy of each tool implies that false positives receive unnecessary treatment, increasing the overall cost, and false negatives miss out on the treatment and potential gains in outcome. Although the second stage (the gold standard) in the two-stage screening process does not deal with the issue of false negatives, it differentiates the true positives from the false positives, ensuring that treatment is provided only to those who are truly depressed.

Table 23 presents the comparison between this two-stage strategy and usual practice. The cost per QALY analysis suggests a significant efficiency gain over the use of single screening tools alone. However, only the MFQ and Short MFQ (disease-specific tools for depression) are potentially within the upper bound of the WTP threshold of £20,000–30,000 per QALY.

TABLE 23 - Cost-effectiveness of two-stage screening strategies to inform the treatment decision (health-care perspective)

NA, not applicable.

Here, the gold standard (e.g. DISC) is a single-stage screen and is included in the table to provide the reference standard.

	Gold standarda	MAYSI-2				MFQ	Short MFQ	DPS	MMPI-A
		Paper version	Paper version	Voice version	Paper version
Study (no. of screening tests in the study)	e.g. DISC (NA)	Wasserman 200443	Hayes 200538 (1)	Hayes 200538 (2)	Kuo 200540 (3)	Kuo 200540 (1)	Kuo 200540 (2)	McReynolds 200741	Cashel 199835
Cost per QALY (two-stage strategy) (£)	43,195	36,632	38,185	33,329	35,521	28,278	29,118	33,915	55,910

It should be noted, however, that the analyses of both single and two-stage screening assume that there is no cost associated with false negatives. This assumption may not hold in practice; however, we did not find any evidence on health service resource use because of untreated depression in the young offending population.

Cost-effectiveness of two-stage screening strategies from an intersectoral perspective

The decision to treat a mental health need in young offenders may have wider-reaching benefits than the health outcomes alone. In adopting the intersectoral perspective of the health and youth justice services, the aim is to examine the extent of the potential costs and benefits of various detection strategies; this would be relevant if an improved mental health status had consequences for future criminal behaviour (specifically, changes in recidivism rates).

Table 24 presents the results of the cost-effectiveness analysis including the expected cost offset from the potential reductions in recidivism rates. Given that only the two-stage detection strategies were found to be cost-effective within the health-care perspective, only these strategies were analysed from an intersectoral perspective.

TABLE 24 - Cost-effectiveness of two-stage screening strategies with the inclusion of the cost offset from reduced rates of recidivism (intersectoral perspective)

NA, not applicable.

Here, the gold standard (e.g. DISC) is a single-stage screen and is included in the table to provide the reference standard.

	Gold standarda	MAYSI-2				MFQ	Short MFQ	DPS	MMPI-A
		Paper	Paper	Voice	Paper
Study (no. of screening tests in the study)	e.g. DISC (NA)	Wasserman 200443	Hayes 200538 (1)	Hayes 200538 (2)	Kuo 200540 (3)	Kuo 200540 (1)	Kuo 200540 (2)	McReynolds 200741	Cashel 199835
Cost per QALY (two-stage strategy)	32,391	25,828	27,381	22,524	24,716	17,473	18,314	23,111	45,106

Including the expected cost offset from reduced recidivism to the evaluated two-stage detection strategies, all strategies except for MMPI-A fall within the range of the decision-maker’s WTP. This model provides a provisional indication that gains in health status (through appropriate detection and treatment) may be justified by the potential cost offset to the youth justice system. However, it should be noted that the base-case intersectoral analysis gives equal weight to costs incurred or saved by the health and youth justice systems (i.e. the analysis assumes that costs incurred by the health system can be compensated in a 3 : 1 ratio by cost savings in the youth justice system).

To compare strategies and indicate which may represent the most efficient use of resources, results can be presented on a cost-effectiveness frontier. The frontier connects incremental cost-effectiveness ratios (ICERs) of strategies on the cost-effectiveness plane to identify strategies that dominate other less cost-effective strategies. Strategies that lie on the frontier line represent value for money and options falling on the line are compared to indicate whether or not incremental health benefits justify any incremental costs. The incremental analysis compared with no active detection is presented in the cost-effectiveness plane in Figure 5.

FIGURE 5.

Cost-effectiveness plane of a two-stage screen detection strategy (single screen + gold standard) including the cost offset by reductions in recidivism attributed to treatment of depression. DISC Pred., the DISC predictive scales (DPS).

Examining the cost-effectiveness plane suggests that two-stage screening using the MFQ or the Short MFQ in the first stage is most cost-effective, as represented by the cost-effectiveness frontier on the cost-effectiveness plane. The cost-effectiveness frontier represents the most efficient points among all screening strategies examined (i.e. the frontier represents the strategies that for a given level of effect have the highest cost or vice versa).

Although the exemplar approach provides a framework for future cost-effectiveness analyses of screening strategies in young offenders, the analyses presented so far have several limitations, in particular the limited data available with which to construct a model. The following section presents the results of sensitivity analyses to illustrate how sensitive the model is to variation in the input parameters.

Discussion

The cost-effectiveness analysis was conducted within the limitations of the available evidence on the effectiveness of screening and treatment strategies for mental health conditions in young offenders. Because of the limited evidence, we developed an exemplar model for depression to evaluate the cost-effectiveness of single-screening and two-stage screening followed by a treatment decision based on the screening outcome. Depression was chosen for the exemplar analysis for the reasons highlighted in the introduction to this chapter.

The limitations of the data are considerable and the results of the exemplar model must be interpreted within the context of these. This includes limitations in data availability for both screening and treatment parameters. For example, data on treatment were limited to a single study. In addition, the lack of data for a number of additional key input parameters, such as the impact of treatment on the rates of recidivism, limits the implications that can be drawn from the exemplar decision model.

The economic model identified key drivers required for decision analysis, which included the prevalence and severity of the mental health condition, the diagnostic accuracy of the screening instruments, the treatment effect on health outcomes and recidivism and the perspective of the economic analysis. The exemplar analysis demonstrated how strongly these key drivers could influence the cost-effectiveness decision. These key parameters are likely to be influential when further cost-effectiveness analyses are conducted in this population (whether for depression or other mental health conditions); hence, the cost-effectiveness analysis also informs areas of research prioritisation to reduce uncertainty in the current evidence base to allow evaluation of screening strategies in the future.

The economic analysis also highlighted the value of having an effective treatment for any screening strategy to be cost-effective. For the exemplar model, our systematic review found only one relevant study (i.e. Rohde et al. 64) that could be used to derive DFDs and in turn QALYs. The treatment model was based on data from this single study, with a relatively small sample size of only 93 adolescents; moreover, although the recovery functions used to derive DFDs showed small gains after CBT, these functions were not statistically significantly different from each other. Moreover, using point estimates of recovery rates from the Rohde et al. study,64 the treatment model showed that, if all depressed young offenders could be treated with CBT (assuming that they could be identified at no cost), the cost per QALY was £17,542. Given that screening strategies are imperfect and result in unnecessary treatment costs because of false-positive individuals and missing out potential health gains for false-negative individuals, the cost per QALY estimates would only become higher, in turn making screening less cost-effective. This puts the emphasis on finding effective evidence-based treatment strategies that would produce reasonable health benefits after identifying screen-positive individuals. This could be achieved by conducting larger, high-quality clinical trials to identify or develop effective treatment strategies for screen-positive individuals.

The cost-effectiveness analysis also raised interesting methodological issues around the perspective of the economic analysis and the valuing of non-health benefits and costs for health services decision-making. The exemplar model suggests that several screening strategies that were not likely to be cost-effective from the health services perspective (given the available evidence) may become cost-effective when an intersectoral perspective is adopted. Moreover, the consumption value of a QALY used in the analysis was found to be another important determinant in the cost-effectiveness analysis.

The evidence on the sensitivity and specificity of screening instruments used in the exemplar model shows that either most instruments have poor diagnostic ability or there is significant uncertainty around sensitivity and specificity or both. Our analysis suggested that, if an instrument produces a high number of false-positive results, introducing two-stage screening (with the second stage being the gold standard) may be more cost-effective if the second screening cost could offset the cost of incorrectly treating the false positives. Although uncertainty in the available evidence does not allow us to reach definite conclusions, the exemplar model indicates that in the presence of poor diagnostic properties a two-stage screening strategy may be more cost-effective than single-stage screening.

In conclusion, the cost-effectiveness analysis presented here is primarily an exemplar. It provides an insight into the decision problem and identifies key drivers of cost-effectiveness and demonstrates the effects of our level of uncertainty about model parameters. This is of use in informing future research priorities, which will be discussed in Chapter 10. Before these future research priorities are identified, the current evidence base is assessed against UK NSC criteria.

Criterion 5

‘There should be a simple, safe, precise and validated screening tool.’

Chapter 4 examined the diagnostic test accuracy of existing screening tools, which relates to the ‘precision’ component of this criterion. In terms of the ‘validation’ component, Chapter 4 also examined the extent to which validity data were available for those mental health needs assessments that did not report diagnostic test accuracy information.

Although a number of screening measures were examined for a range of mental health problems, there were too few studies of any one screening measure for any one mental health problem to firmly establish the precision of a measure in this population. The MAYSI-210 was the most widely examined of all of the screening instruments but even for this measure there was an insufficient number of studies to conduct a diagnostic meta-analysis. Furthermore, the data for the MAYSI-210 suggested that at the recommended cut-off points the measure had only moderate sensitivity and specificity across a number of mental health domains.

We also found limited validation data for those measures for which diagnostic test accuracy information was not reported. Even were substantial validation information to be reported for these measures, they would still fail to meet UK NSC criterion 5 because it assumes that a screen is both valid and precise. Mental health needs assessments that do not provide information on diagnostic test accuracy would necessarily fail this criterion, because information on precision is not available.

Criterion met? No.

Criterion 6

‘The distribution of test values in the target population should be known and a suitable cut-off level defined and agreed.’

Studies typically reported the diagnostic performance of a particular screening measure at a limited range of cut-off points, so the distribution of test values in the target population is not known. However, studies did frequently report sensitivity and specificity at recommended or standard cut-off points commonly cited in the literature. For use as a screening measure, high sensitivity is typically protected at the expense of somewhat lowered specificity, although, as described in Chapter 2, a balance often needs to be struck between high sensitivity and moderate specificity. Often the sensitivity at these recommended cut-off points was lower than that which is typically required for use as a screening instrument. This did appear to be the case for the MAYSI-2,10 for example. However, simply altering the cut-off point to increase sensitivity may not be possible given that, as described above, even at moderate levels of sensitivity the observed specificity was also moderate. Altering the cut-off point to increase sensitivity would inevitably further reduce specificity and this may lead to an unacceptably high false-positive rate. For the MAYSI-2 this may make it difficult to establish an agreed cut-off level, although any conclusions about the MAYSI-2 are limited by the small number of studies examining its performance for the same diagnostic category. For other instruments, for which the evidence base is even more limited, this conclusion also holds.

Criterion met? No.

Criterion 10

‘There should be an effective treatment or intervention for patients identified through early detection, with evidence of early treatment leading to better outcomes than late treatment.’

In Chapter 6 a small number of studies were identified that examined the clinical effectiveness of psychological treatments for a range of mental health difficulties in young people who offend. No studies were identified that examined the effectiveness of psychopharmacological interventions.

Although some studies did report some positive findings about the potential benefits of psychological interventions in this population, caution is needed in interpreting these results. First, there were too few studies of any one intervention for any one mental health difficulty to make firm conclusions about the likely effectiveness of these interventions. Second, the quality assessment of the studies suggested that many had either a high risk of bias or an unclear risk of bias for a number of key methodological features. For example, allocation concealment, the absence of which is empirically associated with increased effect sizes,69 was rated as unclear or at high risk of bias in all but one of the 10 studies reviewed. Blinding of outcome assessment, the absence of which is also likely to be associated with inflated effect sizes, was rated as unclear or at high risk of bias in all but two studies.

The existing studies examine whether or not the interventions were effective in general; they do not specifically examine whether or not early treatment led to improved outcomes relative to later treatment.

On the basis of the current evidence base, therefore, it would be premature to conclude that there are effective interventions for mental health problems in this population in general. It is also unknown if early intervention is more effective than later treatment.

Criterion met? No.

Criterion 13

‘There should be evidence from high-quality randomised controlled trials that the screening programme is effective in reducing mortality or morbidity.’

We were unable to identify any studies that had examined the effectiveness of a mental health screening programme in a population of young people who offend.

Criterion met? No.

Criterion 16

‘The opportunity cost of the screening programme (including testing, diagnosis and treatment, administration, training and quality assurance) should be economically balanced in relation to expenditure on medical care as a whole (i.e. value for money).’

Chapter 8 produced an illustrative decision model to evaluate the opportunity costs of screening programmes for the identification of depression in young people who offend. This model provides a preliminary framework for evaluating the value for money of various screening strategies. However, there exists considerable uncertainty in the input parameters, which, as described in the next chapter, will require substantial future research before conclusions related to this criterion can be drawn.

Criterion met? No.

Summary

The UK NSC provides a detailed list of the criteria that should ideally be met before a screening programme is used in the UK. We were able to evaluate a number of these criteria on the basis of the results of the evidence syntheses. Of the criteria we were able to examine, none was currently met for any screening method for any of the mental health problems we examined as part of this review.

Evidence gaps identified by the systematic reviews

Table 34 summarises the current evidence base on screening for a range of mental health problems in young people who offend. The numbers refer to the number of studies identified by the review for each of the broad areas for which we sought to identify relevant evidence. When more than one study reported data from the same or an overlapping data set, this is counted as a single study in the table.

As Table 32 indicates, there are more gaps than evidence, and when evidence does exist it is limited. For many of the mental health problems examined as part of this review, the evidence was restricted to one or two studies of diagnostic test accuracy; for a small number of problems there was additional information on the effectiveness of clinical interventions. For areas in which some information was available, quality assessments indicated that many of the studies were either at high or unclear risk of bias across a number of domains. The existing evidence therefore does not provide a robust evaluation of the test accuracy, clinical effectiveness or cost-effectiveness of screening or more generally the clinical effectiveness or cost-effectiveness of treatments for mental health problems among young people who offend.

Clearly, many uncertainties remain. Given the limited nature of the current evidence base, both in terms of quantity and quality, a general recommendation is to conduct an extensive programme of more fundamental work before a randomised controlled trial of screening for mental health problems in young people is considered. This more fundamental work should include methodologically robust trials of interventions for mental health problems in this population as well as comprehensive studies of diagnostic test accuracy. The number of relevant studies identified was small for both community and incarcerated settings so there is a need to conduct this more fundamental work in both of these settings.

Before providing detailed research recommendations, however, the insights gained from the decision model are outlined, because these help to more clearly specify the nature of future research priorities.

Insights from the decision model

Value of information analysis evaluates the opportunity cost arising from making a suboptimal (or wrong) decision (such as opting for a suboptimal screening and treatment strategy in this context) on the basis of imperfect current evidence. The underlying idea is that uncertainty in parameters results in uncertainty in a decision, which has an opportunity cost that can be reduced by carrying out further research. By subscribing a monetary value to reducing uncertainty in parameters, the VoI analysis indicates how much decision-makers should be willing to pay to optimise their decision. As conducting research can be expensive, its cost should be contrasted against the consequences of making a wrong decision. Hence, the VoI analysis informs whether or not future research is worthwhile (i.e. the potential ‘payback’ of expenditure on research) The VoI approach ensures that allocation of funds to health research is in proportion to the opportunity cost to the health services.

The expected value of perfect information (EVPI) estimates the expected total losses given decision uncertainty. Should the EVPI suggest that further research is warranted, a decision model can be utilised to indicate specific input parameters that contribute most to decision uncertainty (by parameter we mean the evidence used in the analysis presented in Chapter 8, such as the prevalence of mental health problems). The expected value of partial perfect information (EVPPI) can identify which specific parameter or set of parameters is likely to represent the best value for investment in research.

This review would have ideally liked to quantify the value of further research on mental health screening and treatment in young offenders. However, as we have discussed in previous chapters, the current evidence base to inform decision-making in the context of mental health in young offenders is very limited; therefore, it does not allow us to develop a comprehensive probabilistic framework to formally evaluate the opportunity cost of making a suboptimal decision because of uncertainty in parameters. Despite these limitations, an exemplar model for depression was developed using the limited available evidence, which allowed us to identify the key drivers of decision uncertainty. These key drivers and their potential impact were evaluated in a number of sensitivity analyses in Chapter 8. In the absence of formal VoI analysis, examination of these key drivers can help identify key priorities for future research. The exemplar model identified three key parameters that should be prioritised in future research, namely (1) the prevalence of unidentified mental health problems that in the absence of screening would go undetected; (2) the effectiveness of interventions to improve mental health in young offenders that can be evaluated using generic measures (e.g. QALYs); and (3) the impact of interventions on recidivism. The remainder of this section explores each of these parameters individually by examining the assumptions made, evaluating the sensitivity of the decision to varying levels of input parameters and evaluating the potential impact of resolving uncertainty in these parameters in terms of the allocation decision.

Description of future research priorities

The combination of the results of the systematic reviews, particularly the large evidence gaps identified, and the insights about key drivers offered by the decision model suggest a number of research priorities for diagnostic test accuracy studies, clinical effectiveness and cost-effectiveness trials of interventions and clinical effectiveness and cost-effectiveness trials of screening.

Summary

The series of systematic reviews and insights from the decision model suggest a number of research priorities. These include the need for methodologically robust trials of interventions that permit the calculation of QALYs and assess the impact of interventions on recidivism. These also include the need for methodologically robust diagnostic test accuracy studies that provide an indication of the accuracy of screening instruments in identifying previously unidentified mental health problems. It may be appropriate to answer these more fundamental research questions before trials of screening are undertaken. Research in both community and incarcerated settings is equally limited and so the same research priorities apply equally to both.

Statement of principal findings

Limitations

The results of the current review should be interpreted in the light of limitations of the review itself and limitations of the primary studies. Although the next chapter will summarise the main suggested research priorities, this section on limitations will also offer recommendations for improving the methodological quality and the quality of reporting of future studies in this area.

Implications

Current UK policy recommends the use of the mental health component of the CHAT as the screening measure for the identification of mental health problems among young people who offend. Our review identified no trials of the clinical effectiveness and cost-effectiveness of screening in this population. Furthermore, we identified limited evidence relating to the clinical effectiveness of interventions for mental health problems in this group. Many of the trials were conducted in the USA and many were conducted in custodial settings. In contrast, the majority of young people who offend in the UK are managed in community settings. It remains unclear if such interventions are effective in young people who offend, particularly as applied to a UK setting. This is important because screening can be of value only if it can be linked to an effective intervention.

Our review also identified limited data on the diagnostic test accuracy of screening measures in this population. As with the clinical trial data, the majority of the studies were conducted in the USA and many were carried out in settings that may differ from typical settings in the UK for young people who offend. We are aware, however, that diagnostic test accuracy data for the CHAT have recently been published23 and this may alter this conclusion.

Although we developed a decision model, this is, as described previously, best seen as an exemplar because the data on which the model is based are extremely limited. However, the model is of use in identifying parameters that may act as key drivers determining whether or not screening is likely to be cost-effective; these are of relevance in determining future research recommendations.

Summary of research recommendations

The limited evidence that we identified suggests that there are a number of areas of uncertainty and a need for future research to reduce this uncertainty. Full details of the recommended research priorities were provided in Chapter 10; the main points are summarised here.

• In terms of clinical effectiveness, the limitations of the existing randomised controlled trial evidence base suggest that further feasibility trials of clinical effectiveness are needed to establish important parameters ahead of definitive trials. Future trials should gather information to permit the calculation of QALYs and should seek to assess whether or not treatment for mental health problems alters intersectoral outcomes, including recidivism.

• There is a need for validation studies in which the performance of a range of screening measures is directly compared against a gold standard diagnostic interview conducted to internationally recognised criteria. Screening measures currently recommended for use in the UK to identify mental health difficulties among young people who have offended, specifically the mental health screen of the CHAT, should be directly compared against other available screening measures as part of such studies. Studies should seek to calculate the diagnostic performance of measures in identifying previously unknown cases.

• This fundamental work on diagnostic test accuracy and clinical effectiveness should be conducted before a trial of screening in this area.

• Evidence was lacking for both community and incarcerated settings so these recommendations apply equally to both settings.

Contributions of authors

Rachel Richardson (Research Fellow) was responsible for study selection, data extraction and quality assessment.

Dominic Trépel (Health Economist) contributed to all aspects of the economic sections and took primary responsibility for the drafting of the economic chapters.

Amanda Perry (Senior Lecturer in Forensic Psychology) was responsible for study selection, data extraction and quality assessment.

Shehzad Ali (Research Fellow, Health Economist) advised on all aspects of the economic sections.

Steven Duffy (Information Specialist) carried out the literature searches and took primary responsibility for drafting the search strategies section of the report.

Rhian Gabe (Statistician) contributed to all aspects of the statistical analysis.

Simon Gilbody (Professor of Mental Health Services Research) provided advice throughout the project on systematic review methods and contributed to the writing of the report.

Julie Glanville (Information Specialist) advised on all aspects of the development of the search strategy.

Catherine Hewitt (Statistician) advised on all aspects of the statistical analysis.

Laura Manea (Lecturer in Psychiatry) contributed to all aspects of the statistical analysis.

Stephen Palmer (Professor, Health Economist) advised on all aspects of the economic sections.

Barry Wright (Professor of Child and Adolescent Psychiatry) provided clinical advice throughout the project and contributed to the writing of the report.

Dean McMillan (Senior Lecturer in Mental Health Services Research) had overall responsibility for the project and took primary responsibility for the drafting of the report.

All of the authors contributed to and commented on the report.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health.

Summary of database searches

Search strategies for electronic databases

Background

This guide deals with the use of QUADAS-2 in the young offender review. Whiting et al. 33 state that ‘The QUADAS-2 tool must be tailored to each review by adding or omitting signalling questions and developing review-specific guidance on how to assess each signalling question and use this information to judge the risk of bias.’ This guide aims to tailor QUADAS-2 to the young offender review.

Review question

The review question can be broken down as follows:

• patients: young people (aged 10–21 years) who have offended and who are in contact with the criminal justice system

• index test and target condition: screening measures designed to identify one or more of the following: depression, anxiety problems (including PTSD), risk of self-harm/suicide, schizophrenia, psychosis, ADHD and conduct disorders, autism spectrum disorders and bipolar disorder

• reference tests: for studies reporting diagnostic accuracy, a standardised diagnostic interview conducted to internationally recognised criteria (e.g. ICD or DSM).

The following additional information is required for QUADAS-2:

• setting: any part of the criminal justice system

• intended use of the index test: to identify a target condition or a mental health need

• patient presentation: patients can be tested at any point during their contact with the criminal justice system

• previous testing: patients may or may not have been tested previously.

The review question has been defined and it is now possible to review how each of the four QUADAS-2 domains can be applied to the young offender review

Domain 1: patient selection

Domain 2: index test(s)

Domain 3: reference standard

Domain 4: flow and timing