Systematic review of tools to measure outcomes for young children with autism spectrum disorder

What should be measured?

There are several ways to consider the question of what to measure, including what government departments need in order to measure progress and outcomes, what matters to parents and individuals with ASD, and the theoretical basis of ASD, which has implications regarding important domains to measure.

The UK Chief Medical Officer’s 2012 report focused on Child Health,14 and discussed the poor educational, health and employment outcomes for children with neurodisability. In recent years, there has been consultation about the UK National Health Service Outcomes Framework 2011/12,15 part of a strategy that aims to deliver ‘the outcomes that matter most to people’, using patient-reported outcome measures. The Kennedy report ‘Getting It Right for Children and Young People’16 highlighted the need to identify a common vision between families and professionals for what services are seeking to achieve for children. Measuring outcomes that are valued by families is central to that vision, which, in turn, will influence what services are provided and how, and potentially what services and interventions are prioritised for research evaluation. A recent National Institute for Health Research (NIHR) study has reported agreement on what are the valued outcomes of care for children with neurodisability,17 but it is not clear whether or not these would be the same if a set of core outcomes were sought specifically for children with ASD.

The aims of our MeASURe (Measurement in Autism Spectrum disorder Under Review) review are to identify the validity of tools and outcome measures used in measuring and monitoring young children with ASD, and to consider how well these reflect and measure issues of importance for patients and carers (see Appendix 1). To achieve this we have:

• identified the tools reported in literature on quantitative research involving children of up to approximately 6 years of age with ASD (see Chapter 3)

• conducted a detailed systematic review of the measurement properties of tools within the major domains of development and functioning (see Chapter 4)

• synthesised evidence regarding the most robust and useful tools in these different domains (see Chapter 5)

• identified gaps in measurement of outcomes and made research recommendations.

An important part of the strategy has been to identify what people on the autism spectrum, and parents of children with ASD, think should be measured. As these stakeholders were involved at various stages throughout the project, there is no single section on ‘patient and public involvement’ in the report. Instead, parents and people on the autism spectrum have contributed particularly to Chapters 2, 5 and 6. In Chapter 2, we address the issue of what outcomes should be measured.

Valued outcomes

There exist recommended procedures for agreeing what should be a core set of outcomes in various fields of health care. As Williamson et al. 18 note, ‘insufficient attention has been paid to the outcomes measured in clinical trials’. Consistency and interpretation will be improved if researchers always collect and report on core outcomes. The Core Outcome Measures in Effectiveness Trials (COMET) initiative funded by the Medical Research Council Network of Hubs for Trials Methodology Research aims to develop a set of evidence-based procedures for developing a core outcome set. The suggested steps involve:

• Step 1 Agree the scope of the area of health care.

• Step 2 Identify existing knowledge about outcomes.

• Step 3 Involve key stakeholders, including patients and health-care providers.

• Step 4 Develop consensus about what to measure. Techniques for doing this in an inclusive and objective way are outlined in Williamson et al.,18 including how to determine when consensus has been achieved.

A systematic review of studies that aimed to determine which outcomes to measure in clinical trials in children concluded that in most specialties no research had been undertaken. 19

The scope for this review was determined in the Health Technology Assessment (HTA) commissioning brief (i.e. COMET step 1) and includes a potentially broader use of outcomes than only in trials. This chapter presents the work undertaken on steps 2 and 3, i.e. to identify priorities for child outcomes as valued by parents and professionals, and as explored in qualitative literature. Because of the complexity of ASD as a disorder, and the developmental context of measuring outcomes up to the age of 6 years, the MeASURe review adopted a further step of placing the findings of the consultation stages in a conceptual framework to guide the full review of tools for measurement. This framework also guided further consultation with stakeholders about the relative importance of outcomes to measure. MeASURe did not undertake a further formal process to develop consensus (step 4 above). It may be that ‘consensus’ would be hard to achieve but it would require further procedures. In principle, the choice of outcomes to focus upon depends on the specific research question being asked, and on what is important to particular groups of stakeholders.

Considerations for developing a conceptual framework in autism spectrum disorder

One important potential basis for a conceptual framework for valued outcomes for children with ASD is the International Classification of Functioning, Disability and Health for Children and Youth (ICF-CY),20 so that what is measured can be ‘mapped’ against domains of functioning (e.g. Learning, Communication, Self-Care) and participation (e.g. Relationships, Community Life).

The conceptual framework should also be influenced by an understanding of ASD. The behavioural characteristics of ASD are underpinned by genetic, brain structure and neuropsychological differences from typical development. 21 The conclusions of many studies have led to the revision of the Diagnostic and Statistical Manual of Mental Disorders-Fifth Edition (DSM-5) published in May 2013, such that the impairments in ASD are best considered within two groupings: social/communication difficulties and restricted/repetitive interests and behaviours, including hypo- or hyper-responsiveness to sensory stimuli. The aetiological underpinnings for each of these broad domains of impairments may be different, and both may be targets for interventions targeting ‘core’ features of autism.

Another aspect of complexity in the understanding of ASD is that its measurement is affected by developmental considerations, i.e. children’s profile of skills and difficulties may look very different over time, and those trajectories will also be affected by levels of ability. As an obvious example, we cannot measure children’s social ability to make and keep friendships with validity until they are of an age at which that might be expected in typical development. However, there is some recent agreement on the core early impairments that may be observed. By the age of 2 years, differences in the development of children with ASD (from typically developing children and those with developmental delay) are evident in behaviours such as fearfulness, frustration and lack of co-operation, quick mood changes, and fine and gross motor skills. 22 This knowledge has been enhanced by recent studies of the early development of baby siblings of children with autism (who have an increased chance of themselves developing ASD); for example, Zwaigenbaum et al. 23,24 reported unusual eye contact, a lack of visual attention, orientating to name, imitation, social interest and emotional affect, and heightened sensory-orientated behaviours. These combinations of deficits have consequences for development of relationships, early language and play, and, in turn, for the content and targets of early intervention.

The complexity of understanding ASD is made even greater when considering the interaction between domains of development, and how deficits in one may impact upon another; for example, visual sensory overload may lead to avoidance, which reduces opportunities for visual learning and social experience, leading to poor social skills. Furthermore, there is evidence that pragmatic skills (such as social timing in interaction) are closely associated with particular types of behavioural profiles. 25,26 Thus the conceptual framework for a review of outcome tools should consider both measurement of individual areas of functioning, which are likely to change over time, and also tools that bring together these separate areas into a more holistic assessment. It is particularly important to be able to create profiles for children with ASD, who often have difficulties in generalising learning between areas of skill and also generalising skills from one setting to another.

It is also important to detail other associated difficulties that are not unique to ASD but which, nevertheless, can play a major part in children’s development and the burden of care for families. These commonly include feeding and eating difficulties (resistance to certain food textures, faddiness about types and colours of food, etc.), behaviour and sleeping problems. Children who lack adequate nutrition and sleep are likely to be bad tempered and even more rigid in their thinking and behaviour. Furthermore, adaptive functioning may be more impaired in children with ASD than would be expected from their level of ability. Finally, as young children’s development is intimately affected by their environment, including the health, skills and resilience of parents and carers, it is important to include consideration of the impact on the family.

Although the conceptual framework developed over the process of consultation, and was informed by the separate activities described below, it is presented first for brevity and clarity (Table 1). The framework adopted was informed by discussion within the project team, and inspection of other relevant frameworks such as groupings of target symptoms/skills from interventions studies in ASD,20,27 education outcomes,28 grouping of interventions by Research Autism (http://researchautism.net/pages/autism.treatments.therapies.interventions/) and compilation of measures for children with developmental disabilities. 29 One area discussed was how to categorise quality of life, which is essentially a construct separate from the ICF-CY. The decision was made to include it in the participation domain, as it implies how an individual interacts with their environment. 30 For the MeASURe conceptual framework, there are four primary domains, with subdomains in each.

Question

What child- and/or family-specific outcomes do parents of children with ASD perceive as important?

Methods

Data synthesis

In order to present an overview to the parent advisory groups and the research team, key findings (including illustrative quotes) and analytical frameworks from each paper were extracted and tabulated, and themes identified.

Results

Searches identified 102 relevant papers. Fourteen studies were selected as sufficiently relevant to obtain a full text of the paper. Three of these were excluded because they did not collect qualitative data on outcomes; four were excluded because they contained no data on outcomes. Seven articles remained.

It was clear from an initial inspection of these articles that the quality of data was variable and ranged in focus/topic. Three articles reported data relevant to only parent outcomes,31–33 including process outcomes. Three articles reported data relevant to child outcomes only34–36 and one reported both. 37 One study included data collected directly from children and young people with ASD,34 although original quotes from children were not presented.

The age range of children represented in these studies was 0–21 years. Just two studies31,33 focused specifically on younger children (3–6 years;31 up to 5 years33). The diagnoses represented in the studies typically relied on parental reports. Two studies35,37 focused on particular diagnostic groups (Asperger syndrome;37 ASD with no functional communication35) and others were defined in terms of use of a particular service (speech-and-language therapy;31 assessment and diagnosis;32 preschool educational intervention33).

Data collection methods included focus groups, individual face-to-face interviews and open-ended/free-text questions within postal and web-based surveys.

Although we did not appraise quality of studies formally (using any standard checklist), the quality of reporting sampling and recruitment, data collection methods and data analysis processes was extremely variable.

Given the significant limitations, in terms of quality and relevance, a ‘light touch’ data extraction was undertaken to identify outcomes and themes (see Appendix 2).

In terms of child outcomes, it was notable that some aspects deemed ‘fundamental’ by parents may not be regularly assessed (such as ‘safety’),34 and certainly not as an outcome of an early intervention trial. ‘Awareness of danger’ was added to the conceptual framework (subdomain Social inclusion) as a fundamental issue of safety. The parents’ and young people’s emphasis on participation outcomes (such as being ‘isolated from peers’ or ‘live a normal life’) may also not be reflected in what is usually measured. Constructs concerning child and parent stress, and positive mental health36,37 were supported as important to include as outcome constructs in the conceptual framework.

In addition, parents often highlighted the processes of interaction with professionals, and the utility of information from assessments. Parents expected the service to provide them with information and research literature; to involve them in decision-making processes; and to teach parents how to deliver therapies at home. 31 Braiden et al. 32 reported that parents ‘desired information relevant and applicable to their child to assist them in understanding and making sense of their own child’s presentation’. They also mentioned wanting to have positive times with their child: ‘when he is behaving well and not gearing up for a fight, he’s a very happy and pleasant child’. 37 Such parent priorities have informed the conclusions of the MeASURe project.

Questions

Methods

People on the autism spectrum in Birmingham and Newcastle were approached for their opinions by a person they knew well. In Birmingham, six adults were known to the University and responded by e-mail; 10 children were approached by a member of the Autism Outreach Team and were selected on the basis that they were thought able to give their views on the questions. In Newcastle, two children attending a National Autistic Society social group responded in person and, likewise, two young adults attending a social inclusion group. Responses to the questions were thus received from 12 young people aged 9–15 years, and from eight adults aged 22–43 years. Each respondent was given a shopping voucher in acknowledgement of their contribution. The verbatim responses were collated and common themes extracted.

Results

Question

What constructs are most often measured by early years professionals in monitoring children’s progress?

Methods

A survey was undertaken in autumn 2012 through the British Academy of Childhood Disability database of 240 UK Child Development Teams. Professionals were sent an electronic link to a web-based survey that took 10 minutes to complete. In parallel, education professionals received the survey via (1) ‘4 Children’, a national charity and UK Government strategic partner for early years and child care, through their database of 15,000 Early Years providers across England; (2) the database of an independent specialist centre for early years children with ASD; and (3) 150 practitioners undertaking the Birmingham University School of Education Autism Programme residential weekend.

The survey had five sections. Respondents were asked:

1. About their profession, and the setting in which they work.

2. ‘Do you regularly work with children on the autism spectrum (this includes any activity that aims to improve/change an area of functioning)?’ and ‘Do you ever measure the progress or outcomes of children on the autism spectrum (i.e. more than just seeing the child once for assessment)?’. Those who indicated ‘yes’ were given access to the rest of the questions.

3. To identify all areas in which they formally measure or informally make judgements about progress or outcomes for children on the autism spectrum whom they see more than once, and who are aged ≤ 6 years. The 68 outcome constructs included were taken from the conceptual framework.

4. To indicate how frequently (on a six-point scale) they used different types of tool: standardised measures of progress or outcome (with manual and comparative age-related information); non-standardised measures (either published or created locally); informal judgements.

5. To give the three areas in which they most frequently measure progress over time, or outcome, with which of the three types of tools.

Results

The 836 respondents included 167 health professionals (paediatricians, speech-and-language therapists, clinical psychologists, occupational therapists, physiotherapists, dietitians, health visitors, social workers and educational psychologists), 353 education professionals (teachers, special educational needs co-ordinators, autism education advisors, teaching assistants, intervention practitioners), 125 nursery nurses and 191 other professionals, many of whom were childminders. Professionals worked in a variety of settings. Many health professionals worked in child development centres or hospitals but some were based mainly in educational settings. Education staff were from mainstream and specialist schools or early years settings.

Five hundred and thirty-seven professionals monitored the progress or outcome of children who were seen more than once, and were able to access the remainder of the survey.

Professionals were more likely to measure characteristics such as amount of speech (76%), social interaction (90%) and attention (79%) than life or adaptive skills (measuring for, or trying on, clothes 6%, difficulties with appointments, e.g. hairdresser, dentist 16%, use of knife and fork 29%), features related to ‘quality of life’ for the child (quality of life 21%, happiness 42%) or the family (nature of sibling relationship 18%, family quality of life 22%, impact on the family 33%).

Professionals were more likely to use their ‘own informal judgement in discussion with parents or other professionals’ than standardised measures to rate improvements (442 respondents agreed with ‘used often’, ‘most of the time’ or ‘always’ compared with 253 who checked ‘never’, ‘rarely’ or ‘sometimes’). The specific types of measures used varied very widely due to the broad range of professional respondents. However, consistently across the questions, around one-third of respondents replied that they used standardised measures, and half said that they were most likely to use parent or professional impression to gauge progress or outcome. (The standardised tools identified as used most frequently were later included in searches in Chapter 4.)

In conclusion, this survey found that professionals are most likely to measure features related to core impairment subdomains of autism, rather than aspects of daily living, family functioning, and child well-being and happiness.

Question

What outcomes do parents consider as important to be assessed?

Methods

Parent advisory groups were recruited at three sites (Exeter, South London, Newcastle). In Exeter, the Peninsula Cerebra Research Unit involves families of disabled children as partners in research through a Family Faculty. 38 Parents of children with ASD were e-mailed and invited to volunteer: 12 expressed interest and seven participated in one or more meetings. In London, the Newcomen Neurodisability Team involves families of children with ASD in giving advice on an ad hoc basis; for MeASURe, 10 parents were invited by e-mail and six participated in one or more meetings. In Newcastle, parents of children with ASD aged ≤ 10 years were invited by e-mail; four participated in one or more meetings. Thus a total of 17 parents were involved in discussion meetings. Parents were given a financial acknowledgement in addition to travel expenses, to recognise their time and expertise at each attendance. Meetings were held at three points during the MeASURe project.

Early meeting To explore parents’ priorities and experiences of assessment and identify what outcomes parents saw as important, especially for monitoring their young child with ASD over time. This session involved an explanation of the aims of the project and open discussion, led by a member of the project team and a parent involvement co-ordinator.

Mid-point meeting To undertake a Q-sort of constructs emerging from the conceptual framework. Two members of the MeASURe project team (NL, GM) created ‘lay wording’ versions of the constructs. Sixty-two constructs were presented on cards in a jumbled order (i.e. not including symptom severity, physical indicators, global measure of function, global measure of outcome). The way in which the constructs had been chosen was introduced by the project team member. Through discussion within the parent group, the constructs were sorted on to a ‘forced-choice’ grid in a pyramid shape on a large piece of paper. Columns on the grid were rated for levels of importance (from ‘more’ to ‘less’ on an 11-point scale), i.e. ‘the importance of various things which could be measured when tracking the progress of children with autism aged up to 6 years, or in measuring the outcome of a specific preschool intervention’. It was stressed that none of the constructs was considered unimportant.

End-point meeting Parent groups met again to consider a summary of the findings of the literature reviews and early consultations. This included a question about the reasons for differences between what parents consider important to be measured and what professionals most often measure. The main activity was to examine five questionnaires that had been rated positively in the systematic reviews. Parents were asked to compare and contrast two questionnaires about parent stress, two questionnaires about children’s behaviour problems, and one questionnaire designed as a global measure of outcome. The issues raised were then taken by parent representatives to the MeASURe project Discussion Day on 14 February 2014 (see Chapter 2, Consultation with multiple stakeholders).

Early meeting

Parents expected that professionals would focus on assessment of core features of autism, such as communication and social interaction. However, they suggested that the child’s skills should be acknowledged and more attention be paid to unusual behaviours that the child is exhibiting, as well as measuring what the child is not achieving. For parents, priority areas for measurement included habit behaviours (such as sleep, diet and food-related behaviours, sensory processing issues, toileting) and also challenging behaviours and ‘meltdowns’ (such as self-harm, hitting out, anxiety, stress, happiness, tics). Parents endorsed the importance of social communication and social functioning (interacting, playing with others, playing alone, understanding and communicating) and, furthermore, the building blocks of learning, independence and life skills (reading and academic achievements, hobbies and sport, imagination and creativity, self-care, preparing food, getting dressed, time management, vulnerability and danger). They also stated that they recognised that some activities/skills may not seem that important or be seen as relevant for this young age range but become a more significant priority later on in development and as their child progresses through school. Parents also mentioned difficulties they had with taking children to appointments for health care (vaccination, dental care, shoes, eyes and hearing). These constructs influenced the conceptual framework, and the content of the survey for professionals.

Parents also commented on aspects of the process of assessment. They recommended the use of video in relaxed environments, so that professionals may watch for changes. They stressed the value of information gathered in a range of settings (suggesting use of video to rate change over time and between different settings). Figure 1 illustrates the themes arising from one of the parent group meetings.

FIGURE 1.

Themes from Exeter parent group discussion. DLA, Disability Living Allowance.

Mid-point meetings to undertake the Q-sort

There were four groups that completed this task (two in London to accommodate parents’ availability). Taking an average of the Q-sort ratings from all sites, the items rated on average as ‘more important’ can be grouped as:

• Body functions/impairments Hypersensitivity, anxiety, unusual fears, distress, non-verbal ability, expressive and receptive language.

• Activity level indicators Aggression, sleep problems, school readiness.

• Participation Happiness, self-esteem, relationships with brothers and sisters, being bullied/rejected, no awareness of danger.

• Families Parent stress.

The highest level of consistency in rating these constructs across groups was for aggression and sleep problems. Parents rated happiness as important for all young children but one group did not agree that this could be considered an ASD-specific measurable outcome. In discussion parents mentioned that they had had to learn about autism, and so had not understood at the start of assessments of their child why skills such as ‘joint attention’ were of importance.

End-point meetings

In London it was not possible to arrange an end-point meeting; there were a number of barriers for parents’ attendance including ‘travel time’, ‘difficulties getting child care’ and ‘need to battle the new school statementing system’. In Exeter, a preliminary meeting was held to discuss with parents how to assess the strengths and weaknesses of tools used in assessment (i.e. explanations of terms such as reliability, validity and sensitivity to change).

Within the preliminary report, the ratings on importance by parents were compared with the constructs most often measured by professionals (Table 2), and parents were asked to reflect on the differences.

The only area of overlap between parents and professionals was ‘challenging behaviour’/’fighting, hitting others’. Parents in Newcastle highlighted that they believed that parents tend to focus on broader outcomes than professionals, as they see their children everyday in different environments. Anxieties and distress were emphasised; parents remarked that it is emotional needs that impact on the child’s and family’s quality of life. Parents also believe that professionals tend to be unaware of these important difficulties before a child enters the social environment of education.

The Exeter parents had a rather different way of viewing the table. They suggested that professionals measured aspects that were intermediate outcomes, which facilitate what parents rate as important. For example, they suggested that parents’ identification of ‘fighting, hitting others’, ‘distress’, ‘happiness’ and even ‘parent stress’ could be mapped from what the professionals highlighted as ‘challenging behaviour’. Similarly, when parents highlighted ‘relationships with brothers and sisters’, these benefited from adequate ‘speech’ and ‘understanding of language’. So, despite the different labels, there was general support from parents for what professionals measure, and parents had noticed their children making progress in these areas.

Question

What outcomes is it important to assess when monitoring the progress and outcomes for children with ASD up to the age of 6 years?

Methods

A Discussion Day was held in London on 14 February 2014. Twenty-five participants were invited: four parents of children with autism; three young people with autism, two of those with staff who support them in education; eight speech-and-language therapists, occupational therapists, paediatricians or psychologists; and two researchers working with children with autism; six MeASURe project researchers who work in health or education services also attended.

As one activity, groups of similar background carried out a further Q-sort to rate the importance of constructs, and ascertain similarities or differences between what parents, professionals and researchers consider most important. The set of 21 constructs for the parents and for the young people was drawn primarily from those reported as most often assessed by the early years professionals in the survey. The set for the professionals and the researchers included the 10 rated as most important by the parent groups, and the 10 most often measured by professionals as reported in the survey. Both sets were completed with added constructs to represent a wide span of subdomains.

We hypothesised that:

• The young people and parents might well agree on the top 10 constructs.

• The professionals might also agree with parents, even although what they actually measure is not in accordance.

• The researchers might choose a different set (more based on intervention elements, symptoms and everyday function).

• We had no expectation about parents’ views on the order of importance of what professionals tend to measure.

Results

Adverse weather conditions and train cancellations prevented several participants joining the Discussion Day, including two young people on the autism spectrum. However, four groups of four people each considered the constructs.

One young adult on the autism spectrum joined the parent group; his/her ranking showed a high level of agreement with the averaged ranking of constructs undertaken previously by parent groups (Spearman rank correlation r_s = 0.618). Fine motor skills were rated higher than previously because of the experiences of the young adult as a child. ‘Friendships’ was rated higher than previously, reflecting on the precursor skills needed by the child early on that will lead later to being able to make friendships. Aspects that affect the emotional state of the child, including sensory processing, continued to be rated highly. ‘Participates in mainstream activities’ was rated low: the group thought ‘this means the ASD child has to adapt to the mainstream world rather than ‘mainstream’ adapting/understanding/respecting ASD needs’. They also gave a low rating to ‘not cooperating, throwing, spitting, won’t sit (challenging behaviour)’, as they considered it the role of adults (parents, education and care staff) to try to make the environment right for the child so his/her autism was less ‘disabling’.

The two multidisciplinary groups of health and education professionals, and the group of ASD researchers, had low agreement with the averaged ratings of the parent groups (r_s = –0.268, 0.131 and –0.063, respectively). The health and education professionals commented that they measure what they can (in the setting, given the available tools) and what they traditionally have done. They emphasised as ‘important’ what they see as most urgent to try to change, such as challenging behaviour and communication skills. In contrast, although acknowledging the importance of the construct ‘positive views of self (self-esteem)’, they gave it a lower rating because of the developmental stage of children up to 6 years; researchers similarly rated self-esteem as low because of the lack of a suitable measurement tool. The researchers had rated highest ‘not cooperating, throwing, spitting, won’t sit (challenging behaviour)’ on the basis of its impact on others and on the child’s experience. Both groups of health and education professionals identified a range of additional constructs that they would consider it important to measure, including communicative competence, problems with food, functional adaptive behaviour, etc. They also mentioned the importance of identifying the skill set of support staff, and parent confidence in managing their child’s needs and behaviours.

When all groups came together, the discussion highlighted differences in perspective, in summary a ‘social’ model and a ‘medical’ model. The parents and the young adult on the autism spectrum argued that it is important to focus on what children can do, to see autism as a ‘difference’ rather than always use a ‘deficit’ model, and to focus more on how to enable children through improving their environments. Parents were encouraged that the clinicians had mentioned including assessment of the skills of care and education staff. The clinicians reflected that their approach to assessment and intervention is based on a more ‘medical’ model: early identification of specific impairments, treatment, prevention of secondary impairment, and so on. The measurement of outcomes and tools available reflect this framework, with an emphasis on problems and deficits. For the researchers, the model of intervention and outcome assessment was also primarily embedded in a ‘deficit’ model of autism, with an emphasis on treating and measuring core features of autism. Research outcomes such as helping parents manage better and understand more are seen as ‘soft outcomes’, and not given the same importance as changing children’s characteristics. A certain contradiction was pointed out between the recognition that publicly funded research must now be informed by good patient and public involvement, and yet the priority research questions, commissioning briefs and frameworks for judgement of what is good science do not necessarily value the social model of understanding a condition such as ASD.

Review question

What tools are in use for measuring and monitoring developmental outcomes in young children with ASD?

Search strategy

We included studies published from 1992 to coincide with the publication of the international classifications, International Classification of Diseases, 10th Edition (ICD-10) and Diagnostic and Statistical Manual of Mental Disorders-Fourth Edition (DSM-IV). 48,49

Original searches were conducted in June and July 2012, and re-run in June and July 2013. The databases searched were:

• Applied Social Sciences Index and Abstracts (ProQuest) 1987 to present

• Cumulative Index of Nursing and Allied Health (CINAHL) (EBSCOhost) 1981 to present

• The Cochrane Library [includes Database of Abstracts of Reviews of Effects, HTA, Cochrane Central Register of controlled Trials, Cochrane Database of Systematic Reviews (Ovid)] inception to present

• Education Resources Information Center (ERIC) (ProQuest) 1966 to present

• MEDLINE (including In-Process and Other Non-Indexed citations) (Ovid) 1946 to present

• EMBASE (Ovid) 1988 to present

• PsycINFO (Ovid) 1987 to present

• Sociological Abstracts (ProQuest) 1952 to present

• Linguistics and Language Behavior Abstracts (ProQuest) 1973 to present

• Health Management Information Consortium (Ovid) 1979 to present

• PapersFirst [Online Computer Library Centre (OCLC)] inception to present

• Proceedings (OCLC) inception to present

• Scopus, inception to present

• Social Services Abstracts (ProQuest) 1979 to present

• Web of Science (Science Citation Index, Social Sciences Citation Index, Arts and Humanities Citation Index and Conference Proceedings Citation Index inception to present)

• WorldCatDissertations (OCLC) inception to present.

Additionally, grey literature was searched via Digital Education Resource Archive, Oxford Patient-Reported Outcomes Measurement database, Turning Research into Practice database, internet searches, and searching of selected websites (see Appendix 3). The National Research Register and UK Clinical Research Network were also searched for ongoing research.

A master search strategy was created and modified as needed for searching across the breadth of databases; a list of terms can be found in Appendix 3. Modifications included changes to syntax, fields searched and Medical Subject Heading/thesaurus terms. Full search strategies are available from the first author, and example search strategies for MEDLINE, ERIC and Web of Science are provided in Appendix 3. Searches were limited to English-language articles only. When possible, search filters were used to limit study types returned.

Inclusion criteria

We considered inclusion criteria based on types of studies, participants and types of measurement.

Types of studies

We included:

• all relevant randomised and quasi-randomised trials of social, psychological and educational early interventions for children with a diagnosis of ASD

• observational studies of children with ASD (cross-sectional and longitudinal)

• case–control studies

• cohort studies, including studies of baby siblings of children with autism, which provide information on tools to monitor developmental progress and follow early markers of ASD.

Types of participants

We reviewed all studies that included at least 50% of children with ASD. Child participants had a ‘best-estimate’ clinical diagnosis of an ASD, including autism, ASD, atypical autism, Asperger syndrome and pervasive developmental disorder – not otherwise specified (PDD-NOS), according to either ICD-10 or DSM-IV48,49 criteria. Use of a particular diagnostic tool such as the Autism Diagnostic Observation Schedule (ADOS) or the Autism Diagnostic Interview-Revised (ADI-R) was not required. Children with ASD and another medical condition, and children with ASD and comorbid conditions were included.

All children were aged ≤ 6 years upon entering the study.

Types of measurement included

• Direct assessment of child ASD symptoms by trained assessor.

• Direct assessment of developmental skills, i.e. language, cognition, play skills, fine and gross motor skills, by trained assessor.

• Observational coding of social interaction skills.

• Interview or self-completed (parent, teacher or other professional) questionnaire report of child ASD symptoms.

• Interview or self-completed questionnaire report of developmental skills, i.e. language (vocabulary), adaptive skills, with/by parent, teacher or other professional.

• Interview or self-completed (parent, teacher or other professional) questionnaire report of associated problems, including behaviour that challenges, aggression, sleeping, eating, toileting, anxiety, hyperactivity and others identified through parent consultation.

• Idiographic measures focused on particular behaviours (e.g. goal attainment scaling, target behaviours).

• Measures of impact on parent or family.

Types of measurement not included

• Economic impact on home and family.

• Experimental tasks and measures, for example barrier tasks, reaction time.

• Biophysical measures, medical investigations.

• Process measures, for example fidelity, adherence, parent satisfaction with intervention.

Sifting

Papers were first sifted by title and abstract (Figure 2). The decision categories were ‘potentially include’, ‘exclude’, ‘consider for Chapter 4’ (assesses the measurement properties of a tool only) or ‘unclear’. The two reviewers (NL, IPO) cross-checked sets of 20 papers at a time until they reached a high level of agreement. Regular (at least weekly) discussion of decisions was held throughout the process to maintain consistency. Then 3059 papers were examined at full text. When decisions regarding inclusion were uncertain, a third reviewer (HMcC) made the final decision.

FIGURE 2.

Flow diagram of searching and sifting. Search results up to date as of 17 July 2013 (original search and update combined). Sifting decisions up to date as of 13 August 2013. Final total for data extraction = 184 (with addition of records identified at stage 3).

There was a further stage of sifting of records found during the search of papers about measurement properties of tools (see Chapter 4), with searches completed by 9 September 2013. Those searches revealed 118 records potentially relevant to Chapter 3. Once duplicates were removed (86), 32 additional records were sifted by full text (completed 8 December 2013): of these, 28 were excluded and four were added to the final total for data extraction.

Data extraction

A data extraction tool was created as a web-based instrument and piloted (see Appendix 4). The data extracted included study eligibility; type of study; participant characteristics; number of outcome tools (then for each tool: name, population for which designed, specific subscales, outcomes measured according to authors). Subsequently, two reviewers with expertise in ASD (JR, HMcC) reviewed each paper further and indicated which subdomains in the conceptual framework (see Table 1) were measured by each tool, including subscales.

Types of measurement in use

Standardised norm-referenced assessments all have to be administered by a trained professional. They have the advantage of comparison with children of the same age but for several reasons may be misleading when used for the assessment of young children with ASD. The abilities of the children may be underestimated by lack of co-operation with standardised testing, and they may have profiles that are dissimilar to typical development.

Direct observation includes both highly structured observational procedures (such as ADOS)50 and tools used primarily to measure social interaction in naturalistic settings, especially parent–child interaction. The former are a diagnostic assessment tool conducted by a trained assessor with subsequent rating of the child’s behaviours. The latter have the advantage of providing an in-depth understanding of patterns of responsiveness, which may have long-term effects on language and other development. 51,52 However, one major disadvantage of direct observation is the limited time frame with consequent questions of validity. Further, there are almost as many different coding schedules as studies, depending on the focus of interest.

Standardised semistructured interviews have been used in the characterisation of children’s early development and current ASD characteristics (e.g. the ADI-R),53 in the broad measurement of adaptive behaviour [e.g. Vineland Adaptive Behavior Scales (VABS)]54 and to gather information on additional difficulties, such as behaviour problems, anxiety and sleep. Problems of measurement include a paucity of tools focused on behaviour, which are specifically validated for ASD (e.g. the Autism Comorbidity Interview-Present and Lifetime Version is one such tool but is validated from only 5 years of age). 55

There are very many questionnaires used in studies of children with ASD, completed by parents, teachers and clinicians. However, as with the direct observation and assessment tools, many have not been specifically validated for use in ASD and contain assumptions about patterns of typical development (e.g. standard quality-of-life measures do not ask about children’s special skills and circumscribed interests).

Inclusion criteria

1. Study was published as a ‘full text original article’.

2. The tool measured a domain of interest (see ‘conceptual framework’, Table 1).

3. A tool identified at stage 2 (i.e. used for monitoring and/or to measure outcome in a longitudinal or intervention study with children with ASD up to 6 years old) was the focus of the study. (When a paper reported the measurement properties of a ‘new’ relevant tool this was noted but not included.)

4. The study sample overlapped with the age range 0–6 years (e.g. a sample with age range from 6 to 18 years was judged eligible; one that included 8- to 15-year-olds was ineligible).

5. The study sample included at least 50% of children with ASD. Furthermore, the study sample could be individuals who were being monitored for ASD symptoms even if they had another primary diagnosis (e.g. a paper monitoring ASD symptoms in a Fragile X population could be eligible if exploring measurement properties of a tool used as an outcome).

6. The aim of the study was the development of a measurement tool or the evaluation of one or more of its measurement properties. Note: The property ‘Hypothesis testing’ applies in COSMIN to hypothesis testing within a paper about construct validity of a tool (e.g. convergent/divergent validity against other tools; known-groups validity). Studies that tested research hypotheses about change or differences between groups as the result of an intervention, but did not set out to test the measurement properties of the tool, were excluded.

Exclusion criteria

1. Papers in which the measurement tool was tested only for its properties in diagnostic assessment or screening and not for monitoring or measuring an outcome.

2. A sample drawn only from the general population of children.

3. Sample size of < 20.

4. Studies in which the focus of the paper was not the examination of psychometric properties were not eligible (e.g. if the paper focused only on creating a subtype of ASD, or to group individuals by scores on the tool).

5. With regard to papers on translated tools, if the purpose was simply to validate the translated version then it was not eligible. If the purpose was to explore the tool’s validity in a different culture/country, and the focus was on the properties of the tool, and the findings appear relevant for use in UK then it was included.

Four reviewers (MG, JH, NL, IPO) utilised the criteria to sift 10% of articles (Figure 3) independently and to compare results, resulting in tightening of criteria. Sifting was then conducted by a single reviewer, the team having (at random) divided up assessment of titles and abstracts, selection of full-text articles and consultation of reference lists of the studies retrieved. In case of uncertainty, the paper was discussed with HMcC before making the decision regarding inclusion. As the COSMIN rating procedure (see below) involves two stages, and the second summary stage involved a different member of the team (including HMcC) in rating the content of each article, some further exclusions were made, so that the decision-making procedure was very robust.

FIGURE 3.

Flow diagram of searching and sifting. Original stage 3 search results up to date as of 9 September 2013. Sifting decisions up to date as of 24 February 2014. Final total for data extraction = 128 (with addition of records identified at stage 2).

Evaluation of methodological quality

The methodological quality of the studies of measurement properties identified was then assessed using the COSMIN checklist. 57 The checklist has 10 ‘boxes’ or subscales (Internal consistency; Reliability; Measurement error; Content validity; Structural validity; Hypotheses testing; Crosscultural validity; Criterion validity; Responsiveness; Interpretability) with standards for how each measurement property should be assessed (see Appendix 7). Each item is scored on a four-point rating scale (poor to excellent) and an overall rating for the methodological quality of each study is determined. The full tables are presented in Appendix 8.

At the same time, each reviewer extracted relevant numerical and descriptive information about the properties addressed (available from the first author). Terwee et al. 57 presented criteria for judging the adequacy of each piece of information (Table 4).

The final step was to combine the ratings of quality of the studies with the ratings of strength of the findings (Table 5) in order to make judgements related to each measurement tool.

Autism Behavior Checklist

The Autism Behavior Checklist (AuBC)58,59 was originally constructed as a screening questionnaire completed by parents/carers. It has 57 items grouped into five subscales: Sensory; Relating; Body and object use; Language; and Social and self-help skills, and provides different profile charts for different age groups, ranging from 18 months to 35 years. Three papers60–62 considering measurement properties of the AuBC since 1992 were reviewed, of which two60,61 had very small samples. Miranda-Linné et al. 62 used AuBC with parents of 383 individuals aged 5–22 years. Using factor analysis, they found a five-factor solution that was inconsistent with the five factors suggested by the originators, explaining 80% of the variance but with good internal consistency of subscales.

Autism Diagnostic Interview-Revised

The ADI-R53,63 is a standardised semistructured investigator-based interview, which is administered by a trained clinician usually to parents/caregivers. The 1994 revision had 111 items and the 2003 published version has 93 items. Papers on measurement properties have utilised varying numbers of items; for example, Lecavalier et al. 64 considered only the algorithm items that are used in determining diagnosis. The ADI-R has good internal consistency, although the Repetitive Behaviours domain consistently shows Cronbach’s alpha coefficient to be < 0.7 in the papers examined. Several studies report that inter-rater reliability was monitored and kept at 90% agreement. Studies that formally tested inter-rater reliability reported good levels; for example, Lord et al. 65 reported kappa statistics of between 0.6 and 0.8 on individual items and 92% agreement on diagnostic cut-off.

Extensive work has been done on content validity of the ADI-R as a ‘gold standard’ diagnostic tool. One paper66 used principal components analysis to derive six clusters, but noted that the inclusion of a few specific items on to particular clusters was unexpected. The evidence concerning structural validity was somewhat contradictory; because of the large numbers of items, few papers included sufficient participants, even for reduced-item sets. The confirmatory factor analyses for two- or three-factor solutions were ‘reasonable’ [i.e. root–mean–square error of approximation (RMSEA) of ≤ 0.07],64,67,68 i.e. just above cut-off for a rating of ‘good fit’. Frazier et al. 67 indicated that factor structures fit equally well for 2- to 6-year-olds as for those aged ≥ 7 years. Other than testing level of agreement with ADOS (see Autism Diagnostic Observation Schedule, below), only one study64 explicitly addressed convergent/divergent validity, finding that the ADI-R Social domain correlated with all VABS (–0.41 to –0.45) except for motor skills. A strong indication of validity at the level of individual ADI-R items in distinguishing between children with ASD and those without (mostly non-referred) was provided by Tsuchiya et al. 69 Criterion validity was examined in the original Lord et al. 63 paper. In addition, Chawarska et al. 70 examined whether infants referred at age 14–25 months (and diagnosed 15 months later) could be classified at first assessment; 48% who later received the diagnosis of autism were classified as such, but 78% if the ADI Repetitive behaviours scale was dropped.

As a diagnostic tool, ADI-R might not be expected to be responsive to change; indeed, measurement properties papers65,71 do find general stability in meeting diagnostic cut-off after 6–8 years in each of the domains. However, Lord et al. 65 reported that classifications changed substantially more often from ages 2 to 5 years than from ages 5 to 9 years.

Autism Diagnostic Observation Schedule-Generic

The Autism Diagnostic Observation Schedule-Generic (ADOS-G)50 is a semistructured, interactive schedule designed to assess aspects of communication, social reciprocal interaction, play, and stereotyped behaviours and restricted interests. The ADOS-G consists of four modules, appropriate for children and adults of differing language levels, ranging from non-verbal to verbally fluent. The most able, verbally fluent, children may be assessed with module 3; less able children with modules 1 or 2. The social communication algorithm score is reported as the ‘total’, as repetitive behaviours may not be observed within a limited-duration play-based assessment.

Seven papers50,65,70,72–75 considering measurement properties of the ADOS-G were reviewed. The original study50 assessed internal consistency. For the social communication totals, Cronbach’s alpha coefficients were high (0.91–0.94) for modules 1–3; however, as the sample size included in each of the unidimensionality analyses was small, the study was judged poor. Three studies50,65,72 assessed inter-rater reliability. Two of these studies65,72 found kappa statistics to range from 0.60 and 0.80 on all items, and Lord et al. 65 found 92% agreement for autism/not autism. However, the lack of methodological information regarding these analyses led the study by Kamp-Becker et al. 72 to be judged as of poor methodological quality. Lord et al. 50 presented inter-rater kappa statistics for each module separately and found a mean weighted kappa statistic of 0.78 for module 1, 0.70 for module 2 and 0.65 for module 3. For the social-communication algorithm total, test–retest intraclass correlations (ICCs) ranged from 0.84 to 0.98. However, the small sample size for all reliability calculations within modules (ranging from n = 23 to n = 29) led this paper to be judged as of poor methodological quality.

Three papers50,72,73 assessed structural validity. Lord et al. 50 found almost all social and communication items loaded highly on one factor in each module, accounting for between 72% and 78% (modules 1 and 2), and 52% and 53% (modules 3 and 4) of the variance, leading to the adoption of an algorithm total of social-communication items. However, the small sample size (79 participants and 29 ratings in module 1, 55 children and 28 ratings in module 2, and 59 participants and 28 ratings in module 3) led this study to be judged to be of poor methodological quality. Kamp-Becker et al. 72 assessed structural validity and found in a three-factor solution the amount of variance explained was 47%, in the four-factor solution it was 52% and the five-factor solution explained 57% of variance. The third paper73 investigated both the original algorithm structure and the new algorithms, compatible with DSM-5. For the youngest group (≤ 6 years) they found RMSEAs of 0.057 and 0.059, respectively, for module 1 (indicating good model fit) and RMSEAs of 0.079 and 0.076, respectively, for module 3 (indicating reasonable model fit).

Two studies assessed criterion validity. Grey et al. 74 found high agreement between ADOS and a clinical diagnosis of autism [kappa (κ) = 0.73; p < 0.001] and of ASD (κ = 0.62; p < 0.001) in 209 children aged 20–55 months (120 with autism or ASD). In a study of infants aged 14–25 months, Chawarska et al. 70 found between 79% and 95% agreement between the ADOS-G module 1 diagnostic classification outcomes and clinician-assigned diagnosis of autism, although agreement with an ASD diagnosis was low. They comment that in this infant sample the ADOS tended to under-diagnose children with higher verbal and non-verbal skills.

This tool had three papers65,70,75 assessing ‘responsiveness’, or rather testing a hypothesis of stability. The first paper65 found that regression prediction of each ADOS domain score at age 9 years, by the set of three domain scores at age 2 years, showed significant continuity within the same domain (with one exception – the communication score at age 9 years – which was predicted by the ADOS Social and Repetitive domains at age 2 years, with no significant independent contribution from communication). Chawarska et al. 70 tested stability at 15 months from first assessment and found no significant diagnosis by time interactions. Ben Itzchak and Zachor75 also reported stability for 78% of their sample of 68 children, mean age 26 months, in terms of ADOS classification over 1 year; however, the lack of specific hypotheses led this paper to be judged as being of poor methodological quality.

Autism Diagnostic Observation Schedule-Toddler Module

The need for a tool that could assess children for autism at an earlier age led to development of the Autism Diagnostic Observation Schedule-Toddler Module (ADOS-T). The development paper76 involved 182 children with best estimate diagnoses of ASD, non-spectrum developmental delay or typical development, aged 12–30 months. Content validity was good; items were revised, rewritten or removed, as necessary, until all remaining items were deemed relevant. Two algorithms were developed: one for toddlers who were verbal and aged 21–30 months, and the other for younger, less-able toddlers. Internal consistency was good for the social affect scale for both groupings, and poor for RRBs. Test–retest reliability was not high (ICC = 0.6) for RRBs for the verbal toddler algorithm (and n = 8), but otherwise good (0.83–0.94). Inter-rater reliability was high, but the paper was judged to be of poor quality for this property, as the report was for the agreement of seven raters and 14 videos of assessment.

Autism Diagnostic Observation Schedule-Calibrated Severity Score

Refinement of the ADOS algorithm scores led on to the development of the ADOS-Calibrated Severity Score (ADOS-CSS). 77 The ADOS-CSS potentially allows for greater understanding of the manifestation of core autism symptom severity over time, independently of factors such as age, IQ and language level. Gotham et al. 77 tested the hypothesis that severity scores would be less related to factors such as IQ than the raw scores, and this was found. Two studies78,79 have examined the ADOS-CSS in independent samples, with somewhat mixed findings. De Bildt et al. 78 found in a large clinical Dutch sample that CSS discriminated the autism, non-autism ASD and non-spectrum classifications well, and were more comparable over various developmental groupings than the raw scores on the ADOS, especially in module 1 and somewhat less so in module 3. For module 2, the larger proportion of children with non-autism ASD relative to the Gotham sample probably contributed to differences in findings. Shumway et al. 79 examined whether or not calibrated severity scores were independent of other factors. They found that a regression model accounted for 56% of the variance in ADOS raw score, but for only 18% of the variance in calibrated severity score, i.e. independent of verbal and non-verbal developmental quotient. In addition, they found good stability of scoring after an interval of 12–24 months.

Autism Observation Scale for Infants

The Autism Observation Scale for Infants (AOSI)80 was developed to detect and monitor early signs of autism as they emerge in high-risk infants. It is an 18-item direct observational measure designed to detect and monitor putative signs of autism in infants aged 6–18 months. Data on inter-rater reliability was good, test–retest reliability less so, but the sample size was only 2081 and thus judged to be of poor methodological quality. Georgiades et al. 82 found good discrimination between high- and low-risk infants (i.e. infant siblings of children with ASD vs. no ASD).

Baby and Infant Screen for Children with aUtIsm Traits-Part 1

The Baby and Infant Screen for Children with aUtIsm Traits-Part 1 (BISCUIT-Part 183) is designed to assess symptoms of ASD in children between the ages of 17 and 37 months. It comprises 62 items scored on a three-point, Likert-type scale. Parents are asked to rate the child on each item, comparing them to a typically developing child as ‘0’ (not different; no impairment), ‘1’ (somewhat different; mild impairment) or ‘2’ (very different; severe impairment). For the factor analysis study,84 405 infants with a diagnosis of ASD were selected from a total of 1287 enrolled in a US state-funded early intervention programme for children at risk for a developmental disability. In factor analysis, a three-factor structure (socialisation, repetitive behaviour, communication) accounted for only 33% of the variance (with seven items that did not load on to any factor) but the internal consistency of the factors was good [alpha (α) = 0.93, 0.90, 0.87, respectively]. Factor scores were significantly higher than for infants without ASD. Matson et al. 85 demonstrated convergent validity of the BISCUIT-Part 1 with the Modified Checklist for Autism in Toddlers (M-CHAT) and the Personal–Social domain of the Battelle Developmental Inventory-Second Edition, and divergent validity with the Battelle Adaptive and Motor domains.

Behavioral Summarized Evaluation and Behavioral Summarized Evaluation-Revised

The Behavioral Summarized Evaluation (BSE)86 is a 20-item instrument that examines the scope and severity of behaviour problems in autistic children. Items are rated by a clinician on a five-point scale ranging from 0 (never) to 4 (continuously). A global score can be obtained by summing the 20-item scores. A revised version later added nine items. 87 Four papers considering measurement properties of the BSE were reviewed. One study88 assessed internal consistency and found that Cronbach’s alpha coefficients ranged from 0.83 to 0.90. However, the small sample size included in the unidimensionality analysis led this study to be judged as methodologically poor (20 items, 53 participants). Two studies assessed inter-rater reliability86,87 and found ICCs for the global score to range from 0.96 to 0.97. However, the small sample size (n = 29) led one of these studies87 to be judged as having poor methodological quality. The content validity was good, having been developed over several iterations in practice, with correlation of the global score or first factor with IQ and not with age. 86,87 Four studies86–88,89 assessed structural validity; however, the small sample size of two studies86,88 led them both to be judged as having poor methodological quality. Construct validity in the remaining studies was not strong overall; two main factors were found together accounting for almost 50% of the variance,86–88 with the first factor, labelled ‘autism’ or ‘interaction disorder’, relatively consistent, but the second factor very variable. Roux et al. 89 further examined the structure of the first factor and found it to account for 61% of the variance. One study88 assessed convergent validity, and found correlations between this tool and the Childhood Autism Rating Scale (CARS) to all be > 0.77. One study assessed known-groups validity87 and found all relevant hypotheses supported. This same study assessed criterion validity87 and found partial correlations between this tool and an expert clinical rating to range from 0.24 to 0.63 but the quality of the evidence was judged to be poor.

Infant Behavioral Summarized Evaluation

The Infant Behavioral Summarized Evaluation (IBSE) is an observational rating scale adapted from the BSE for the assessment of behaviours of young children having autistic disorders. The original paper90 describing the development and measurement properties of the IBSE was reviewed. Eighty-nine children aged 6–48 months, referred for clinical assessment, were included. The study assessed inter-rater reliability of the initial 33 items and found an ICC of 0.97. The same study assessed the tool’s structural validity: 59.4% of total variance was explained by a two-factor solution, with 19 items constituting the first factor labelled ‘autism’. However, the small sample size (89 participants and 31 reliable items) led this paper to be judged as having poor methodological quality.

Childhood Autism Rating Scale

The CARS91,92 is a behavioural rating scale that is widely used in the diagnosis of children with autism and pervasive developmental disorders. The CARS is a 15-item observation and parent interview measure that quantifies the severity of behaviours associated with autism. Items are rated on a scale from 1 (‘normal’) to 4 (‘severely abnormal’). Total scores ≥ 30 strongly suggest the presence of autism. In a range of small studies (see Appendix 8), internal consistency of the total score was found to be good, as it was in one large study in India. 93 However, Magyar et al. 94 conducted principal components analysis, finding four factors with only one alpha coefficient of > 0.70: social communication (0.78), social interaction (0.61), stereotypies and sensory abnormalities (0.54), and emotional regulation (0.59). At the item level, average inter-rater reliability was r = 0.71, range for the items 0.55–0.93 (only one was > 0.80). 91 However, for the total score, inter-rater reliability was reported to be good: ICC = 0.74;93 ICC = 0.73. 95 Test–retest reliability after 1 year, in children referred for ASD assessment, was high (ICC = 0.81). 93 A number of different factor solutions have been proposed. Stella et al. 96 reported five factors accounting for 64% of the variance: emotional reactivity, social communication, social orienting, odd sensory exploration, and cognitive and behavioural consistency. The four-factor structure reported by Magyar et al. 94 accounted for only 41% of the variance. Stella et al. 96 examined convergent and divergent validity of the factor scores in relation to the VABS but did not find the hypothesised pattern of correlations.

Gilliam Autism Rating Scale and Gilliam Autism Rating Scale-Second Edition

The Gilliam Autism Rating Scale (GARS) is a behavioural checklist developed for use by parents, teachers and professionals to discriminate individuals who are autistic from those with other developmental disabilities. It is intended for use with individuals aged from 3 to 22 years. The GARS has 56 items, divided into four scales; Social interaction, Communication, and Stereotyped behaviours are rated on a four-point scale of frequency, and Developmental disturbances rates early milestones on a dichotomous scale. The summary score is the Autism Quotient. South et al. 97 raised concerns about the capacity of the scale to detect autism in a sample of 119 children aged 3–10 years with strictly defined autism, finding that the mean Autism Quotient was significantly lower than the reference of 100. Lecavalier98 raised similar concerns with a broader sample aged 3–21 years. Furthermore the exploratory factor analysis (EFA) of the first three scales accounted for only 37% of the variance. Internal consistency of those three scales was good; the Developmental disturbances scale was lower, with Cronbach’s α = 0.68. However, parent–teacher inter-rater reliability was low (ICC average = 0.40). Pandolfi et al. 99 examined the GARS-Second Edition (GARS-2), a revision and normative update,100 which has very similar content in the main three scales. The EFA accounted for 34.1% of the variance and the factor model was not entirely consistent with the conceptually derived organisation of the GARS-2. A four-factor model was preferred, for which scale reliability estimates were good.

Modified Checklist for Autism in Toddlers

The M-CHAT was designed as a screening tool with 23 ‘yes/no’ items that can be given to parents by clinicians, with a focus on 18–24 months of age. It does not rely on the professional’s observation of the child, but on parents’ report of current skills and behaviours. In the original study,101 internal consistency for the whole scale (α = 0.85) and for six critical items found on discriminant function analysis (α = 0.83) was good. Snow et al. 102 reported internal consistencies of 0.80 and 0.74, respectively, in a sample of clinically referred 18- to 48-month-old children. However, both papers were judged to be of poor quality for this property, as the unidimensionality of the scale was not checked. Snow et al. 102 used the Social Communication Questionnaire (SCQ) to assess criterion validity and found a correlation of 0.77. Inada et al. 103 tested inter-rater reliability (mother–father pairs, r = 0.93) and test–retest reliability (r = 0.99) after a mean of 8 days in a Japanese translation; however, the paper had a small sample and so was judged to be poor.

Parent Observation of Early Markers Scale

The Parent Observation of Early Markers Scale (POEMS) is a new parent report instrument to monitor prospectively the behavioural development of infants at risk for ASD. 104 The target age is 1–24 months, and the development study involved 108 infants. POEMS includes 61 items that are rated on a four-point scale. Internal consistency was good at each of six age groupings. Test–retest reliability over a 1-month period was checked at 11 different age groupings and was high (with one exception); however, the evidence is of poor quality given small sample sizes. Convergent and divergent validity were established for the POEMS through correlations with domains of the Ages and Stages Questionnaire. 105 Relationships with the ASQ were stronger with the core features of ASD (social and communication problems) than with gross motor problems; however, the correlations with social and communication domains were only –0.41 and –0.45, respectively.

Pervasive Developmental Disorders Rating Scale

The Pervasive Developmental Disorders Rating Scale (PDDRS)106 is a rating scale designed to assist in the screening and diagnostic process for autistic disorder. It contains 51 items, which comprise three subscales: Arousal, Affect and Cognition. Items are rated by a parent (or teacher who has known the individual for at least 2 months) on a five-point Likert scale according to the degree of severity of the behaviour described. Two papers were evaluated. 42,107 Williams and Eaves107 reported on 456 participants with a diagnosis of ASD ranging in age from 1 to 12 years (as well as 111 adolescents and young adults). Ratings by the same teacher at a mean interval of 9.5 months, with 62.5% of the ratings having a ≥ 6-month time gap, were used to assess test–retest reliability; reliability coefficients ranged from 0.86 to 0.92 for subscales, and 0.92 for the total score. Internal consistency was also good, with subscale Cronbach’s alpha coefficients ranging from 0.75 to 0.86, and total score of 0.89. The second paper42 considered teacher ratings of 168 children aged 1–12 years. EFA found three factors, accounting for 64% of the variance, with internal consistency at least 0.80; however, because of the low sample size in relation to number of items, the evidence was judged to be poor.

Real Life Rating Scale

The Real Life Rating Scale (RLRS)108 is a behavioural rating scale used in the diagnosis of autism; in comparison with other such scales it is noted to place emphasis on disturbances in response to sensory stimuli. The 47 items of the RLRS are completed by trained assessors, for example in Sturmey et al. ,61 based on clinical assessment observations of children during a 30-minute free play period. Sturmey et al. 61 examined internal consistency; this was good for the total score (α = 0.84) but poor for the subscales, ranging from 0.42 to 0.68. However, the sample was small – 34 children and adolescents with a diagnosis of ASD – and so the evidence was judged to be poor. Similarly, Sevin et al. 109 reported on inter-rater reliability for a small sample of 24 children and adolescents. Reliability was poor for the 40 items observed, with mean agreement-level kappa = 0.31, and the highest-item kappa = 0.64 (disturbs others).

Social Communication Questionnaire

The SCQ (originally called the Autism Screening Questionnaire)110 is a 40-item questionnaire based on the ADI-R,63 which enquires about characteristic autistic behaviours. Parents are asked to indicate whether or not their child shows a particular symptom (current), and whether they did so at age 4–5 years (lifetime). Language items not suitable for non-verbal children can be omitted. Scores are out of a total of 39 or 32, depending on a child’s language level, with higher scores indicating more severe symptoms.

The factor-based and total scores evidenced good-to-excellent scale reliability using confirmatory factor analysis parameters (factor loadings and error variances) in children with Down syndrome. 111 Snow et al. 102 also found good internal consistency for total score (α = 0.81); however, for the domain scores internal consistency was not good (Reciprocal Social Interaction domain 0.70; Communication 0.47; and Restricted, Repetitive, and Stereotyped Patterns of Behaviour 0.76). Magyar et al. 111 conducted EFA, which suggested a two-factor solution accounting for 54.4% of the variance: social communication, and stereotyped behaviour and unusual interests. Some evidence of criterion validity was found; for example, Magyar et al. 111 showed agreement of the two factor scores with corresponding domains in the ADI-R.

Charman et al.,112 in a longitudinal study, aimed to compare the utility of three scales to measure developmental change in children’s profiles over time. On the SCQ (current behaviour) there was no change in 57 children with ASD (aged < 6 years at the start), whereas on the VABS-Screener version (VABS-Screener) the children gained 9 months equivalent in 11 months on the Socialisation scale, and gained 10 months on the Communication scale. Therefore, evidence suggests that the SCQ does not detect change in measurement of autism characteristics when reduction in severity might have been expected.

Social Responsiveness Scale

The Social Responsiveness Scale (SRS)113,114 is a 65-item questionnaire which takes 15–20 minutes to complete by parents, teachers or other adults who routinely observe the child in a naturalistic social setting. The item content of the two versions (36–48 months and 4–18 years) differs only on the basis of developmental appropriateness of the wording for rating the behaviours of children in the respective age groups, therefore they are considered together. Factor analyses support a one-factor solution; for example, principal components analysis revealed a primary factor explaining > 30% of the variance, with five factors accounting for 49% of the variance in total. 114 Good internal consistency115,116 has been found for the total score. However, because of poor goodness of fit (e.g. differing relevance of items across ages), Duku et al. 116 have since explored a 30-item version that correlates strongly with the 65-item scale (r = 0.94). For reliability, findings were mixed. For the 36–48 months version, Pine et al. 117 reported reasonable maternal SRS test–retest reliability (r = 0.74) measured at variable intervals (6, 24 and 42 weeks) in a mixed sample; however, Bolte et al. 115 reported r = 0.97 in an older clinical sample, including children with ASD. Agreement between parents and teachers was found to be reasonable: r = 0.78;118 ICC = 0.66,117 whereas mother–father agreement in an older clinical sample was reported to be high115 (0.97). Good convergent and divergent validity have been shown, for example, with the Child Behavior Checklist (CBCL),115,116 and Bolte et al. 115 also showed good discrimination between groups at a total and at item level. For criterion validity, correlations even with the SCQ are moderate (r = 0.58) and lower with domains of the ADI-R (e.g. r = 0.46 with the Social domain).

Autism Treatment and Evaluation Checklist

The Autism Treatment and Evaluation Checklist (ATEC)119 is a 77-item, one-page checklist designed to be completed by parents, teachers and/or primary caretakers of children with ASD and to measure response to treatment. Three papers considering measurement properties of the ATEC were reviewed. One study120 assessed internal consistency and found Cronbach’s alpha coefficients for the four subscales to range from 0.86 to 0.96. Owing to the small sample size (n = 22), this paper was judged to be of poor methodological quality. Two studies120,121 assessed convergent validity and found correlations between the ATEC and the British Picture Vocabulary Scale-II to range from –0.53 to –0.63, correlations with the Expressive One-Word Picture Vocabulary Test to range from –0.60 to –0.67, correlation with the VABS composite to range from –0.79 to –0.88, correlations with ADI-R total raw score to range from 0.82 to 0.88,120 and correlation with the CARS of 0.71. 121 Two studies112,120 assessed responsiveness. Charman et al. 112 found 50% of their hypotheses regarding change scores to be supported, whereas clear evidence of change was found for all subscales of the VABS-Screener (see Table 11, below). Magiati et al. 120 also found that 50% of ATEC scales changed ATEC total scores at age 4–6 years significantly predicted the extent of progress made 5–6 years later. However, there were large individual differences in ATEC score changes over time.

Pervasive Developmental Disorders Behavior Inventory

The Pervasive Developmental Disorders Behavior Inventory (PDDBI) is an informant-based questionnaire that is designed to assess responsiveness to intervention in children diagnosed with ASD. The PDDBI items are organised into six maladaptive and four adaptive scales, with parent (176 items) and teacher (144 items) versions. The subscales independently address different types of behaviours, so that each subscale can be used separately or as part of the entire inventory. This is to enable researchers to assess, more frequently, behaviours that may change over the short term (e.g. stereotypies or aggressiveness), but, less frequently, other behaviours that would be expected to change over a longer time span (e.g. non-verbal prosocial skills or expressive language skills). The PDDBI development paper122 described a comprehensive establishment of content validity from an initial large item pool and field testing. The authors then reported questionnaires completed by 311 parents of children with ASD between the ages of 1 and 17 years (mode 5 years). Alpha coefficients ranged from 0.79 to 0.97 in the parent version, and from 0.73 to 0.97 in the teacher version. Inter-rater reliability (between teachers, and between teachers and parents) was good for the verbally mediated subscales: Learning, Memory, and Receptive language; Phonological skills; and Semantic/pragmatic ability. Differences in agreement were most obvious for the Sensory/perceptual approach behaviours, Aggressiveness, and Social approach behaviours subscales. Principal components analysis of all subscales resulted in two factors together accounting for 65% of the variance. Separate factor analyses within subscales mostly found the predicted factor structure. Cohen123 found significant correlations with the ADI-R subdomain Current behaviour scores, but all were < 0.60.

Communication and Symbolic Behavior Scales-Developmental Profile-Behavior Sample

The Communication and Symbolic Behavior Scales-Developmental Profile (CSBS-DP)124,125 is a standardised tool for the assessment of communication and symbolic abilities of children in their second year of life. It consists of three measures: 24-item Infant–Toddler Checklist; Caregiver Questionnaire; and Behavior Sample, which is a face-to-face evaluation of the interaction between a child and parent, and clinician. Those three measures aim to assess a range of social, speech and symbolic skills. The one paper identified in stage 3126 investigated the inter-rater reliability of the CSBS-DP-Behavior Sample, and reported g coefficients ranging from 0.92 to 0.97 for the composites and total score. However, a small sample was used (20% of the data) so the evidence is judged to be poor.

Early Social Communication Scales

The Early Social Communication Scales (ESCS)127,128 measures non-verbal social communication, through rating by a trained investigator of directly observed skills in social interaction, joint attention and behaviour regulation, in children up to 30 months of age. The live scoring [Early Social Communication Scales Live (ESCS-L)] is an abbreviated version of the original ESCS coding scheme. 128 Inter-rater reliability was calculated in the one paper reviewed129 and the average percentage agreement was 88.3; however, only a proportion of the sample was used in this analysis so the evidence was judged to be poor.

Imitation Battery

The Imitation Battery (IB)130 examines imitation skills in children, including those diagnosed with ASD. Luyster et al. 129 presented a nine-item battery of manual actions, oral–facial actions and actions on objects to 164 toddlers with ASD, aged 18–34 months. Inter-rater reliability of the IB was reported as 97.2% agreement but the sample size (10%) was small. Young et al.,131 who used a 10-item battery, reported inter-rater reliability with a mean weighted kappa statistic of 0.84. This study also found, as hypothesised, lower imitation abilities in the ASD group than in typically developing children, but not in the group with other developmental delays.

Imitation Disorders Evaluation scale

The Imitation Disorders Evaluation (IDE) scale132 is a nine-item clinical scale evaluating atypical imitation in infants and young children with autism. Items are rated by a trained observer on a five-point Likert scale, ranging from 0 (behaviour never observed) to 4 (behaviour always observed). In the Malvy et al. 132 development paper, insufficient information is given on both inter-rater reliability (kappa statistics ranging between 0.4 and 1) and structural validity (72.9% of the total variance accounted for) to draw conclusions about the IDE scale measurement properties.

Motor Imitation Scale

The Motor Imitation Scale133 was developed as a structured imitation assessment for children with ASD. It includes 16 tasks, split equally between object and body imitation tasks, half of those involving meaningful and half non-meaningful actions. Items are rated by a trained investigator on a three-point scale, with a ‘0’ score when there is no imitation, ‘1’ for an emerging response and ‘2’ for exact imitation. Ingersoll and Meyer134 investigated the relationship between imitation and other social-communication skills in 27 children with autism, average age 38.7 months. After controlling for developmental level, the total imitation score was found to be significantly and positively correlated with expressive vocabulary (r = 0.36); however, the sample size used in the study was small so the evidence was judged poor.

Preschool Imitation and Praxis Scale

The Preschool Imitation and Praxis Scale (PIPS)135 is a 30-item observational scale with 10 task categories (six gestural, three procedural and one facial) assessing imitation performance in young children. In the development paper, Vanvuchelen et al. 135 reported good overall internal consistency (Cronbach’s α = 0.97) and a four-factor structure explaining 66.6% of the variance. Also they found positive and strong associations (amid r = 0.59 and 0.74) between the PIPS score and scores on language and motor measures in children with ASD. The second paper reviewed on the measurement properties of the PIPS136 demonstrated excellent inter-rater reliability for the scale (ICC = 0.986) and investigated the smallest detectable difference for the scale.

Social Communication Assessment for Toddlers with Autism

The Social Communication Assessment for Toddlers with Autism (SCATA)137 is a semistructured observational tool eliciting social communication behaviours in young children with ASD with an unfamiliar adult. Four dimensions of communicative act are scored: form, function, communicative role and complexity. Reliability was found to be excellent for the total number of communication acts (ICC = 0.93). Frequency of communication over time was stable in the study, with greater communication difficulties shown by children with more severe diagnosis. The early social communication scores were also associated with later language scores. However, the paper is judged to be of poor quality because of the small sample.

Autism Diagnostic Interview-Revised

A total of 20 eligible papers assessed the ADI-R Restricted and Repetitive Behaviours and Interests domain. Internal consistency was assessed in five papers;63,64,68,138,139 none reached the COSMIN cut-off for internal consistency (Cronbach’s α > 0.70) with the exception of the Snow et al. 68 paper, for which Cronbach’s alpha coefficient was 0.70 for verbal children but not non-verbal children (0.61). Test–retest reliability was assessed in two papers63,140 but both were of poor quality. Inter-rater reliability was acceptable in three papers. 65,67,69 In all three papers65,67,69 the raters were trained and monitored to maintain quality and consistency of ratings. Lord et al. 63 and Moss et al. 71 also assessed inter-rater reliability but the studies were of poor methodological quality due to small sample size.

For content validity, Lord et al. 63 selected items that most closely resembled clinical descriptions and diagnostic guidelines from DSM-IV and ICD-10. By inspection of the face validity of constructs generated by factor analysis, Tadevosyan-Leyfer et al. 66 demonstrated that the ADI-R had good content validity. Thirteen papers,63,67,68,74,138,141–148 of varying quality, assessed structural validity of the Restricted and Repetitive Behaviours and Interests domain with conflicting results. Four papers68,138,141,142 provided results that supported the structural validity of the Restricted and Repetitive Behaviours and Interests domain of the ADI-R. Generally, a two-factor structure was found to be the best fit, with the factors labelled ‘insistence on sameness’ and ‘repetitive sensory and motor behaviour’. Three papers of good quality did not provide enough information to assess structural validity. 143–145 However, three good-quality papers67,146,147 and one paper,148 judged as being of excellent quality, did not support the structural validity of the ADI-R Restricted and Repetitive Behaviours and Interests domain. In Frazier et al. 67 none of the factor solutions produced acceptable model fit, as RMSEA did not reach the < 0.06 cut-off or the Tucker–Lewis fit index > 0.95 cut-off. Lecavalier et al. 146 reported a three-factor solution accounting for just 38% of the variance. Similarly, Szatmari et al. 147 reported that a two-factor solution accounted for just 36% of the variance. Finally, the excellent quality paper148 reported a two-factor solution accounting for 43% of the variance. Convergent and/or divergent validity was supported in all eight relevant papers. Both Lord et al. 63 and Grey et al. 74 showed that ASD and non-ASD groups differed significantly on RRB scores.

Responsiveness (in effect, stability) of the ADI-R was supported in two papers,65,71 both of fair methodological quality. In the Moss et al. 71 study, there was no statistically significant change in the number of participants (n = 35, average age 3.5 years) meeting autism criteria on the ADI-R at follow-up after 7 years. Lord et al. 65 also demonstrated that ADI-R ‘ever’/lifetime scores for restricted and repetitive behaviours and interests (RRBI) were higher at the age of 9 years than at 2 years, as expected, and that mean ‘current’ scores showed a marked reduction.

Autism Diagnostic Observation Schedule-Generic

Seven papers assessed one or more measurement properties of the ADOS-G in relation to RRBI. Lord et al. 50 assessed internal consistency, test–retest reliability and structural validity of modules 1–3; however, the methodological quality was poor for all assessments due to small sample sizes for each module. Inter-rater reliability did not reach the COSMIN criterion in Lord et al. 65 for modules 1 and 2.

Support for structural validity was excellent for module 1 but not for module 3, as none of the proposed models produced model fit statistics that satisfied the COSMIN criteria. 73 Kamp-Becker et al. 72 assessed the structural validity of ADOS modules 3 and 4 together, and showed that a four- and five-factor solution explained 52% and 57% of the variance, respectively. In the five-factor solution, factor 4 ‘stereotyped behaviour’ and factor 5 ‘interests and compulsions’ were relevant to the Repetitive and Restricted Behaviour domain of functioning.

Criterion validity was supported by Chawarska et al. 70 for module 1, with 91% of cases matching clinician-assigned diagnosis of autism. Grey et al. 74 provided good support for criterion validity in both modules 1 and 2 with high agreement between ADOS and clinical diagnosis (κ = 0.70; p < 0.001) and significant differences between ASD and non-ASD groups in the Repetitive and Restricted Behaviour domain.

Finally, three papers,65,70,149 all judged as of fair methodological quality, supported the responsiveness of ADOS-G. Scores in the Stereotypic Behaviours domain were shown to be stable, as expected, over a period of 15 months. 70 The ADOS-G was able to measure improvement in functioning over time in Ben Itzchak et al. 149 and change scores for ADOS-G and ADI-R gave similar findings in Lord et al. 65

Autism Diagnostic Observation Schedule-Toddler Module

The development paper76 for the ADOS-Toddler Module reported poor internal consistency for the Repetitive and Restricted Behaviours domain (Cronbach’s α = 0.50). Test–retest reliability was moderate (ICC = 0.6) for RRBs for the verbal toddler algorithm but otherwise good. Inter-rater reliability was high, but the paper76 was judged to be of poor quality for this property, as the measurement was for the agreement of seven raters and 14 videos of assessment. Luyster et al. 76 also reported that numerous drafts and pilot analyses for content validity were conducted, and items removed and added as appropriate.

Repetitive Behavior Scale-Revised

The Repetitive Behavior Scale-Revised150 is a 43-item questionnaire designed to assess problem behaviour and was revised from the original RBS to tap into some of the complex RRBs observed in people with autism. The questionnaire is completed by parents/caregivers. The items have been conceptually grouped into six subscales: (1) Stereotyped behaviour; (2) Self-injurious behaviour; (3) Compulsive behaviour; (4) Ritualistic behaviour; (5) Sameness behaviour (insisting that things stay the same); and (6) Restricted behaviour/interests.

Both Lam et al. 151 and Mirenda et al. 152 provided evidence of good internal consistency for the overall score and for all subscales. Lam et al. ’s151 assessment of test–retest reliability was judged to be poor because of a small sample size. Structural validity was not supported, although in both cases statistical tests fell just below the COSMIN cut-offs. In Lam et al.,151 47.5% of the variance was explained by a four-factor solution. Five- and six factor solutions provided a good fit to the data in Mirenda et al.,152 with RMSEA = 0.064, just missing the COSMIN cut-off of RMSEA = 0.06. Convergent validity was supported by both Mirenda et al. 152 and Lam et al. 151

Sense and Self-Regulation Checklist

The Sense and Self-Regulation Checklist (SSC)153 is a 65-item caregiver questionnaire of children’s sensory and self-regulatory difficulties, rated on a four-point Likert-type scale, ranging from ‘0’ (never) to ‘3’ (often). It contains six sensory subdomains (Touch–Pain; Auditory; Visual; Taste–Smell; Hyper-reactive to non-injurious stimuli; Hyporeactive to injurious stimuli) and six self-regulatory subdomains (Sleep, Appetite–Digestion, Self-soothing, Orienting–Attending, Aggressive behaviour, Self-injurious behaviour). Only one paper153 considering measurement properties of the SSC was reviewed and used the SSC data from 265 children (including 99 children with ASD) aged < 6 years. The study153 reports good internal consistency (α = 0.87 for total) and acceptable test–retest reliability at 4 months interval (r = 0.68) and shows strong relationships between sensory and self-regulation impairment and severity of autism (however, only a subsample of 38 parents was used in the reliability study). The study showed predicted differences between children with ASD, and developmentally delayed and typically developing children, on the SSC.

Sensory Profile

The Sensory Profile (SP)154 is a caregiver questionnaire that measures a child’s sensory processing abilities. The questionnaire consists of 125 items, rated on a five-point Likert scale, ranging from almost never to almost always. The measure is divided into three main sections – Sensory Processing, Modulation, and Behavioural and Emotional Responses – and 14 sensory-processing categories. Children can also be classified as fitting into one of the four general sensory processing ‘quadrants’: sensation seeking, sensation avoiding, sensory sensitivity and low registration. The SP can be used with 3- to 10-year-olds. The reviewed paper155 reported good known-groups validity of the SP between children with autism and typically developing children in the Australian sample.

Short Sensory Profile

The Short Sensory Profile (SSP)154 is a 38-item, five-point Likert scale caregiver questionnaire intended to assess sensory processing and sensory systems. The questionnaire consists of seven factors: tactile sensitivity, taste/smell sensitivity, movement sensitivity, seeking sensation, auditory filtering, low energy levels, and visual/auditory sensitivity. Two papers156,157 looked at measurement properties of the SSP; however, in one of them,156 an adapted version of the SSP was used, with 10 additional items from the SP,154 and seven additional items from the researchers’ clinical experience. In that study,156 the information provided is not sufficient to determine the rating for internal consistency; however, it is reported that the SSP successfully classified 80.9% of the cases correctly among typically developing, learning-disabled children and those with autism. Wiggins et al. 157 found support for hypotheses that children with ASD show more sensory abnormalities than children diagnosed with developmental delays, and that sensory abnormalities are associated with stereotyped interests and behaviours as measured by ADOS.

Comprehensive Assessment of Spoken Language

The Comprehensive Assessment of Spoken Language (CASL)158 is a direct assessment of oral language skills in four areas: lexical/semantic, syntactic, supralinguistic and pragmatic. The subtests in the CASL can be either be administered individually or a total score can be obtained. In the Reichow et al. study,159 six specific CASL subtests were examined: Nonliteral Language, Pragmatic Judgment, Antonyms, Syntax Construction, Paragraph Comprehension and Inference. The study159 showed significant correlations between the Pragmatic Judgment and Inferences CASL subtests and the VABS Communication and Socialisation domains (r = 0.45; r = 0.62, respectively), suggesting that those two CASL subscales are not acceptable measures of language skills in individuals with ASD, as the correlation values were below COSMIN cut-offs.

MacArthur–Bates Communicative Development Inventories

The MacArthur–Bates Communicative Development Inventories (MCDI)160,161 is a parent report of children’s early language skills. It consists of two forms: ‘Words and Gestures (Infant)’ and ‘Words and Sentences’. The former is an assessment of vocabulary comprehension, vocabulary production and use of gestures in infants between 8 and 16 months. The latter measures vocabulary production, sentence complexity, grammatical development and the mean length of the child’s three longest utterances, in children between 16 and 30 months of age. Bruckner et al. 162 performed differential item functioning analysis and reported items that weakened the validity of the MCDI-Infant when scores of typically developing infants were compared with those with ASD. Luyster et al. 129 investigated associations between different measures of early language in toddlers with ASD, including both forms of MCDI. They reported high correlations between the MCDI, Mullen Scales of Early Learning (MSEL) and VABS (correlations between the MCDI receptive language scores and MSEL r = 0.52, and VABS r = 0.77; correlations between the MCDI expressive language scores and MSEL r = 0.82, and VABS r = 0.88).

Mullen Scales of Early Learning

The MSEL163 is a developmental test for young children aged 0–69 months assessing visual reception, receptive language, expressive language and fine motor skills. One study164 investigated the measurement properties of the MSEL. Burns et al. 164 found support for the hypotheses that children with developmental delays would present significantly more difficulties regarding expressive and receptive language skills than typically developing children matched for age, race and gender. Also the authors found that children with ASD were more likely to exhibit impairment in receptive language skills than children diagnosed with cerebral palsy.

Preschool Language Scale-Fourth Edition

The Preschool Language Scale-Fourth Edition (PLS-4)165 is an assessment of language skills identifying children with language disability. It can be used with children up to 7 years of age and provides a total language score, auditory comprehension and expressive communication scores. Volden et al. 166 investigated the relationship between language skills and both autism severity and adaptive communicative behaviour. The authors166 reported a small correlation with ADOS scores (r = –0.12) and a strong correlation with the Vineland Adaptive Behavior Scales-Second Edition (Vineland-II) Communication domain (r = 0.75).

Vineland Adaptive Behavior Scales

The VABS167 consist of a semistructured interview administered to the parent/carer or an adult with detailed knowledge of the functioning of the child. The ratings assess adaptive behaviour in four main domains: Communication, Daily Living Skills, Socialisation, and Motor skills (the last domain, however, is measured only for children of < 6 years of age). Also, the VABS includes a Maladaptive Behavior Scale. All of the items are rated on a three-point Likert scale, ranging from ‘0’ (seldom or never present) to ‘2’ (always present). One study168 showed that 20 children with ASD (average age 47.4 months) made significant developmental progress in the Communication Skills domain of the VABS, from baseline to year 1 and year 2; however, the small sample means that the evidence was judged to be poor. Paul et al. 169 investigated differences between 20 children with autism and 20 diagnosed with PDD-NOS, aged 4–11 years. The authors found that group differences were observed only in very specific areas, but their hypothesis was supported with regard to use of expressive language.

Vineland Adaptive Behavior Scales-Classroom version

The VABS-Classroom version (VABS-Classroom) is a 244-item questionnaire that aims to assess adaptive behaviours in a classroom environment and is usually completed by teachers. Wells et al. 170 investigated the relationship between autism severity and adaptive behaviour, including communication skills. They reported correlations between autism severity (measured by CARS) and VABS-Classroom receptive language (r = –0.27, not significant) and expressive language (r = –0.55).

Vineland Adaptive Behavior Scales-Screener version

The VABS-Screener version (VABS-Screener)171 is an adaptation of the VABS, with 45 items rated from ‘0’ (when a child does not demonstrate behaviours) to ‘2’ (when a child usually demonstrates a behaviour). Charman et al. 112 examined developmental change in children’s profiles over 11 months. The sample of 40 children gained 10 age-equivalent months in communication skills. Also paired t-tests indicated that age equivalents at time 2 were significantly higher than at time 1 for communication skills.

Leiter International Performance Scale-Revised

The Leiter International Performance Scale-Revised (Leiter-R)172 was designed to assess non-verbal cognitive ability in people with a variety of language complications. It is divided into two test batteries that include 10 subscales each. The Visualisation and Reasoning battery is used to obtain a composite IQ. The Attention and Memory subscales are used to evaluate deficits in Attention or Memory domains. Three papers173–175 considering measurement properties of the Leiter-R were reviewed. The tool’s convergent validity was assessed by two studies. 173,174 The first173 found the Leiter-R to be moderately correlated with the Kaufman Brief Intelligence Test-Second Edition (r = 0.62). Tsatsanis et al. 174 found positive correlations (r ranged from 0.73 to 0.97) between the original Leiter International Performance Scale and Leiter-R IQ scores. However, the small sample size of this study (n = 26) led this paper to be judged as having poor methodological quality. One study175 assessed the tool’s known-groups validity, and found that only two of the study’s four hypotheses were supported. The tool’s criterion validity was assessed by Grondhuis and Mulick,175 who found the Leiter-R to be moderately correlated (ICC = 0.66) with the non-verbal Stanford–Binet Intelligence Scales-Fifth Edition (SB5).

Mullen Scales of Early Learning

The MSEL163 are a comprehensive measure of cognitive functioning in young children, and consist of four scales: Visual reception, Receptive language, Expressive language and Fine motor. Two papers164,176 considering measurement properties of the MSEL when used with children with ASD were reviewed. Burns et al. 164 assessed known-groups validity (their sample included 19 children with ASD) and found support for only one of the study’s two hypotheses. Bishop et al.,176 in a sample of 59 children with ASD, assessed criterion validity and found correlations between the Differential Ability Scales and the MSEL Non-verbal IQ scores to equal 0.74, and MSEL Verbal IQ scores to equal 0.83.

An Early Learning Composite of the MSEL (MSEL-Early Learning Composite) can be calculated based on scores from four scales for children aged 0–69 months. One paper82 considering the measurement properties of the MSEL-Early Learning Composite was reviewed. This study82 assessed known-groups validity and found all relevant hypotheses to be supported.

Stanford–Binet Intelligence Scales-Fifth Edition

The SB5177 evaluates general intellectual abilities. The full-scale IQ is derived from the Verbal and Non-verbal scales, each with five subtests bearing the same names: Fluid reasoning, Knowledge, Quantitative reasoning, Visual spatial processing and Working memory. One paper,175 considering the measurement properties of the SB5, was reviewed. Grondhuis and Mulick175 assessed known-groups validity and found that only two of the study’s four hypotheses were supported. This same study175 also assessed convergent validity and found the non-verbal SB5 to be moderately correlated (ICC = 0.66) with the Leiter-R.

Wechsler Preschool and Primary Scale of Intelligence-Revised

The Wechsler Preschool and Primary Scale of Intelligence-Revised (WPPSI-R)178,179 assesses the intelligence of children aged between 3 and 7 years. One paper180 considering the measurement properties of the WPPSI-R was reviewed. Yang et al. 180 assessed responsiveness and found R-squared for change in IQ from time 1 to time 2 was 0.50 in the total sample, and 0.37 in the ASD sample only.

Behavior Assessment System for Children-Second Edition

The Behavior Assessment System for Children-Second Edition (BASC-2)181 is a widely used tool for assessing behaviour and emotions in children, adolescents and young adults, ranging in age from 2 to 25 years old. The BASC-2 consists of a Structured Developmental History, an Observation System, a Parent Rating Scale, a Self-Report of Personality Scale and a Teacher Rating Scale. Two papers182,183 considering measurement properties of the BASC were reviewed. Hass et al. 182 found internal consistency of the Teacher Rating Scale only, as measured by Cronbach’s alpha, to range from 0.66 to 0.85 for the Attention problems subscale. The absence of a factor analysis, however, led this paper to be judged to be of poor methodological quality. This same paper182 assessed known-groups validity of the Teacher Rating Scale and found significant difference between groups on the Attention problems subscale. Mahan and Matson183 assessed the known-groups validity of the Parent Rating Scale only, and found support for the hypothesis that the ASD group would score higher on the Attention problems subscale.

Child Behavior Checklist 1.5–5

The CBCL 1.5–5184 is a norm-referenced measure that assesses for a wide range of emotional and behavioural disorders in children aged 1.5–5 years. It has 99 items, reported by parents on a three-point scale. Pandolfi et al. 185 found internal consistency, as measured by Cronbach’s alpha, for items relevant to Attention Problems to equal 0.68. The same study185 also found 27–52% of a typical item’s variance was attributable to the single underlying factor.

Child Behavior Checklist 6–18

The CBCL 6–18184 is a norm-referenced measure that assesses for a wide range of emotional and behavioural disorders in children aged 6–18 years. One study186 considering measurement properties of this version of the CBCL 6–18 was reviewed. Internal consistency was good, with a median scale reliability of r = 0.83 for Attention Problems. Structural validity missed the cut-off for the Attention Problems scale [RMSEA > 0.06; comparative fit index (CFI) = 0.955].

Baby and Infant Screen for Children with aUtIsm Traits-Part 2

The BISCUIT-Part 2 is a 65-item parent questionnaire that was developed to assess infants and toddlers for comorbid mental health conditions reported as common in children with ASD. Internal consistency was reported as good,83,187,188 as was inter-rater reliability. 188 The sample size was rather small for an adequate assessment of structural validity. 187 The BISCUIT-Part 2 was able to distinguish between children with and without ASD in two papers. 187,188 Finally, Matson, Fodstad et al. 188 provided supportive evidence of the criterion validity of the BISCUIT-Part 2, showing that scores were correlated as expected with diagnostic categorisation made by psychologists.

Behavior Assessment System for Children-Second Edition

The BASC-2 is tool widely used for assessing behaviour and emotions in children and young people (see Attention, above). Hass et al. 182 used the Teacher Rating Scale and reported that the BASC-2 had acceptable internal consistency for children (n = 30) for the Anger control (α = 0.75), Emotional self-control (α = 0.86) and Anxiety (α = 0.88) scales. There were significant differences between ASD and matched control groups for the Anxiety (Cohen’s d = 1.23), Anger control (Cohen’s d = 1.89) and Emotional self-control scales (Cohen’s d = 1.94). A second paper183 also assessed known-groups validity of the BASC-2 using the Parent Rating Scale. ASD children scored significantly greater than typically developing children on the Depression subscale, but did not differ as expected on the anxiety and internalising composite scales.

Brief Infant–Toddler Social and Emotional Assessment

The Brief Infant–Toddler Social and Emotional Assessment (BITSEA) is a 42-item screener for parents and child-care providers, designed to identify children with social emotional/behaviour problems. The BITSEA has two scales: BITSEA/P measures emotional and behavioural problems, and BITSEA/C measures competence. Measurement properties were assessed in two papers. 189,190 Assessment of internal consistency was judged as methodologically poor in both papers. 189,190 Test–retest reliability was acceptable with 10- to 45-day test–retest ICCs of 0.87 for BITSEA/P and 0.85 for BITSEA/C. 189 Ratings by both parents were significantly correlated in both Briggs-Gowan et al. 189 (ICC = 0.68 for BITSEA/P and 0.61 for BITSEA/C) and Karabekiroglu et al. 190 (Spearman’s correlation = 0.66 for BITSEA/P and 0.63 for BITSEA/C). However, agreement did not reach the COSMIN cut-off for acceptable inter-rater agreement. Agreement between parent and child-care provider was lower than between parents. 189 Hypothesis testing showed that the BITSEA had good convergent and divergent validity, and distinguished between toddlers with and without diagnosable social and emotional problems.

Child Behavior Checklist 1.5–5

The CBCL 1.5–5 behaviour scale was assessed by one paper185 of good methodological quality. This paper185 provided evidence of good internal consistency for the Internalising Behaviour domain (Cronbach’s α > 0.70) but was just below the COSMIN cut-off for the Emotionally reactive subscale (0.67) and the Anxious/depressed subscale (0.63). Structural validity was good overall, supporting the original factor structure of the Internalising and Externalising domains. However, model fit for a one-factor model for Emotionally reactive and Anxious/depressed subscales did not reach the COSMIN cut-off of RMSEA of < 0.06 (RMSEA > 0.09 and 0.07, respectively) indicating that there may not be a single latent factor underlying these subscales.

Child Behavior Checklist 6–18

The CBCL 6–18191 was assessed with a sample of 122 ASD youth (6–18 years) in one paper186 of good methodological quality. Internal consistency was good, with a median scale reliability of r = 0.94 for anxious/depressed and 0.85 for withdrawn/depressed. Structural validity was also good for Anxious/depressed (RMSEA < 0.06; CFI = 0.995) but missed the cut-off for the Withdrawn/depressed scale (RMSEA > 0.06; CFI = 0.975). Overall, the analysis supported the original factor structure of the CBCL 6–18. Criterion validity was assessed by comparing ASD children with and without a co-occurring emotional and behavioural difficulty (EBD). Children with a co-occurring EBD scored significantly higher than those without EBDs on Anxious/depressed and Withdrawn/depressed subscales and on the Internalising domain.

Children’s Global Assessment Scale

The Children’s Global Assessment Scale (CGAS)192 is a measure of overall psychosocial functioning (including home, school, with peers and across other settings). One study193 was identified that measured responsiveness in a large sample of children attending child psychiatric outpatient services, including 1053 participants with ASD. Mean CGAS ratings improved between first visit to outpatient services and at case closure after treatment (the raters were not the clinicians involved in treatment but did have access to baseline rating when making the end-point rating). Change in CGAS ratings was significantly moderately correlated with clinician assessment of treatment response (r = 0.47).

Infant–Toddler Social–Emotional Assessment

The 169-item Infant–Toddler Social–Emotional Assessment (ITSEA)194 is a parent-completed questionnaire that assesses three broad problem domains – Externalising, Internalizing and Dysregulation – along with Competence. Visser et al. 195 failed to report statistics for scale reliability (but did report mean ICCs of 0.7) for mothers and fathers, indicating acceptable inter-rater reliability. Visser et al. 195 and Georgiades et al. 82 both demonstrated that the ITSEA could distinguish between diagnostic groups. ITSEA Internalising and Externalising domains also correlated positively with the corresponding Internalising and Externalising domains of the CBCL and the Distraction and Mood scales of the Parenting Stress Index,195 supporting convergent validity.

Mullen Scales of Early Learning

The MSEL163 offer a developmental test for young children aged 0–69 months (see Cognitive ability, above), which includes direct assessment of fine motor skills, and in children of < 30 months gross motor skills are also examined. Burns et al. 164 found, as hypothesised, that children with developmental delays present significantly more difficulties regarding fine motor skills than typically developing children matched for age, race and gender.

Vineland Adaptive Behavior Scales

The VABS167 provide a structured interview measuring adaptive behaviour in four main domains: Communication, Daily Living Skills, Socialisation and Motor Skills. Motor Skills is measured only for children of < 6 years of age. One study168 showed that children with autism made significant developmental progress in the Motor Skills domain of the VABS, from pretest to year 1 and year 2.

Vineland Adaptive Behavior Scales-Screener version

The VABS-Screener171 is an adaptation of the VABS. Charman et al. 112 examined developmental change in children’s profiles over 11 months. The sample of 40 children gained 5.5 age-equivalent months in Motor Skills. Also paired t-tests indicated that age equivalents at time 2 were significantly higher than at time 1 for Motor Skills.

Autism Diagnostic Interview-Revised

Fourteen papers63,65–72,139,140,146,196,197 were reviewed considering measurement properties of the ADI-R in relation to Communication. Four studies63,68,139,146 assessed internal consistency, which found Cronbach’s alpha coefficient from 0.45 to 0.83 for the Communication domain. Six studies63,65,67,69,71,196 assessed inter-rater reliability of the Communication domain; 78% agreement was reported between raters by Robertson et al. 196 and kappa statistics ranging from 0.69 to 1.0. 63,65,67,69,71 The small sample size of three studies63,71,112,195 led these papers to be judged as methodologically poor. One paper140 found test–retest reliability for the Communication domain to be 0.73. However, the small sample size (n = 20) led this paper140 to be judged as being of poor methodological quality. Two studies63,66 assessed content validity and found that most items were considered relevant for the construct to be measured. Seven papers assessed structural validity providing contradictory conclusions (see Autism symptom severity, above). One paper197 found good structural validity for 28 social communication items but the paper was judged of poor quality. Three studies66,68,146 assessed convergent validity and found correlations between the ADI-R Communication domains and the VABS, the Aberrant Behavior Checklist (ABC), the Children’s Yale–Brown Obsessive Compulsive Scale, the Child’s Symptom Inventory, the Ritvo–Freeman Real Life Rating Scale, the Expressive Vocabulary Test and the Peabody Picture Vocabulary Test to range from –0.47 to 0.30. Three studies63,69,196 assessed known-groups validity and found > 75% of the study hypotheses were supported. One study70 assessed criterion validity and found the correlation between the ADI-R Communication domain and clinician impression to equal 0.49. Two studies65,71 assessed responsiveness and found that 75% of the hypotheses regarding stability in scores were supported.

Autism Diagnostic Observation Schedule-Generic

The original ADOS study50 assessed internal consistency and found Cronbach’s alpha coefficients to range from 0.74 to 0.84 for the Communication domain. Three studies50,65,72, assessed inter-rater reliability. Two of these studies65,72 found kappa statistics to range from 0.60 and 0.80 on all items. However, the lack of methodological information regarding these analyses led the study by Kamp-Becker et al. 72 to be judged as having poor methodological quality. Lord et al. 50 reported good inter-rater reliability (ICC = 0.84) and test–retest reliability (ICC = 0.73) for the Communication domain across modules. Conclusions regarding structural validity, criterion validity and for responsiveness were as described for symptom severity (see Autism symptom severity, above).

Autism Diagnostic Observation Schedule-Toddler Module

Luyster et al. 76 presented the new Toddler Module of the ADOS (see Autism symptom severity, above). Internal consistency for all groupings was high (younger and non-verbal children: α = 0.88; verbal α = 0.90) for the Social Affect score. For inter-rater reliability, ICCs for the Social Affect total were 0.84 and 0.99, respectively; however, the small sample size (n = 13) led this to be judged as being of poor methodological quality. Test–retest reliability ICCs were 0.83 and 0.94; however, there were only eight children assessed twice in the older verbal group.

Early Social Communication Scales-Live

The ESCS-Live (ESCS-L) is an abbreviated version of the original ESCS observational coding scheme,127 and is used as a measure of non-verbal social communication skills for children up to age 30 months. One study129 assessed inter-rater reliability, and found that the average inter-rater reliability for this tool was 88.33%. Yet, owing to the small sample size for assessment of this property (n = 16), the paper was judged to be of poor methodological quality.

Social Communication Assessment for Toddlers with Autism

The SCATA137 was designed to measure non-verbal communication, including early and atypical communication, in young children with ASD. Drew et al. 137 assessed inter-rater reliability and found the ICCs to range from 0.03 to 1.00, with 16 results of < 0.70 and 24 results of > 0.70. This same study135 assessed known-groups validity and found most of the results to be in accordance with the hypotheses. However, the small sample sizes (n = 17 and n = 23) led both assessments from this study135 to be judged as of poor methodological quality.

Autism Diagnostic Interview-Revised

Twelve papers63,64–71,139,140,146 considering measurement properties of the Reciprocal Social Interaction domain of the ADI-R were reviewed. Four studies 63,68,112,139,146 assessed internal consistency; Cronbach’s alpha coefficients were consistently good for Reciprocal Social Interaction, and higher than for other domains. Five studies63,65,67,69,71 assessed inter-rater reliability and reported kappa statistics to range from 0.64 to 1.0. The small sample size of two studies63,71 led these papers to be judged as being methodologically poor. One paper140 found test–retest reliability for Reciprocal Social Interaction to be 0.84. However, the small sample size (n = 20) led this paper to be judged as being of poor methodological quality. Two studies63,66 assessed content validity and found that most items were considered to be relevant for the construct to be measured. Seven papers63,64–66,69,70,71 assessed structural validity providing contradictory conclusions (see Autism symptom severity, above). Two studies64,66 assessed convergent/divergent validity. The former found ADI-R Reciprocal Social Interaction correlated with VABS scores as hypothesised, –0.41 to –0.45 (below the COSMIN criterion) and not with motor skills. Two studies63,69 assessed known-groups validity and all of the study hypotheses were supported regarding Reciprocal SociaI Interaction. One study70 assessed criterion validity and found the correlation between ADI-R Reciprocal Social Interaction and clinician impression to equal 0.46. Two studies65,71 assessed responsiveness and found that all of the hypotheses regarding stability were supported.

Nisonger Child Behavior Rating Form

The Nisonger Child Behavior Rating Form (NCBRF)198 is a rating scale designed to assess social competence and behaviour problems in children and adolescents with developmental disabilities. It has 76 items, completed by parents or teachers. Lecavalier et al. 199 assessed internal consistency and found alpha coefficients for the social competence items to range from 0.63 to 0.85 in a sample of 330 children and adolescents with ASD. Lecavalier et al. 146 assessed inter-rater reliability and report ICCs between the parent and teacher ratings to range from 0.17 to 0.23 on the social competence items. Their sample was 293 children with ASD, of whom one-third attended preschool or kindergarten. Test–retest reliability was also assessed,146 and ICCs ranged from 0.63 to 0.73 for the social competence items. One study199 assessing structural validity found RMSEA ranging from 0.000 to 0.031 for the social competence items. Lecavalier et al. 146 assessed convergent validity and found Spearman ranked correlation coefficients between Parenting Stress Index-Short Form (PSI-SF) and the social competence items of the NCBRF to range from 0.41 to 0.45.

Vineland Adaptive Behavior Scales

The VABS167 assesses four domains: Communication, Daily Living Skills, Socialisation and Motor Skills. Tyminski and Moore200 assessed test–retest reliability and found that follow-up VABS Socialisation scores were positively related to baseline VABS scores (r = 0.74). Known-groups validity was assessed by Paul et al.,169 who reported a significant difference between groups of children with autism and those with ASD in the Socialisation domain, as expected. Harris et al. 201 assessed responsiveness and found that children made significant improvements in the Socialisation domain. However, the small sample size (n = 20) led this study to be judged as being methodologically poor. One paper170 considered measurement properties of the classroom edition of the VABS. This study170 assessed convergent validity and found a correlation between Autism Severity, as measured by CARS and VABS-Classroom socialisation scale to equal –0.58. Measurement properties of the VABS-Screener171 were assessed by Charman et al. ,112 who reported that the change score for the Socialisation domain was significant, with children making 9 months’ progress in 11 months.

Test of Pretend Play

The Test of Pretend Play (ToPP)202 is an assessment of symbolic play ability in young children (aged between 18 months and 6 years). It assesses how children substitute one object for another, and refer to an absent object, as if it was present, and attribute an imaginary property to an object. Only one study203 in our review investigated measurement properties of the ToPP. Clift et al. 203 reported a moderate positive correlation between the ToPP scores and language scores (r = 0.41), measured by FirstSTEP, a screening assessment used to identify children who may be at risk of developmental delay. The study203 showed also that the ToPP correctly classified 75.9% of the participants (children with developmental problems and/or a psychological disorder, and typically developing children).

Aberrant Behavior Checklist

The ABC204 is a 58-item caregiver report checklist designed to assess maladaptive behaviours in people with developmental disabilities. The ABC was assessed in three studies. 205–7 Internal consistency was reported as good (Cronbach’s alpha coefficients ranged from 0.68 to 0.90) by Karabekiroglu and Aman,205 whereas inter-rater reliability was reported as poor by Sigafoos et al. 206 (mean Spearman’s rank correlation coefficient between parent and teacher ratings was 0.62, range 0.50–0.83). Brinkley et al. 207 demonstrated that the ABC had good structural validity, although the irritability subscale item placement did not match the standard ABC factor structure. One note of caution here is that in the ASD sample items on self-injury clustered into one factor and the remaining items from the standard ABC irritability subscale shifted to the hyperactivity subscale. Sigafoos et al. 206 also showed that the ABC had good structural validity, with five factors that closely matched the standard ABC factor structure; however, because of the small sample size (n = 32) this paper was judged of poor methodological quality. Known-groups validity and criterion validity were shown to be acceptable by Karabekiroglu and Aman. 205 The ABC distinguished between clinical subgroups and showed significant positive correlations with related constructs measured by the CBCL and the AuBC.

Baby and Infant Screen for Children with aUtIsm Traits-Part 3

The BISCUIT-Part 383 was designed to assess challenging behaviours. Internal consistency of the BISCUIT-Part 3 was reported as good with Cronbach’s alpha coefficient of > 0.70 in two papers. 83,208 Structural validity, assessed in Matson et al. 208 was not acceptable, with EFA resulting in a three-factor solution explaining just 38.32% of the variance.

Behavior Assessment System for Children-Second Edition, Parent Rating Scales

The BASC-2, Parent Rating Scales is an omnibus instrument widely used for assessing behaviour and emotions in children and young people (see Attention, above). Hass et al. 182 showed that the BASC-2 had acceptable internal consistency for the 10-item Aggression scale and the nine-item Conduct problem scale. There were also significant large differences between children with ASD and matched control subjects on the Aggression scale (Cohen’s d = 0.58) and the Externalising problems composite scale (Cohen’s d = 0.75). Mahan and Matson183 also assessed known-groups validity of the BASC-2. ASD children scored significantly greater than typically developing children on the Conduct problems and Externalising composite scales but did not differ as expected on the Aggression subscale.

Child Behavior Checklist 1.5–5

The CBCL 1.5–5 Behaviour scale was assessed by one paper185 of good methodological quality. This paper provided evidence of good internal consistency for total problems (Cronbach’s α = 0.93) and both the Externalising Behaviour domain (Cronbach’s α = 0.90) and Aggressive behaviour subscale (Cronbach’s α = 0.80). Structural validity was also good, with acceptable model fit for a one-factor model for aggressive behaviour (RMSEA < 0.06; CFI > 0.95), indicating that there was a single latent factor underlying this subscale.

Child Behavior Checklist 6–18

The CBCL 6–18191 was assessed with a sample of ASD youth in one paper186 of good methodological quality. Internal consistency was good, with r = 0.92 for the Aggressive behaviour scale. Structural validity for the complete measure was good, and analysis supported the original two-factor structure of the CBCL 6–18 (internalising and externalising factors). Tests of unidimensionality of scales did not reach the cut-off for acceptable fit for aggressive behaviour (RMSEA = 0.10; CFI = 0.95); however, convincing arguments were provided to allow for correlated disturbances in the model for two-item pairs (destroys own things/destroys others things and disobedient at home/disobedient at school). This adjusted model demonstrated acceptable fit (RMSEA < 0.06; CFI > 0.95). Finally, criterion validity was assessed by comparing ASD children with and without a co-occurring EBD. Children with a co-occurring EBD scored significantly higher than those without EBDs on total problems. There were no significant differences between the two groups for aggressive behaviour or externalising behaviour. Given that the most commonly co-occurring EBDs were anxiety disorders, it is reasonable to assume that in this sample we would not expect to see group differences in aggressive or externalising behaviour.

Home Situations Questionnaire-Pervasive Developmental Disorders version

The Home Situations Questionnaire-Pervasive Developmental Disorders version (HSQ-PDD) is caregiver questionnaire designed to assess behavioural non-compliance in everyday situations by children with ASD. It was developed in studies of typically developing children, and was modified by Chowdhury et al. 209 and its properties assessed in a sample of 124 children with ASD, aged 4–13 years. Structural validity for a two-factor solution was a reasonable fit (RMSEA 0.06) and internal consistency good for the 25-item version thus derived (α = 0.90 for the Socially inflexible subscale and α = 0.80 for Demand-specific subscale). Known-groups validity and responsiveness (change over time) were also good for the HSQ-PDD. Responsiveness was shown related as hypothesised to change in the VABS Daily living skills scale. 210

Nisonger Child Behavior Rating Form

The NCBRF198 is a rating scale designed to assess social competence and problem behaviour in children with developmental disabilities. There are parent and teacher versions of the scale. Internal consistency of the problem behaviour scales was reported as good, with Cronbach’s alpha coefficient of > 0.70 for all subscales in both parent and teacher versions. 199 Test–retest reliability for the parent version was reported to be strong (ICC for total problem behaviour > 0.80) but the teacher version fell short of the COSMIN criterion (ICC for total problem behaviour = 0.68); however, over a 1-year time interval some change might well be expected. Agreement was low between parents and teachers on common items from the parent and teacher version of the scale, indicating that inter-rater reliability was poor. 146 Structural validity was also shown to be poor for problem behaviour with a five-factor solution accounting for 47.5% of the variance. 199 Finally, Lecavalier et al. 146 provided fair evidence for divergent and convergent validity of the NCBRF.

Child Behavior Checklist 1.5–5

The CBCL184 1.5–5 was originally shown to measure two higher-order factors (internalising and externalising behaviour) and seven second-order factors (emotionally reactive, anxious/depressed, somatic complaints, withdrawn, attention problems, aggressive behaviour and sleep problems). One study185 of good quality assessed the CBCL 1.5–5 in an ASD sample. Internal consistency was good for total scale and sleep problems (α > 0.80) but not acceptable for somatic complaints (α = 0.49). Structural validity was not acceptable for sleep problems (RMSEA = 0.13) but was acceptable for somatic complaints (RMSEA = 0.06), just reaching the cut-off for acceptable fit. Overall, the findings supported the original structure of the CBCL 1.5–5 but there were conflicting findings for the scales related to the Habit Problems domain.

Child Behavior Checklist 6–18

The CBCL 6–18191 was assessed with a sample of ASD youth in one paper186 of good methodological quality. Internal consistency was good, with a median scale reliability of r = 0.85 and r = 0.88 for somatic complaints. Structural validity was also strong for somatic complaints (RMSEA < 0.001; CFI = 1) and analysis supported the original factor structure of the CBCL 6–18. Criterion validity was assessed by comparing ASD children with and without a co-occurring EBD. Children with a co-occurring EBD scored significantly higher than those without EBDs on total problems and on the Somatic complaints subscale.

Sense and Self-Regulation Checklist

Silva and Schalock,211 provided an assessment of the properties of the SSC (see Sensory processing, above). Internal consistency was rated as good with Cronbach’s alpha coefficient of > 0.80 (range from 0.85 to 0.89). Although test–retest reliability overall did not reach the cut-off, the reliability coefficient was 0.83 for the Self-Regulation domain. This relates most closely to habit problems and so was considered supportive evidence of test–retest reliability. This paper211 also demonstrated that the SSC was able to distinguish between children with and without ASD.

Vineland Adaptive Behavior Scales

The VABS (see Language and social functioning, above) had two papers168,169 that assessed domain-level validity169 and responsiveness,168 respectively. Paul et al. 169 demonstrated that children with autism and ASD differed on Communication and Socialisation domains but not on the Daily Living Skills domain. Groups did differ on ‘phone use’, which is part of the DLS scale but was deemed to be more relevant to verbal communication than daily living skills. Hypotheses were not specifically set out in the paper and so it was rated as ‘fair’ quality. Harris et al. 168 assessed rate of change and change in developmental age. This paper168 showed that VABS was able to identify change over time but was of poor quality due to the small sample size (n < 20).

Wells et al. 170 evaluated the VABS-Classroom. The paper170 was rated as fair quality. The VABS-Classroom Daily living skills scale demonstrated convergent validity with mental age (r = 0.87) and severity of autism (r = –0.46).

Charman et al. 112 assessed responsiveness of the VABS–Screener tool in young children with ASD. This paper112 was of good methodological quality and showed that there was developmental progress from time 1 to time 2 in daily living skills as predicted.

Assessment, Evaluation, and Programming System

The Assessment, Evaluation, and Programming System212 for infants and children is a curriculum-referenced comprehensive system assessing six key developmental areas in young children: Fine Motor, Gross Motor, Cognitive, Adaptive, Social Communication and Social. Each domain has 15–54 items: 228 in total. Wang et al. 213 assessed the internal consistency of the Social domain only and found the Cronbach’s alpha coefficient to be 0.98. The same study213 assessed the tool’s responsiveness, and found significant change in the Social domain from pre-test and post test, as hypothesised. However, owing to the small sample size (n = 22), both assessments from this paper were judged to be of poor methodological quality.

Behavior Assessment System for Children-Second Edition

The BASC-2181 is a tool for assessing behaviour and emotions in children, adolescents and young adults (see Attention, above). Hass et al. 182 report internal consistency of the Teacher Rating Scale, as measured by Cronbach’s alpha, to range from 0.76 to 0.90 for the Adaptive Functioning subscales. The absence of a factor analysis, however, led this paper to be judged as being of poor methodological quality. Mahan and Matson183 assessed the known-groups validity of the Parent Rating Scale and found that all relevant hypotheses were supported.

Psychoeducational Profile-Revised

The Psychoeducational Profile-Revised (PEP-R)214 is used to assess abilities and formulate treatment programmes for children with autism and related developmental disorders. The tool consists of a Developmental scale, with a total of 153 items, and a Behavioural scale, with a total of 43 items. Four studies215–218 assessed the internal consistency of the tool’s Developmental scale and found that Cronbach’s alpha coefficients ranged from 0.81 to 0.99. Three of these studies216–218 also assessed internal consistency of the Behavioural scale and found that Cronbach’s alpha coefficients ranged from 0.74 to 0.99. However, none of these studies included a factor analysis, and therefore all assessments were judged to be of poor methodological quality. Three studies215,216,218 assessed the inter-rater reliability of the Developmental scale and found ICC values ranged from 0.84 to 0.99. Shek et al. 216 and Villa et al. 218 assessed the inter-rater reliability of the Behavioural scale and found that ICC values ranged from 0.56 to 0.88. Owing to small sample size, Alwinesh et al. 215 was judged to be of poor methodological quality. Two studies215,216 assessed the test–retest reliability of the Developmental scale and found ICC values ranging from 0.87 to 0.99. Shek et al. 216 assessed the test–retest reliability of the Behavioural scale and found ICC values ranging from 0.76 to 0.92. Steerneman et al. 217 and Heimann et al. 219 both assessed known-groups validity and, in both, the relevant hypotheses were supported. However, the small sample size led the second study219 to be judged as being of poor methodological quality. Two studies215,217 assessed the tool’s convergent validity and found correlations between the PEP-R and Snijders-Oomen Non-Verbal Intelligence Test-Revised to range from 0.90 to 0.95,217 and correlations between PEP-R and Gesell’s Developmental Schedule to range from 0.34 to 0.84. 215 Criterion validity was assessed by two studies,216,218 and found correlations between PEP-R Developmental Score and VABS to equal 0.85,218 and correlations between the PEP-R Developmental scale and Merrill–Palmer Scale of Mental Tests to equal 0.71. 216 Heimann et al. 219 assessed responsiveness and found all relevant hypotheses regarding difference in change scores to be supported. However, the small sample size (n = 20) meant that this study219 was judged to be of poor methodological quality.

Psychoeducational Profile-Third Edition

The third edition of the Psychoeducational Profile (PEP-3)220 is specifically designed for children with ASD – ranging in developmental age from 2 years to 7 years 6 months – to assess their development of communication and motor skills and the presence of maladaptive behaviours. It consists of 10 performance subtests that are combined into three composites: Communication, Motor, and Maladaptive behaviours. Fu et al. 221 tested a Chinese translation; they assessed internal consistency and found Cronbach’s alpha coefficients to range from 0.92 to 0.98. The same study221 assessed inter-rater reliability, and found ICC of the PEP-3 ranged from 0.57 to 0.94 for the performance subtests, and 0.63 to 0.89 for the composites (4 out of 13 ICCs were < 0.70). This study also assessed measurement error, and found that the standard error measurement of the PEP-3 ranged from 2.6 to 6.5 for composite scores, whereas the smallest real difference of the PEP-3 ranged from 5.8 to 12.8. Chen et al. 222 assessed responsiveness and found that composite scores and most of the subtest scores of the PEP-3 changed in raw scores and developmental ages, but were stable in percentile ranks. Owing to the absence of specific hypotheses, this study222 was judged to be of poor methodological quality.

Measurement properties of the Caregiver report of the PEP-3 were reviewed by Fu et al.,223 who assessed internal consistency and found Cronbach’s alpha coefficients to range from 0.15 to 0.85. However, as no factor analysis was conducted, this study223 was judged to be of poor methodological quality. The same study223 assessed inter-rater reliability and found the ICCs of the subtests ranged from 0.66 to 0.79, and criterion validity reporting correlation coefficients between the PEP-3-Caregiver and the VABS ranged from 0.04 to 0.82. However, the small sample size (n = 20) meant that this study was judged to be methodologically poor.

Scales of Independent Behavior-Revised

The Scales of Independent Behavior-Revised (SIB-R)224 is a comprehensive norm-referenced test used to assess adaptive behaviour. It contains 14 subscales distributed into four areas: (1) Motor Skills; (2) Social and Communication Skills; (3) Personal Living Skills; and (4) Community Living Skills. Lecavalier et al. 146 assessed internal consistency and found Cronbach’s alpha coefficients to range from 0.87 to 0.96. The same study146 also assessed known-groups validity, and found support for the hypothesis. Brown et al. 225 assessed convergent validity and found that Full Scale IQ did not positively correlate with Broad Independence, and that Verbal IQ did not positively correlate with Social Interaction, as expected. However, the small sample size (n = 25) led this study to be judged as being of poor methodological quality.

Vineland Adaptive Behavior Scales

Paul et al. 169 explored the domains and subdomains of the VABS and assessed known-groups validity, finding support for all of their relevant hypotheses. Harris et al. 168 assessed responsiveness. A series of significant differences at years 1 and 2 were presented, but, owing to the lack of specific hypotheses, it is difficult to determine if the results were as expected. Therefore, the methodological quality of the paper was judged as poor.

Wells et al. 170 examined the measurement properties of the classroom edition of the VABS and reported convergent validity between the VABS-Classroom adaptive behaviour composite and CARS (r = –0.53).

The responsiveness of the Screener version of the VABS171 was assessed. 112 The VABS-Screener age-equivalent domain scores showed clear evidence of developmental progress from times 1 to 2, although change in the composite score was not significant.

Autism Parenting Stress Index

The Autism Parenting Stress Index (APSI)226,227 is a caregiver questionnaire designed for clinical use to identify what aspects of parenting skills would benefit from additional support, and to measure the outcome of intervention on parenting stress relative to core and comorbid symptoms of a child’s autism. The measure consists of three categories: the core social disability, difficult-to-manage behaviour and physical issues. All of the items are rated on a five-point Likert scale, ranging from ‘not stressful’ to ‘so stressful that sometimes we feel we cannot cope’. Only one study153 has examined the measurement properties of the APSI, collecting data from parents of 274 children aged < 6 years (including 109 children with ASD). Authors reported good internal consistency and 4-month test–retest reliability (however, a subsample of only 18 parents was used). The factor analysis of the APSI revealed a four-factor solution (overall parental stress scale; core autism symptoms; comorbid behaviours; comorbid physical issues); however, the explained variance is not stated. The APSI discriminated between children with ASD and those who are typically developing or have other developmental delays.

Parenting Stress Index-Short Form

The PSI-SF228 is a 36-item self-report questionnaire measuring parenting stress. It contains three subscales: Parental distress, Parent–child dysfunctional interaction, and Difficult child (the extent to which the parent considers the child to be ‘difficult’). Items are rated on a five-point Likert scale ranging from ‘strongly agree’ to ‘strongly disagree’. Three studies reported data on measurement properties of the PSI-SF. Lecavalier et al. 146 reported excellent internal consistency for the total score (Cronbach’s α = 0.93). Good internal consistency for the subscales, and support for a primary dimension for each of the PSI-SF subscales, were reported by Zaidman-Zait et al. 229 However, in Zaidman-Zait et al. 230 the three-factor model was rejected and a six-factor solution was suggested. Parenting stress was showed to be stable across time (over 1-year period) and associated with behaviour problems,146 greater severity of autism and other psychological problems. 230

Questionnaire on Resources and Stress-Friedrich Short Form

The Questionnaire on Resources and Stress-Friedrich Short Form (QRS-F)231 is a 52-item questionnaire assessing the level of stress in families of children with disabilities. It contains four subcomponents of parental perceptions of difficulties: parent and family problems (stressful aspects of the impact of the child with disability on parents and the wider family), pessimism (parents’ pessimistic beliefs about the child’s future), child characteristics (features of the child that are associated with increased demands on parents) and physical incapacity (the extent to which the child is able to perform a range of typical activities). Honey et al. 232 investigated measurement properties of the QRS-F. A total of 174 mothers and 43 fathers of a child with autism aged between 26 and 82 months completed 31 items from the QRS-F (from the child characteristics, parent and family problems and pessimism scales). The authors reported good internal consistency for the total score. Support for the expected two- or three-factor solutions for the QRS-F was not found. The study232 showed convergent validity of the QRS-F, with parents reporting more stress when raising children with more severe symptoms of autism and less stress if children were more able (as indicated by higher VABS scores).

Discussion Day

As described in Chapter 2, 25 participants came to a Discussion Day in London on February 2014, including parents, a young adult on the autism spectrum, researchers, and health and education professionals. In addition to the Q-sort activity described in Chapter 2, participants were divided into mixed groupings and asked to evaluate tools set out on display (four sets for each grouping). The subdomains represented were symptom severity, global measure of outcome, sensory processing, cognitive ability, behaviour problems and parent stress. Two direct child assessments were shown, with videotape accompaniment. The questionnaire tools were presented in pairs to allow participants to compare and contrast. Summary information about each tool was available. Participants rated each tool for (1) the assessment experience (including questionnaire wording) and (2) how likely it would be to capture change. They were asked also to state what they liked or did not like about the tool.

Subdomains for which tools are lacking

In the Social awareness subdomain, evidence on measurement properties with children on the autism spectrum was limited. This is an example of when tools are likely to have been developed first with typically developing children. Examples include the ESCS270 and the CSBS-DP. 125 In the recent review of social communication scales for use in medication trials in ASD,238 both of these scales were rated ‘appropriate with conditions’ on the basis of evidence of reliability and of apparent sensitivity to change in ASD intervention studies. Nevertheless, both tools have ceiling effects, as they are appropriate up to ages equivalent to 30 months and 24 months, respectively, and both tools require detailed training and time for rating. Thus, the strategies adopted in the MeASURe review have not necessarily been overly exclusive.

A second subdomain to be discussed further is that of Habit Problems. Only two tools had been used in longitudinal or intervention studies in this review: subscales of the CBCL and the SSC. In addition, one study271 had used sleep diaries. In the process of searching for papers on measurement properties of tools identified in Chapter 3, some new tools were found for which the paper had included tools searched for by name in establishing convergent validity (see Appendix 9). For example, the Children’s Sleep Habits Questionnaire272 is designed for children aged 4–12 years, and has been reviewed by Majnemer29 for use with children with developmental disabilities. The Family Inventory of Sleep Habits for children with ASD273 focuses on the particular problems of sleep anxiety and bedtime resistance. The Brief Autism Mealtime Behavior Inventory274 is a tool recently developed specifically to assess eating problems in children with ASD. Thus future reviews should provide additional evidence regarding measurement tools in ASD for this important subdomain.

Domains of Participation and Family Measures are particularly under-represented in this review of tools. Although several Family tools had been used in studies in Chapter 3, no examination of their measurement properties in studies with children with ASD has been identified. This is a clear research gap. Similarly it would be desirable to have further exploration in ASD of tools related to social inclusion, such as the School Liking and Avoidance Questionnaire,275 which can be teacher rated or self-reported by children as young as 3 years. 276

Other approaches to measurement

The search for papers on measurement properties of tools has the effect of missing some approaches that have a legitimate place in monitoring the progress of young children with ASD. As mentioned in Chapter 4, the tool used most often in nurseries in UK is the Early Years Foundation Stage Profile. A number of other curriculum-based tools had been used in observational studies in the review. Such tools have the benefit of being closely related to programme planning for individual children. However, criterion-referenced assessment approaches are not usually examined in research for their measurement properties such as reliability and validity.

Other approaches that individualise assessment for children include ‘Target Behaviours’ and Goal Attainment Scaling. With the individuality of needs of young children with ASD, it may be particularly appropriate to adopt an idiographic approach to outcome measurement. Yet for the purposes of research, the ability to compare across individuals is required. A Target Behaviours (or target symptoms) methodology was included in the battery of tools recommended by the Research Units on Pediatric Psychopharmacology277 and used by one study278 in this review. When a specific behaviour is the target of intervention, the parent is interviewed about its nature, frequency and intensity, and a vignette description is prepared. At follow-up, the same questions are asked about the behaviour; the two vignettes are then compared and rated for degree of change on a nine-point scale by an expert panel. Thus this idiographic measure allows for ‘blind’ rating, and provides an opportunity to capture change. Inter-rater reliability across the expert panel can be assessed. Goal Attainment Scaling requires greater professional input (than Target Behaviours), including training and practice, to enable a suitable behavioural goal to be defined and scaled (with description of outcomes on a five-point scale between ‘worst expected outcome’ to ‘best expected outcome’). There are continuing debates about appropriate statistical analyses of Goal Attainment Scaling scores, such as whether accomplishment of different individual goals can be summed into a group score. Nevertheless, if the Goal Attainment Scaling scores are done by observation, the assessor can be ‘blind’. 279 These approaches to responsive measurement of relevant and individualised outcomes merit further exploration for young children with ASD.

In future, there will be a need to examine the measurement properties of biophysical tools, such as actigraphy and sleep recordings, i.e. of particular relevance to certain types of intervention, which have the apparent benefit of objectivity in measurement.

Finally, the review has not identified and evaluated tools used in economic analyses. Very little research has investigated the measurement of preference-based health-related quality-of-life outcomes in children with ASD. 280 Tools that have been examined for children with ASD include the Health Utilities Index Mark 3280 and for parents the European Quality of Life-5 Dimensions three-level version. 281 These studies indicate promising findings on the measurement properties of the tools, but further research is required to establish whether they are appropriate for use in studies of children with ASD and how they may contribute to the evaluation of the cost-effectiveness of therapies and services.