Informing the NHS Outcomes Framework: what outcomes of NHS care should be measured for children with neurodisability?

Search strategy

The search strategy was designed by an information specialist (MR) following consultation with the systematic review team, and with reference to the methodological filters published by the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) group27 and the construct filters developed by the Oxford PROMs group. 28 The strategy used a combination of medical subject headings (MeSH) and free-text terms. Search terms were grouped as follows:

• group 1: general names for measures (e.g. questionnaires, instruments or tools)

• group 2: multidimensional health construct terms (e.g. quality of life or health status)

• group 3: terms to describe children and young people (e.g. children, teenagers or adolescents).

The terms within each group were combined with a Boolean OR command and were searched in combination using a Boolean AND command. Piloting this search strategy produced a total of 38,893 citations. Systematic screening of this number of citations was judged to be too burdensome within the confines of the resources allocated to the project. A fourth set of terms was therefore added to increase the specificity of the search:

• group 4: terms relevant to psychometric performance (e.g. validity or reliability).

As this project was conceived to inform the NHS Outcomes Framework in the UK, we were interested only in PROM questionnaires that were available and evaluated in English; hence, the search was limited to English language. The search was also limited by date to publications from 1992, as the team agreed that it was unlikely that PROMs had been developed before this date.

The search strategy was designed for MEDLINE (via OvidSP) and modified for EMBASE and PsycINFO (via OvidSP) and Cumulative Index to Nursing and Allied Health Literature (CINAHL) (via EBSCOhost). The searches were run on 20 and 21 March 2012. Separate searches were carried out on the Oxford PROM bibliographic database and Patient-Reported Outcome and Quality of Life Instruments Database. Reference lists of systematic reviews were also checked. 14,29–32 The search strategy (for MEDLINE/OvidSP) is shown in Appendix 1 . All search results were exported to reference manager software (EndNote X6, Thomson Reuters, CA, USA) and deduplicated. EndNote was used to manage the citation database throughout the project.

Inclusion and exclusion criteria

The aim of this stage of the review was to identify potential candidate PROMs. Eligibility criteria were developed to guide selection ( Table 1 ).

Study selection

Two groups of reviewers (group 1: AJ, VS, SB; group 2: CM, JTC, MR, DM, RW, RA) independently screened all titles and abstracts to locate papers in which potential candidate PROMs were cited. Discrepancies were discussed and resolved with the arbitration of a third reviewer (either CM or AJ), where necessary. A flow chart describing the process of identifying relevant literature for this stage of the review can be found in Figure 1 .

FIGURE 1.

Flow chart showing identification and selection of potentially eligible candidate PROMs. a, Some instruments were excluded for more than one reason.

Data extraction

The result of this stage of the search was a list of potentially eligible candidate PROMs. Names and acronyms of all PROMs cited were extracted by one reviewer (AJ) who applied the eligibility criteria. Where there was insufficient detail provided in the paper to address all eligibility criteria, additional information was sought by internet searching.

Search strategy

The search for the second stage of the review was designed using the names, alternative names and standard acronyms of the candidate PROMs identified in stage 1. For each candidate PROM, an individual search strategy was created to identify studies where the PROM had been used and evaluated in general populations (search 2.1). Three groups of search terms were used:

• group 1: name(s) of the PROM

• group 2: terms to describe children and young people

• group 3: psychometric terms (e.g. validity or reliability).

The electronic search was designed for MEDLINE (via OvidSP) and modified for EMBASE and PsycINFO (via OvidSP). No language or date limits were applied to the search. The search used in MEDLINE (OvidSP) can be found in Appendix 2 . In total, 51 searches were run on each of the three databases between 18 July and 5 September 2012.

A further search strategy was designed to identify studies where candidate PROMs might have been used specifically with neurodisability (search 2.2). Three groups of search terms were used:

• group 1: name(s) of the PROM

• group 2: terms to describe children and young people

• group 3: neurodisability terms, including key exemplar conditions.

The terms used included MeSH terms, and variations of the three exemplar conditions set out in our original proposal, namely cerebral palsy, autism and epilepsy. The search was designed in MEDLINE (OvidSP) and modified for EMBASE, PsycINFO and the Allied and Complementary Medicine Database (via OvidSP), CINAHL (via EBSCOhost) and NHS Database of Economic Evaluations. Searches were run between 12 and 25 September 2012. The strategy used for MEDLINE/OvidSP can be found in Appendix 3 .

Backwards citation chasing (one generation) was carried out using all reference lists from papers included in this stage of the review. Forward citation chasing was carried out between 28 January and 6 February 2013 using Science Citation Index and Social Science Citation Index (via Web of Knowledge) for included studies. Search results were exported into separate EndNote libraries created for each PROM.

We sought to locate a copy of each questionnaire; if a copy was not readily available, authors and/or developers of the PROMs were contacted to request a copy. We also contacted the authors or developers of all PROMs for which no evidence of the psychometric performance in an English-speaking population was found.

Inclusion and exclusion criteria

The aim of this stage of the review was to identify evidence for the performance of candidate PROMs when evaluated with children and young people. Criteria to guide inclusion and exclusion are shown in Table 2 .

Study selection

Titles and abstracts of all unique citations were screened against the eligibility criteria by one reviewer (AJ) and a sample of 10% of decisions was checked by a second reviewer, (CM) with disagreements resolved by discussion with a third (CJ) where necessary. The full text of any potentially relevant article was retrieved and screened using the same procedure. A flow chart describing the process of study selection for this stage of the review can be found later in this chapter ( Figure 2 ).

Data extraction

Data were extracted using standardised, piloted data extraction forms. For each included candidate PROM, the following were extracted: name of PROM and acronym, purpose of measurement, number of items, the responder, completion time, age range, recall period, response options, key reference paper, and types of domains/dimensions assessed.

We determined that there were three types of eligible candidate generic PROMs: (i) generic PROMs, designed for use across all people; (ii) chronic-generic PROMs, intended for use across people with chronic conditions; and (iii) preference-based measures (PBMs). Scores from generic and chronic-generic PROMs are typically determined directly from responses to items in the questionnaires. PBMs have two components; the responses to patient questionnaires are transformed using a weighting system, based on valuation of health states by a reference population, to produce a single index score between 1 and 0 (or less), where 1 equates to full health and 0 is dead. 33,34

The domain scales and items of each candidate PROM were inspected with reference to the WHO’s International Classification of Functioning, Disability and Health, Children and Youth Version (ICF-CY)11,35 to provide an indication of the constructs each instrument was measuring. It was not our aim to allocate every item from candidate PROM questionnaires to a precise ICF code using proposed linking rules. 36 Instead, our mapping sought to use the ICF-CY to describe what the scales of each candidate PROM proposed to measure. We coded at the higher levels of the ICF-CY, and also identified separately constructs not represented in the ICF-CY.

For each paper describing a study evaluating the psychometric performance of an eligible candidate PROM, the following descriptive data were extracted: instrument version, first author name, publication year, study aim, study population, number of participants, age range, mean age [standard deviation (SD)], and setting/country where the study was conducted. Data were extracted by one reviewer (KA/AJ) and checked by a second reviewer (AJ/KA), with disagreements resolved by discussion with a third (CM) where necessary.

For each included paper, the methodological quality of the study and the completeness of the report were assessed using the COSMIN checklist. 37 The COSMIN checklist assesses the methods and reporting of internal consistency, reliability, measurement error, content validity, structural validity, hypothesis testing, cross-cultural validity, criterion validity and responsiveness. Cross-cultural validity is not reported as we only included studies using an English version of the eligible PROMs. The checklist was administered by one reviewer (CM) and a 10% sample was rated by a second (AJ). Studies that used Rasch analysis were also assessed by one of the team with expertise in these techniques (AT). The COSMIN checklist uses a ‘worst score counts’ rating for methods used to test psychometric properties, producing a quality assessment of excellent, good, fair or poor. 37 Any discrepancies were resolved by discussion, or with the involvement of a third reviewer (CJ), where necessary.

Then, any data on evidence of the psychometric properties or performance of instruments were extracted including content validity (theoretical framework and/or qualitative research), construct validity (structural validity and hypothesis testing), internal consistency, test–retest reliability, proxy reliability, precision, responsiveness, acceptability and feasibility. Data were extracted by one reviewer (KA/AJ) and checked by a second reviewer (AJ/KA), with disagreements resolved by discussion with a third (CM) where necessary.

Appraisal of evidence for psychometric performance

Our original proposal focused on evidence of psychometric properties when evaluated with children and young people affected by neurodisability. However, we were mindful that some emerging PROMs may have only been tested with general populations, and it would be advantageous to be aware of the state of the evidence of the psychometric performance of such PROMs. Therefore, the evidence of psychometric performance for each instrument was organised by whether the sample in the study was (i) a general population of children and young people, (ii) children and young people with mixed chronic conditions that included neurodisability, or (iii) specifically, children and young people affected by neurodisability.

Evidence for each of these populations was studied separately for the three groups of PROMs: generic PROMs, chronic-generic PROMs, and PBMs.

Appraisal criteria

Standardised criteria and thresholds were used to judge the evidence of psychometric performance of each candidate PROM. 38,39 The criteria used to assess and select patient-completed instruments included their appropriateness for measuring the health of children and young people affected by neurodisability, and an appraisal of their validity, reliability, responsiveness, precision, interpretability, acceptability and feasibility. 38 A summary of the criteria and indices used to judge psychometric properties is provided in Table 3 .

To demonstrate content validity, PROM developers should describe a clear conceptual framework underpinning the instrument, and incorporate qualitative research with potential respondents to inform development of the items in the questionnaire. This is also likely to ensure that the questionnaire has face validity to future potential respondents.

Construct validity concerns whether or not a scale is measuring what is stated as the purpose of measurement. Construct validity can be seen as comprising two aspects, internal and external. Internal construct validity is concerned with the valid structure of the scale, and is often examined through factor analysis. 40,41 Item response theory approaches also can apply an initial factor analysis, as they assume unidimensionality,42 but Rasch analysis often implements a post-hoc test of unidimensionality based upon analysis of the residuals. 43 External construct validity can take several forms; for example, hypothesis testing examines evidence of whether scales correlate well with other scales measuring a similar construct (convergent validity) or correlate poorly with instruments that are measured something unrelated (divergent validity). Correlations are considered low if r < 0.3, moderate if r lies between 0.30 and 0.49 and high if r < 0.5. 44 Hypotheses should be stated a priori, including the postulated direction and magnitude of correlation. 37 Discriminative validity describes whether or not an instrument detects ‘known differences’ between respondents.

Internal consistency is the extent to which all items in a scale are measuring the construct of interest and is assessed by Cronbach’s alpha statistic (α). Scales with an α statistic between 0.7 and 0.9 are considered to be composed of items that adequately measure a uniform construct. 45 The statistic assumes unidimensionality. If the assumption of local independence is violated, or there are simply a large number of items, Cronbach’s alpha may be inflated and/or an unreliable indicator. 45

All scores from PROMs include some level of measurement error, which can be estimated using calculations such as standard error of measurement;45 Rasch analysis estimates the measurement error for individual items (and persons) rather than the average of the scale level. 46

The reliability of instruments is determined by repeating administration on different occasions when respondents have not changed with respect to the construct being measured (test–retest). The level of agreement is also reported where child and parent responses are typically compared (inter-rater reliability). 38 Reliability coefficients are directly related to the variability in the population in which they are used;45 however, intraclass correlation coefficients (ICCs) exceeding 0.7 are generally regarded to indicate reliability for population-based research and ICCs exceeding 0.9 are considered to indicate reliability for use clinically with individuals. 45

Responsiveness describes the ability of instruments to detect important change when it has occurred. 47 Methods for evaluating responsiveness are either distribution based [effect size, minimal detectable change (MDC)] or anchor based [minimal important difference (MID)]. The effect size is a standardised measure of the magnitude of change, calculated by dividing the amount of change by the SD of the baseline score. 44 MDC is an indication of the amount of change required to have confidence that it is change beyond measurement error; a common standard is to use a 90% confidence level (MDC90). 48 The MID is the mean change in score reported by the respondents who indicate that they had noticed some small change. 49

Precision is concerned with the number and accuracy of distinctions made by an instrument. 50 Indices include how well the possible responses to each item are distributed over a true range. Use of Rasch analysis in scale development has suggested that scale scores vary in their precision (standard error) across the spectrum of the scale, with greater precision at the centre of the scale. Interval-level scales, such as those derived in the weighting of PBMs, inherently offer greater precision. A further aspect of precision is to examine whether there is any evidence of floor or ceiling effects, typically judged to occur when more than 15% of respondents’ scores aggregate at one end of the scale. 38

Acceptability to respondents is influenced by the design of a questionnaire, the number of items and the time necessary to complete the questionnaire. 51 Cognitive interviewing is a process to investigate how potential respondents understand PROM questions.

Appropriateness in this context is also dependent on evidence of excellent psychometric properties of candidate PROMs when evaluated with children and young people affected by neurodisability. Given that the purpose of the review is to identify and recommend a generic PROM for children < 18 years, and with different diagnoses under the umbrella of neurodisability conditions, we looked particularly for evidence of group invariance across age groups and different conditions. This would indicate that valid comparisons could be made across age and diagnostic groups.

Two practical issues considered were interpretability and feasibility. Interpretability is concerned with how meaningful the scores are produced by an instrument; indicators of interpretability are their face validity to those using the scores from PROMs. 51 Feasibility is also concerned with the researchers’ perspectives, and assesses whether or not the instrument is easy to administer and process, in terms of managing data and the calculation of scores. 52

Summarising evidence of psychometric performance

Several similar systems have evolved for summarising evidence to support psychometric properties of PROMs in systematic reviews. In our original protocol, we proposed using the system of the Oxford PROMs group: 0 for not reported; – for no evidence in favour; + for some evidence in favour; ++ for some good evidence in favour; and +++ for good evidence in favour. 28 The COSMIN group later proposed something similar: + positive rating; ? for an indeterminate rating; – negative rating; and 0 for no information available. 53 On balance, we elected to use a combination of these systems to summarise available evidence ( Table 4 ).

Summary ratings of evidence of psychometric performance were made using data extracted from included studies, with reference to whether or not standard criteria33,34 were met. When making the ratings, we also took account of the methodological quality of studies, number of studies, and giving further weight to any apparently independent studies that appeared not to have been conducted by the original developers. 54 We made an overall judgement separately for evidence emerging from studies conducted with (a) samples from general populations and (b) samples of young people with neurodisability.

Search results stage 1: identification of patient-reported outcome measures

The first search, to identify potentially eligible PROMs, resulted in 13,770 records after duplicates were removed. Following screening of the records by two independent reviewers, 832 abstracts were reviewed for names and/or acronyms of potentially eligible PROMs. This resulted in a list of 131 PROMs (see Appendix 4 ), of which 78 were excluded based on the exclusion criteria. The flow chart in Figure 1 illustrates the different steps in the selection process.

In total, stage 1 identified the names of 53 potentially eligible candidate PROMs, including 39 generic PROMs, seven chronic-generic PROMs and seven PBMs.

Search results stage 2: identification of studies evaluating psychometric performance of candidate patient-reported outcome measures

Search for eligible studies in general population samples (search 2.1)

The combination of the searches for each of the 53 individual PROMs in general populations resulted in 4830 records. Screening the deduplicated file of 2750 records resulted in 238 records that were selected for full-text screening. In total, we retrieved 218 full-text papers. We excluded 12 further PROMs (and their corresponding 38 papers), as they did not match our inclusion criteria. These were instruments that were developed for adults or were dimension-specific to mental health, assessed health-related behaviours, or were screening tools.

This reduced the number of eligible papers to 180, which described 41 PROMs. Another 75 records were excluded on closer examination of the full text; most papers were excluded because they had not administered an English version of the PROM (n = 24) or because a clinical but non-neurodisability group of children and young people had been studied (n = 15). The flow chart in Figure 2 shows the different steps in the process and details on the different exclusion criteria and number of papers excluded. This search process resulted in the selection of 105 papers reporting on psychometric evidence on one of the 41 eligible candidate PROMs.

FIGURE 2.

Flow chart showing identification and selection of studies evaluating psychometric performance of PROMs in general populations (search 2.1). ND, neurodisability. a, Some papers were excluded for more than one reason.

Search for eligible studies in neurodisability samples (search 2.2)

The combination of searches for the 53 individual PROMs used with neurodisability resulted in 2952 records. A total of 68 papers studying psychometric properties were selected based on title and abstract. After full-text screening, 13 papers were identified as eligible: nine were duplicates from search 2.1 and four new papers were selected for data extraction. Figure 3 gives an overview of the selection process.

FIGURE 3.

Flow chart showing identification and selection of studies evaluating psychometric performance of PROMs in neurodisability (search 2.2). ND, neurodisability. a, Some papers were excluded for more than one reason.

Search results summary

In total, 126 papers were selected for data extraction. These papers report evidence from evaluations of the psychometric performance of 25 PROMs (notwithstanding that some PROMs have more than one version) evaluated in an English-language questionnaire. No evidence was found for 16 other PROMs.

Eligible evidence was found for:

• 19 generic PROMs

• two chronic-generic PROMs

• four PMBs.

The evidence is grouped according to the study population:

• Eighty-four papers report results collected in a general population.

• Fourteen papers report evidence for a PROM administered in a group of children with various chronic conditions including neurodisability; the results are not presented separately for each individual chronic condition.

• Twenty-eight papers present results gathered in a neurodisability population.

The flow chart in Figure 4 illustrates the process of the selection of papers for data extraction.

FIGURE 4.

Summary of selection of papers for data extraction. CYP, children and young people; ND, neurodisability. a, Some papers have been excluded for more than one reason.

Data presentation

The results are presented within each category of PROM:

1. generic measures

2. chronic-generic measures

3. preference-based measures.

The following data are presented for each type of PROM:

1. Descriptive characteristics of all versions of the candidate PROMs.

   1. There are substantively different versions of some PROMs, either with different target age groups, varying items, domains or dimensions assessed, or responder, short and long versions, or revised versions. Each variation has been catalogued.

2. Content assessed by the PROMs coded using the WHO ICF-CY.

   1. The items of the questionnaires were mapped, as far as possible, to the chapter levels and domains of the WHO ICF-CY version.

3. Evidence of psychometric performance for candidate PROMs.

The data were further categorised by the sample with which the evaluation was conducted:

• general population

• mixed chronic conditions population including neurodisability

• neurodisability population.

The evidence in each of the study populations is presented as follows:

1. a description of the study reported in the selected paper: instrument version, author, publication year, aim or purpose, study population, number of participants, age range, mean age (SD) and setting/country

2. the methodological quality of the paper rated following the COSMIN checklist

3. a summary of evidence of the psychometric performance of the PROM with reference to whether or not the study population was a sample of the general population or neurodisability.

Generic patient-reported outcome measures

Initially, in stage 1, 30 generic PROMs were identified. For 11 PROMs, no evidence was found from eligible studies meeting our inclusion criteria: Pictured Child’s Quality of Life Self Questionnaire [Autoquestionnaire de Qualité de Vie Enfant Image (AUQUEI): QUALIN,55 AUQUEI Soleil,56 AUQUEI Ours,57 OK.ado Questionnaire58], the Child’s Health Assessed by Self-Ladder (CHASL),59 the Duke Health Profile – Adolescent version (DHP-A),60 Health And Life Functioning Scale (HALFS),61 How Are You? (HAY),62,63 the Illness Perception Questionnaire (IPQ, IPQ Revised, Brief IPQ),64 Infant Toddler Quality of Life (ITQoL long and short versions),65,66 Nordic Quality of Life (QoL) Questionnaire,67,68 Quality of Life Questionnaire for Children (QLQC),69 the Quality of My Life (QoML)70 and the Dutch Organization for Applied Science Research – University Medical Centre Leiden [Toegepast Natuurwetenschappelijk Onderzoek – Academisch Ziekenhuis Leiden (TNO-AZL)] questionnaires [TNO-AZL Questionnaire for Preschool Children’s Health-Related Quality of Life (TAPQOL),71 TNO-AZL Questionnaire for Children’s Health-Related Quality of Life (TACQOL)72 and TNO-AZL Questionnaire for Adult Health-Related Quality of Life (TAAQOL)73].

The authors and or developers of these PROMs were contacted to verify that we had identified all available peer-reviewed papers. We received responses from the authors of the AUQUEI, the CHASL, the HALFS, the Nordic Quality of Life and the QLQC who sent us a full-text version of the PROM; no additional eligible papers were received. The characteristics of these PROMs can be found in Appendix 5 .

Generic PROMs with evidence from studies using an English-language questionnaire that have more than one version include:

• Child Health and Illness Profile (CHIP) – age groups and short/long (four versions)74

• Child Health Questionnaire (CHQ) – child/parent and short/long (three versions)75–77

• Functional Status II Revised (FSIIR) – age group and short/long (six versions)78

• KIDSCREEN – long/short (three versions)79–81

• KINDL – age group (three versions)82

• Pediatric Quality of Life Inventory (PedsQL) – age group and short/long (10 versions)83–85

• Personal Wellbeing Index (PWI) – revised and age groups (three versions)86

• Student Life Satisfaction Scale (SLSS)87/Multidimensional SLSS (MSLSS) – revised and short/long (three versions)88–90

• Youth Quality of Life instrument (YQoL) – short/long (two versions). 91

Chronic-generic patient-reported outcome measures

In the selection process, five families of chronic-generic PROMs were identified. For three chronic-generic PROMs no evidence was found in an English-speaking population: the Child Health Ratings Inventory (CHRIs),188 the Children’s Life Quality Index (CLQI)189 and the Perceived Illness Experience scale (PIE and PIE-R). 190,191 The authors and or developers of these PROMs were contacted to verify that we had not missed any peer-reviewed papers; no response was received. These PROMs are not included in the report; characteristics of these PROMs can be found in Appendix 6 .

Patient-reported outcome measures with evidence that have more than one version include:

• DISABKIDS Chronic Generic Module (DCGM or DISABKIDS) – age group and short/long (three versions). 192–194

Preference-based measures

We found six eligible PBMs. Two PBMs were not included for further analyses, as no evidence was found in an English-speaking population: 16 Dimensional (16D)198/17 Dimensional (17D),199 and Assessment of Quality of Life Mark 2 – 6D Adolescents (AQoL-6D). 200 Characteristics of these excluded PROMs can be found in Appendix 7 .

Preference-based measures with evidence that have more than one version include:

• Health Utilities Index (HUI) – age group and short/long (three versions). 201–203

Key findings

We identified 25 PROMs: 19 generic PROMs, 2 chronic-generic PROMs and 4 PBMs. In total, 126 papers reporting evidence of the psychometric performance of 25 PROMs were selected. The psychometric performance has been tested either in a general population (84 papers, 23 instruments), a chronic-condition population including neurodisability (14 papers, five instruments) or a neurodisability population (28 papers, eight instruments).

Methodological reflections

The methodology for developing and evaluating PROMs has progressed over recent years. Most notably, there has been an increasing use of Rasch analysis to evaluate the structural validity and provide evidence for the precision of scores across the spectrum of measurement scales. Rasch analysis can also be used to test for any evidence of group invariance for items across age groups, sex, and between those with different diagnoses. Evaluation of these aspects of generic PROMs appears to have been relatively unexplored. An exception is that item invariance between general populations and children with cerebral palsy has been evaluated for KIDSCREEN, with only minor differences reported. 125 Therefore, although several generic PROMs have been used with children and young people affected by neurodisability, little can be concluded with respect to whether or not generic PROMs perform consistently across the various conditions within neurodisability.

The International Society for Pharmacoeconomics and Outcomes Research Good Research Practices Task Force for Developing Pediatric Patient-Reported Outcomes sets out five good practices relevant for PROMs for children and young people: (i) attention to age groups and age-based cut-offs, (ii) establishing content validity using concept elicitation to inform item generation, and cognitive interviews to assess and refine all aspects of the draft instrument, (iii) consideration of whether or not proxy report is necessary, (iv) ensuring that the instrument is designed and formatted appropriately for the target age group, and (v) considering cross-cultural validation.

Strengths and limitations

This was a comprehensive systematic review utilising a two-stage search strategy to maximise the possibility of identifying all candidate PROMs, and any eligible evidence of their psychometric performance in general and neurodisability populations. In addition, we contacted the authors of instruments for whom no evidence was found and conducted a forward citation search using a key reference for each PROM.

Our aim was only to appraise evidence of the psychometric performance of generic PROMs with children and young people affected by neurodisability. However, we decided to include appraisal of evidence from studies with general populations in order not to exclude emerging measures or those instruments with promising possibilities for use with neurodisability. This made the task much larger, but makes the findings more likely to represent the actual state of knowledge about psychometric performance of generic PROMs. We included studies with children and young people with chronic conditions, providing that the samples included neurodisability. Hence, there is a body of research that was not included in this review, which is those studies examining performance of PROMs with children and young people with conditions that would not be considered neurodisability (e.g. arthritis or asthma).

Excluded from the scope of the review were clinical assessments and instruments designed to be administered by clinicians or trained assessors. Hence, there are several outcome measures that may be commonly used in research that would not be consistent with the definition of a PROM. Examples include various forms of adaptive behaviour assessments, and for instance, the interview-administered Assessment of Life Habits (LIFE-H). 222 Also specifically excluded were domain-specific instruments. Thus, instruments that only measure one aspect of health, for example the Strengths and Difficulties Questionnaire, which is often used to assess mental health and behavioural problems, were not considered. 223

There are some limitations to this systematic review; most are a consequence of the strict inclusion criteria. Limiting the search to studies where an English version of the PROM was administered excluded some PROMs from further analyses. In addition, cross-cultural validity was not comprehensively assessed for the selected PROMs. Two PROMs excluded from this review that may warrant further investigation are ITQoL (for infants), which was developed in the Netherlands and for which an English translation is available but no published studies of this version were found, and the TNO-AZL family (TACQOL, TAPQOL and TAAQOL). If studies were included that were conducted using versions of questionnaires in languages other than English, then further evidence would emerge, for instance regarding the KINDL and the plethora of translated versions of the more popular instruments such as PedsQL. Nevertheless, psychometric performance cannot be assumed across languages and cultures;217 therefore, in our view, limiting the review to evaluations of English-language versions is a relative strength of the review.

The WHO ICF-CY was used to create a content profile for each of the questionnaires. In carrying out this analysis, many items could not be mapped readily onto the ICF-CY, and some items could not be mapped at all. This is both a problem with the conceptualisation and design of the ICF-CY, which focuses on function and is less concerned with aspects of well-being, and problematic because the questionnaires conflate these concepts. PROMs are often composed of items that assess various concepts in a single item; the latter is also problematic in view of the cognitive task faced by respondents.

Another problem encountered with using the ICF-CY to describe the content of the questionnaires was how to deal with the way a question is asked. For example, the KIDSCREEN asks ‘have you felt fit and well?’ and the CHIP asks ‘how often did you play hard enough to start sweating and breathing hard?’. Both questions can be mapped on ‘b130 Energy and drive functions’; however, each item assesses something quite different, and may be answered differently. The KIDSCREEN predominantly assesses how children and young people feel about their health; the CHIP focuses mostly on functioning, asking what children and young people can do. Describing the content of PROMs using the ICF-CY is likely to lead to a loss of information, without reference to the context and precise focus of the question.

The COSMIN checklist was used to rate the methodological quality of the studies. The aim and purpose of this exercise should be carefully considered in future systematic reviews, as it is a time-consuming task. There is undoubted benefit from identifying and considering the methodological quality of studies evaluating psychometric properties of PROMs. However, in our quality assurance checks with a second reviewer, we found an issue in the consistency of how those making the ratings interpret of some parts of the COSMIN checklist. The most difficult COSMIN item to code consistently was ‘how missing items are dealt with’, and this item has a strong influence on the overall quality rating for most psychometric properties. The procedures for handling missing data may not have been reported in all papers that were included in the review, but may have been detailed in other papers or be detailed in the manual of the PROM.

We did not examine the methods used to derive the preference weights applied in the scaling of the PBMs appraised in the review; the methods for creating the preference weighted scales were assumed to meet the requirements for creating interval-level measurement. 224 As the fundamental purpose of preference-based measurement is to quantify the value or strength of preference for health change, the means for assuming and eliciting preference values should be critically assessed. 225 Not all of the standard criteria for appraising PROMs are proposed to be appropriate for evaluating PBMs; for instance, the requirement for internal consistency may conflict with the underlying theory. 225 Nevertheless, most criteria for face, content and construct validity and test–retest and proxy reliability remain applicable;225 these properties of PBMs were found to be largely untested with children and young people.

Our inclusion criteria only considered published peer-reviewed studies that specifically set out to evaluate psychometric properties of PROMs. Hence, we will have excluded studies that may present incidental evidence of psychometric performance. For instance, there may be indications of ability to measure change, over time or with interventions, for any candidate PROMs that have been used in longitudinal observational or experimental studies. In addition, we will have omitted any information that may be contained in manuals, if these data have not been published in peer-reviewed journals.

Theoretical framework

This was an applied health services research project with specific objectives and required to relate to the other research streams. This qualitative research was not seeking to elicit a deep understanding of the ‘lived experience’ of the young people and parents. Hence, the analytical approach reflected these issues and purposefully addressed focused policy-relevant questions. The WHO ICF-CY was used as a theoretical framework to link the findings to the other research streams.

Topic guide development

The first discussions to develop a topic guide were convened in a meeting of the qualitative research team and four parents, held as part of the co-investigator meeting in autumn 2011. These discussions highlighted two key issues. First, there was a need to distinguish between ‘health outcomes’ as defined for this research study, and people’s experiences of health services; second, the concept of ‘health outcomes’ was not readily understood by parents.

Children and young people

For children and young people, an approach was developed using a hypothetical character in order to depersonalise issues and to encourage them to raise issues that they might otherwise be reluctant to reveal. They were presented with a scenario where an imaginary young person was affected by a health condition, and asked to suggest what health problems they had. Participants were then asked about ways in which the character’s health could be improved. After a break, participants were shown examples of PROM questionnaires, and exemplar questions were extracted from PROMs that were ‘blown up’ as posters, on which young people could write directly or make suggestions for the facilitator to annotate.

The approach was piloted with an established disabled young people’s group in Torbay that was convened on behalf of the local authority by The Children’s Society. The pilot focus group was led by AF and observed by CM. Some modifications were made in the light of this experience, particularly that prompts were helpful to encourage young people to consider aspects of physical, emotional and social health and that there should be less focus on the potential health problems of the character.

This led to the development of a poster to introduce the concept of ‘health outcomes’ to the participants ( Figure 5 ). The poster showed a disabled young person, images of various health professionals, and a road leading to a sunny destination. Participants were invited to give the disabled young person a name and then asked to talk about which health outcomes would be important to the character when they arrived in ‘Outcomes Town’. To help structure the conversation in a way that enabled the participants to better understand and engage with the concept of outcomes, the facilitator (AF) introduced broad outcome areas, including physical health, emotional and mental health, and social and community life. This facilitated discussion of more specific topics; issues such as communication, pain and having friends were introduced as appropriate. Prompts were used to explore participants’ views on (i) how the character in the poster might feel about each of the issues, (ii) whether or not an issue was likely to be important to them and (iii) what could be done could help improve the outcome. This structured conversation gave the participants the opportunity to explore and define the outcomes according to the importance they attached to them.

FIGURE 5.

Poster used to conceptualise the journey towards outcomes.

After the outcomes discussion, the children and young people were shown examples of PROM questionnaires in small groups. They were encouraged to write on these forms to express their reactions, and supported by the facilitator (AF) and support staff to explore and evaluate the content of the forms in small groups. The small groups then came back together to discuss the forms as a single group and were asked to make suggestions on how the forms could be improved to be more attractive to other children and young people affected by neurodisability.

Adaptations and modifications to the methodology

After the fourth focus group, three themed posters were developed to help participants to explore key emerging concepts from the research in more detail ( Figure 6 ). The posters encouraged the participants to explore some of the physical health outcomes, mental and emotional health outcomes, and wider life outcomes that were emerging from the research, and to suggest additional outcomes that they felt were missing.

FIGURE 6.

Three health posters used to conceptualise physical health outcomes, mental and emotional health outcomes.

Further significant modifications to the topic guide were required to include young people with profound communication impairments. This group was included by using a ‘Talking Mat framework’ ( Figure 7 shows an example). The facilitator (AF) collaborated with a speech and language therapist to create cards with illustrations that would be mostly familiar to young people using augmented communication to represent all the aspects of health that had been identified in the previous focus groups.

FIGURE 7.

Example of Talking Mat board used to enable young people with communication impairments to express their views.

A total of 157 cards were compiled as a resource to facilitate discussion about 10 outcome topics: moving about, senses and pain, mental functions, activities, communication, participation, education, feelings, body functions, and diet. Participants were invited in turn to choose which of the outcome topics they wished to discuss. Subsets of symbol cards were then presented, relating to the specific outcome selected. For example, when a participant selected the illustrated card for ‘mental functions’, they were then presented sequentially with cards that related to sleep, memory and concentration. Participants were asked if each of these symbols was for them ‘good’, ‘bad’ or ‘sometimes good and sometimes bad’, and asked to indicate which topics were ‘more important’ to them, ‘somewhat important’ or ‘not important’.

Young people’s personal choices of outcome cards were grouped on their Talking Mat boards, and their interactions were captured using audio recording and photographs of completed boards.

Recruitment and sampling

Interviews and focus groups were led by AA and AF of Council for Disabled Children (CDC). CDC is the leading national policy and practice improvement organisation for disabled children in England. CDC has considerable experience of consulting disabled young people and their parents on a broad range of issues. CDC maintains networks of disabled young people’s groups and parent groups. These networks provided expedient sampling frames for contacting children, young people and parents, including various conditions and age groups. CDC is also able to contact large numbers of families of disabled children through news items on their website and using Twitter and Facebook social networking sites.

The Making Ourselves Heard network comprises 271 contacts across England, including the major providers of services to disabled children from within the voluntary sector and local authority leads from within both youth services and disabled children’s teams. CDC also contacted professionals working with individual and groups of disabled children and young people who applied to be on the Department for Education Young People’s Advisory Group. CDC works closely with the National Network of Parent Carer Forums, which is the umbrella body for all 152 local authority forums for parents of disabled children.

An invitation was advertised through these networks and using other media for individuals and groups of young people affected by neurodisability, and similarly but separately for parents, to volunteer to participate in this research. Volunteers registered their interest directly with CDC. Volunteer individuals and group leaders were screened by telephone to ensure that they were eligible, as CDC has a non-categorical approach to disability and some volunteers were not affected by health conditions included within the project definition of neurodisability.

Focus groups and interviews were purposively selected and strategically arranged to capture diagnostic, demographic and geographic variation. For instance, two groups were held in London in order to include black and minority ethnic communities. Interviewees were subsequently selected to address gaps identified in the sampling frame; for instance, interviews were arranged in the north-west, where it had not been possible to arrange a focus group.