A programme of studies including assessment of diagnostic accuracy of school hearing screening tests and a cost-effectiveness model of school entry hearing screening programmes

Search strategy

The search strategy used and published in the 2007 HTA report on SES12 was reviewed and updated by an information specialist in the Peninsula Technology Assessment Group and re-run to identify studies published in the period January 2005 (search cut-off date of 2007 report, May 2005) to July 2014 (see Appendix 2). The following electronic databases were searched: The Cochrane Library (via Wiley) (Cochrane Database of Systematic Reviews, Cochrane Central Register of Controlled Trials, Database of Abstracts of Reviews of Effects), MEDLINE and MEDLINE In-Process & Other Non-Indexed Citations (via Ovid), EMBASE (via Ovid), Cumulative Index to Nursing and Allied Health Literature (via Ovid), PsycINFO (via Ovid), Science Citation Index (via Web of Science), Education Resources Information Center and ongoing trial databases (National Research Register, ClinicalTrials.gov and Research Findings Register).

Inclusion/exclusion criteria

The inclusion and exclusion criteria were identical to those of the 2007 HTA report12 unless stated otherwise:

• Study design: we included all primary diagnostic accuracy studies regardless of their specific design.

• Population: we included children 4–6 years old. We excluded studies with a discrete age range that was completely outside our criteria (e.g. 1–3 years or 7–10 years). If a study included 4- to 6-year-old children but the age range was too wide (e.g. 6–19 years) and the results for different age categories were not reported separately, the study was also excluded from the review. We included, however, studies that at least partially covered but slightly exceeded the 4–6 years age range (e.g. 5–10 years) provided they met all other inclusion criteria. Whenever relevant and possible, the age of the included children is noted in the list of excluded full-text studies (see Appendix 2).

• Screening tests or programmes: studies that evaluated one or more of the following hearing screening tests were included: sweep PTA, single-frequency PTA, transient-evoked otoacoustic emissions (TEOAE), distortion product otoacoustic emissions, questionnaires, otoadmittance tests, tympanometry, reflectometry and speech audiometry. (Note: sweep PTA and PTS are alternative terms for the same procedure. Where publications have used the term sweep PTA this description has been maintained.) Tests had to have been undertaken in either a primary school or the community [e.g. community clinic, family home or general practitioner (GP) surgery]. This could include hearing screening as a component of a multifaceted screen, such as a school entry medical examination.

• Comparator: no hearing screening or hearing screening based on different tests or test protocols. We did not exclude studies without a comparator (studies that evaluated a single screening test) as long as they measured the performance of the evaluated test against an acceptable reference standard.

• Reference standard: we included all studies that assessed the accuracy of the evaluated test(s) against a reference standard that included PTA.

• Outcomes: the 2007 HTA review12 had a wider scope and also included studies that reported (1) the screen performance, that is, uptake (the number of children who actually received the screen) and yield (the number of cases identified); and (2) screen effectiveness, that is, language skills, health-related quality of life, communication skills, social interaction and educational performance. The current update focused on diagnostic accuracy only and included studies that reported the diagnostic accuracy of the evaluated test(s), regardless of whether or not the reported data were sufficient to reconstruct two-by-two table(s).

• Language: no language restrictions were applied to the search and selection of studies.

Selection of studies and data extraction

After removing the duplicates, all records identified by the electronic searches were screened independently by two reviewers (HF and ZZ) at title and abstract level. Full-text copies were obtained for all publications identified as potentially relevant by at least one of the reviewers. Their suitability for inclusion in the review was assessed by one reviewer (ZZ) and checked by a second reviewer (CH) against the criteria specified above. Data were extracted from included studies by one reviewer (ZZ), entered into an Excel 2010 (Microsoft Corporation, Redmond, WA, USA) spreadsheet and checked by a second reviewer (CH). Disagreements in the selection of studies and data extraction were resolved through discussion. Data were extracted from studies published in a language other than English with the help of a translator.

Assessment of the methodological quality

Although a new version of the quality assessment of diagnostic accuracy studies (QUADAS) tool used in the 2007 HTA review12 is now available, the difference between the two versions is mainly in the structure and process of customising the tool and does not concern the contents of the actual checklist. 18,19 Therefore, we assessed the methodological quality of the included studies using the original QUADAS checklist from the 2007 HTA review. This allowed for a direct comparison between the quality of the studies included in the 2007 report and in the current update. However, we provided more specific definitions of some QUADAS items that were not explicitly defined in the original checklist (see Appendix 2).

Statistical analysis and data synthesis

Whenever possible, two-by-two tables were used to report the numbers of true-positive, false-positive, false-negative and true-negative results. The sensitivity and specificity of the index test (the test under evaluation) were calculated using The Cochrane Collaboration’s Review Manager (RevMan) software version 5.3 (The Cochrane Collaboration, The Nordic Cochrane Centre, Copenhagen, Denmark). The same software was used to create forest plots of the paired sensitivity and specificity estimates and to plot the true-positive rate (sensitivity) versus the false-positive rate (1 - specificity) in receiver operating characteristic (ROC) space. Heterogeneity was investigated visually by examining the forest plots and the ROC plot, and a decision was made whether or not to pool the results across studies. As standard funnel plots and tests for publication bias are not recommended in systematic reviews of diagnostic accuracy studies, we did not investigate publication bias. 20

Search results and selection of studies

The electronic searches returned 892 papers after removal of duplicates. Screening at title or/and abstract level led to 39 citations being identified as potentially relevant. A further seven citations were found through hand-searching.

We obtained full-text copies of 45 of these studies and failed to obtain one. 21 After full-text evaluation 10 studies were selected for inclusion in the review. 17,22–30 Figure 2 details the selection process while the reasons for exclusion of full-text papers are given in Appendix 2.

FIGURE 2.

Flow diagram of the selection process.

Characteristics of included studies

The characteristics of the included studies are summarised in Tables 1 and 2. Briefly, 10 primary studies were included in this update. 17,22–30 One was published before 200524 but was included here as it had been missed by the original 2007 HTA report. 12 The other nine studies were published in the period 2005–2014. 17,22,23,25–30 Four of them were conducted in China22,23,27,30 and one in each of the following countries: Brazil,25 Greece,29 Japan,26 Kenya,24 the Philippines17 and the USA. 28

Although the majority of the included children were at or around school entry age, there was marked variability in terms of age range. Children’s age ranged from 225,28 to 13 years23 with the mean age ranging from 5.1 years [standard deviation (SD) 0.7 years]27 to 9.4 years (SD 1.8 years)22 (three studies25–27 did not report mean age and only gave age range as an inclusion criterion). The studies were relatively balanced in terms of participants’ gender, the proportion of males ranging from 48.4%26 to 62.5%30 (two papers23,28 did not report details). Apart from lack of consent, most studies did not specify any other exclusion criteria. One study26 excluded children who were unable to lie down still for several minutes, which was a necessary requirement for completing the screen procedure; one study27 excluded children with learning disabilities and two studies26,30 commented that no children with known cognitive or hearing impairments were included but it was unclear if this had been a prespecified exclusion criterion. Seven studies17,22–25,27,30 were conducted in countries without established UNHS.

Studies evaluated the performance of questionnaires (n = 4), audiometry (n = 3), TEOAE (n = 3) and automated auditory brainstem response (AABR) (n = 1). AABR is usually used to screen infants and was not listed in the initial inclusion criteria. However, we decided to include the study, as it evaluated, probably for the first time, the use of AABR in SES. Two studies22,23 evaluated different versions of the Chinese Hearing Questionnaire for School Children (CHQS). Only one study22 compared directly the performance of two different tests, a questionnaire and TEOAE, and one study29 evaluated a combination of TEOAE and tympanometry but did not report the results as adding tympanometry had not improved performance.

Questionnaires were completed by parents or carers in two studies,22,23 either by parents/carers or by teachers or community nurses interviewing parents (three different testing conditions) in one study24 and by unspecified evaluators in one study. 25 All other index tests were performed either by audiologists, otolaryngologists or other professional staff who had some preliminary training. With the exception of questionnaires, all other tests were performed at school or a community health centre in a quiet but not sound-treated room and the ambient noise level was monitored.

All studies except those evaluating questionnaires used a prespecified positivity threshold to define ‘pass’ and ‘refer’ outcomes. Data-driven selection of a threshold to achieve optimal performance (best sensitivity and/or best specificity) could lead to overly optimistic diagnostic accuracy estimates and the test is likely to perform worse when the same threshold is used in an independent sample of patients. 31

The reference standard was a combination of otoscopy, tympanometry and audiometry in four studies,22,23,25,29 otoscopy, tympanometry, audiometry and OAEs in one study,27 ear, nose and throat (ENT) examination including audiometry in two studies,24,26 and audiometry only in three studies. 17,28,30 The tests were performed by audiologists and otolaryngologists, usually in quiet but not sound-treated rooms with monitored ambient noise.

The mean prevalence of the target condition across all studies that reported outcomes at the level of the individual was 10.8% (SD 12.6%) and ranged from 0.7%28 to 46.7%. 25 The prevalence in the only study that reported ear-level outcomes was 4.8%. 17 These numbers should, however, be treated with caution, as the studies varied in their pass/refer criteria and included convenience samples drawn from populations likely to be different in terms of prevalence and spectrum of hearing impairments.

Methodological quality of included studies

The definitions of the assessment criteria are given in Appendix 2 and the results from the methodological quality assessment are summarised in Table 3. All studies had a prospective cross-sectional single-gate design. ‘Single-gate’ is a term introduced by Rutjes et al. 32 to describe diagnostic accuracy studies in which a single sample drawn from the target population receives the index test and the reference standard to allow calculation of the test’s sensitivity and specificity. In contrast, in ‘two-gate’ designs sensitivity and specificity are calculated separately based on two different samples of participants, that is, one with and one without the target condition. The two-gate design is prone to spectrum bias as the mix of participants in the two samples combined is unlikely to be representative of the target population. 32 Therefore, studies using a single-gate design are considered to be of better methodological quality.

Although all included studies had a single-gate design, none of them included a representative sample of a relevant target population defined for the purpose of this review as 4- to 6-year-old children (± 1 year) at or around school entry stage who have no high-risk features (such as Down syndrome, cytomegalovirus infection or meningitis) and have been tested and found to have no hearing impairment at birth. Therefore, the studies are likely to suffer from a selection bias and to have limited applicability to the UK context because of the following methodological issues. First, some studies included children younger or older than the defined 4–6 years age range, which reflects, to some extent, the fact that school entry age varies across countries. Second, seven of the studies17,22–25,27,30 were conducted in countries without established UNHS, which is likely to impact on the prevalence and spectrum of hearing impairments in the included children. Third, most of the studies included small self-selected samples recruited from a single locality and, therefore, may not be representative, even when drawn from a relevant target population.

The execution of the index test(s) and the reference standard were reported in sufficient detail in the majority of the studies. In five studies,17,24,26,28,30 however, the reference standard was suboptimal and did not meet the quality criterion ‘PTA + tympanometry’. The time between the performance of the index test and the reference standard was < 1 month in four studies,17,22,23,30 was > 1 month in one study28 and was not reported in five studies. 24–27,29 Blinding of the index test evaluators to the results of the reference standard was not reported in three studies22,26,29 and blinding of the reference standard evaluators to the results of the index test was not reported in five studies. 22,23,26,27,29 Those criteria that were consistently met across the studies were questions 5–9 (see Appendix 2) concerning the application of the same reference standard to the whole or random sample, regardless of the index test result (verification bias); the independence of the reference standard from the index test (incorporation bias) and the description of index test and reference standard in sufficient detail to allow replication. According to the criteria for calculating a total quality score published in the 2007 HTA report,12 three studies25,26,29 were of ‘moderate’ quality (total score 7–9) and the remaining seven studies17,22–24,27,28,30 were of ‘good’ quality (total score of > 9).

Test accuracy

Given the significant heterogeneity in the study characteristics and the reported test accuracy estimates (see Table 2 and Figure 3) we considered quantitative synthesis inappropriate and, instead, summarised the performance of different test types in tables and figures (Figures 3 and 4, and Table 4).

FIGURE 3.

Forest plot of sensitivity and specificity of different types of hearing screening tests. Please note that the results for Bu et al. 22 are based on reading off the ROC plot in the paper and do not refer to a particular positivity threshold; the results for Li et al. 23 are based on ‘1+’ answers suggesting hearing impairment; and the results for McPherson et al. 30 are based on the data set that did not include results from 0.5 kHz. CI, confidence interval; FN, false negative; FP, false positive; TN, true negative; TP, true positive.

FIGURE 4.

Summary ROC plot of different hearing screening tests. Please note that the results for Bu et al. 22 are based on reading off the ROC plot in the paper and do not refer to a particular positivity threshold; the results for Li et al. 23 are based on ‘1+’ answers suggesting hearing impairment; and the results for McPherson et al. 30 are based on the data set that did not include results from 0.5 kHz. CI, confidence interval.

Summary of the findings from the 2007 Health Technology Assessment report

The 2007 HTA report12 included 25 primary studies reporting the performance of a wide range of hearing screening tests, including parental questionnaires, impedance audiometry/tympanometry, spoken word tests, otoscopy, audiometry, TEOAE and other tests and combinations of tests. The overall conclusion was that evidence was of unacceptable variability in terms of methodological quality and study characteristics and, as a result, drawing strong conclusions about the performance of different tests was not possible.

With those caveats taken into account and including only the subset of studies that used PTA as a reference standard, the findings from the 2007 HTA report suggested that: sweep PTA had high sensitivity and specificity; spoken word tests had acceptable levels of sensitivity and specificity; TEOAE had high specificity but somewhat lower sensitivity; tympanometry and acoustic reflectometry had variable sensitivity and specificity; parental questionnaires and otoscopy had poor sensitivity and specificity; and there was insufficient evidence to comment on the accuracy of combinations of tests (p. 48).

Combining the results from the 2007 Health Technology Assessment report and the current update

The studies identified in the current update provide additional test accuracy data for the following three categories of tests evaluated in the original HTA review: parental questionnaires, sweep PTA tests and TEOAE. Below we discuss the combined evidence from the two sets of studies for each category of tests.

Design

This study used a directly comparative two-gate (‘case–control’) design,32 with separate sources (gates) used to recruit those with the target condition (hearing impairment – cases) and those without (no hearing impairment – controls). Case children were recruited through audiology services in England whereas the control children were recruited largely from Nottinghamshire primary schools. The implications of this are discussed later in this chapter (see Discussion).

Recruitment

Cases were children aged 4–6 years recruited between February 2013 and August 2014 who were identified by collaborating audiology services [in Nottingham, Sheffield, Leicester (City and County), Chesterfield, Derby, Mansfield, Lincoln, Birmingham, Huntingdon, Bradford, Rotherham and Doncaster] and who had permanent sensorineural or conductive hearing impairment averaged across the four frequencies 0.5, 1, 2 and 4 kHz, either bilaterally (average of 20–60 dB HL) or unilaterally (any level ≥ 20 dB HL). The senior paediatric audiologist in each centre drew up a list of all children meeting the definition criteria. Audiologists were given stamped envelopes to address and post to potential participants and agreed together with the researchers when to send the letters. Each identified family was sent a letter on behalf of the research team inviting them to take part in the study, together with information about the study. Children were not invited to take part if they were unwell such that their illness would affect the results of the tests or if the responsible audiologist felt it would be inappropriate or cause added unnecessary burden (e.g. seriously or terminally ill family member). Parents willing to take part replied directly to the research team. Eligible children for whom agreement was provided to take part were invited to undergo the two screening tests, either in their own homes, or at Nottingham Hearing Biomedical Research Unit (NHBRU), depending on their preference. Researchers sent holding letters if necessary for children < 4 years old, explaining that an appointment would be made for them after their fourth birthday. The reference standard definition of hearing impairment was based on PTA results, thus case children were also excluded if there was no record of a PTA in the previous 12 months or planned for the following 3 months, and if the family was unwilling to travel to their local service or to Nottingham to undergo the assessment.

Control children, defined as having no previously identified hearing impairment, were recruited from the Foundation Year and Year 1 of schools in the Nottingham area, between February 2013 and June 2014. The study researchers provided a letter of invitation and information pack for the school to distribute to all parents of children in the aforementioned year groups; invitation methods were agreed with the school. Children for whom agreement to take part was provided were invited to undergo the two screening tests and a PTA at NHBRU. Children needed to have the PTA measured in a soundproofed room, and hence the option to have the screening test at home was not possible.

To help ensure that the cases and controls were representative of the sources from which they were drawn, the audiologists and researchers were advised to not just pick those whom they thought might be easy to test. For all children the invitation pack contained an invitation letter, a one-page summary information sheet, a pictorial information sheet for the children and a pre-paid return envelope. Full participant information sheets were sent to the parent (either via e-mail or post) once an appointment was made (see Appendix 3) (the full sheet was originally included in the invitation pack, but a revision was made in December 2013, to be less overwhelming for the parent and to save paper).

Parents of all children involved were offered the opportunity to receive a short summary of the findings at the end of the study. A £20 book token (revised from £10 after 7 months of recruitment in an attempt to improve recruitment) was given to each child as a thank you for his/her time and inconvenience. Travel expenses to NHBRU were reimbursed in line with University of Nottingham standard policy. Schools that participated in the recruitment of control children were entered into a prize draw with a chance to win £100.

Assessment

Once a reply slip was received, the researcher contacted the parent (via telephone, e-mail or letter) to make an appointment either at NHBRU (case and control children) or at their home (case children only). They were also asked about access issues or translator needs. Parents were usually reminded about the appointment by telephone or e-mail on the day prior to the appointment. After case children had their appointment, the researcher phoned the audiologist, asking them to post their most recent PTA results to NHBRU. This could be a PTA measured in the previous 12 months or one that is scheduled to be measured in the following 3 months.

For all appointments the researcher checked that the parent had all the study information, explained what would happen at the appointment and how that fitted into the project, and answered any questions they might have. The consent form was explained and written consent obtained. The researcher emphasised that participation was voluntary and consent regarding the child’s participation in the project could be withdrawn at any time without penalty or affecting the quality or quantity of the child’s future medical care, or loss of benefits to which the child and his/her family was otherwise entitled. It was explained that in the event of withdrawal, their data collected so far could not be erased and that consent would be sought to use the data in the final analyses where appropriate. The informed consent form was signed and dated by the parent or legal guardian and researcher before the child entered the study – usually at the start of the screening appointment. Copies of the consent form were kept by the parent or legal guardian, the researchers, and, for case children only, in the child’s hospital records (see Appendix 3).

Data collection was planned prospectively in advance of administering the tests and reference standard to the children. Background details [visit date, school for control children, hospital for case children, date of birth, postcode, background sound level, gender, ethnicity, medical conditions on the day (including respiratory infections)] were recorded on the case report form (CRF) and testing was then administered.

The audiometer (used for the PTS testing for all children and for the PTA for control children) and HC device were checked each day, by listening to the tones at the testing frequencies, to ensure they were audible. Valid calibration was undertaken in the previous 12 months for the audiometer and sound level meter and in the previous 3 years for the HC screener according to manufacturers’ recommendations. Headphones were cleaned with a disinfectant wipe before each child was tested and new disposable cups were used for the HC screener. Items likely to cause noise (e.g. mobile phones, washing machines, televisions) were turned off where possible, and the level of background noise was measured with a sound level meter. Although, for control children, the door of the room in which the two screening tests were conducted was left open, and sometimes siblings were present, the ambient noise conditions were much quieter than rooms that are usually available in schools for conducting the hearing screen. For case children, where the screens were conducted at home, attempts were made to minimise the noise levels.

The researchers were trained in administering the PTA and PTS tests by the audiologists in the Children’s Hearing Assessment Centre (CHAC) in Nottingham, using a mixture of observation, practice with children and feedback. Further familiarisation with the equipment and procedures was gained by testing staff from the research department.

The order of administering the two screening tests and which researcher undertook them were each determined by separate lists based on computer-generated simple (unrestricted) random numbers. Each test was administered to both ears. For case children, one researcher performed the PTS and another researcher performed the HC screen. For control children, one researcher carried out both the screening tests and then another researcher performed the PTA measurement. An effort was made to blind the second researcher to the results of the first test(s) by asking them to leave the room. The PTA result for case children obtained from their audiologist (measured within 12 months before or 3 months after the study visit) was examined only after the result of the screening tests were known.

Screening tests

HearCheck screener

The HC41 screener (Figure 5) automatically generates six tones in total – one tone for each combination of frequency and level: 1 kHz at 55, 35 and 20 dB HL and 3 kHz at 75, 55 and 35 dB HL. The tasks and test were explained, in particular that there was one tone much louder than the others. Children were asked to raise a hand when they heard a tone. The response method for the HC screen had to be quick to perform as there was only a very short automated gap between the tones, and was also chosen to be different from that for the PTS to keep the child’s interest. For some children a different response method (e.g. tap the table) was used if they did not seem able to raise their hand for whatever reason. The child was asked to remove hearing aids, and also glasses and earrings if necessary for a good fit. A disposable cardboard ear cover was put onto the HC screener for each ear. The HC screener was held against the ear to be tested, often holding the child’s head still with the free hand. The button was pressed and the first three tones were allowed to play for the first frequency. The button was then pressed again for the remaining three tones for the second frequency. The procedure was repeated on the other ear. A tick was recorded on the CRF for a response, and a cross for no response. The refer criterion for each ear is anything less than all six tones being responded to.

FIGURE 5.

The HC screener.

Data collection

Each page of the CRF for each child was identified with the participant identity code, date of birth and partial postcode. The researchers entered the data from the CRFs into the study database. Copies of the CRFs were then sent to Peninsula Clinical Trials Unit (PenCTU) for second data entry and consistency checks. Emerging queries were noted to the Nottingham researchers and corrected where appropriate. Recruitment logs (for children nominally recruited as cases and controls, audiologist and school), the password-protected appointment spreadsheet and progress documents were completed as appropriate. Consent forms, reply slips and CRFs were filed in a secure place at NHBRU and then archived in a secure facility.

Sample size calculation

The target sample size was 80 hearing impaired (HI) case children and 160 not hearing impaired (NHI) control children. Eighty HI children is a large enough sample to estimate a sensitivity of 80% with a margin of error of 10.4% based on the lower bound of the 95% CI; 160 NHI children is large enough to estimate a specificity of 80% with a margin of error of 7.0%. The margin of error for these estimates provides plausible ranges of values within which to test the sensitivity of the results from the economic model to our assumptions about screening accuracy (see Chapter 8).

Statistical analysis

The characteristics of the children were summarised separately for those nominally recruited as case children (i.e. with impairment) and those nominally recruited as control children (without impairment), using mean and SD for age, and numbers and percentages for gender and ethnicity. For case children, the time interval in weeks between measurement of the PTA reference standard and administration of the screening tests was summarised using the mean with SD and median with interquartile range (IQR).

The criteria used to define recruitment of case children were based on information available from the PTA. The children who were recruited as controls had no known hearing impairment. However, it was possible for children nominally recruited as cases to pass some elements of the reference standard and for children nominally recruited as controls to receive a refer result from the reference standard. Evaluation of the diagnostic accuracy of the screening tests was based on using the PTA reference standard, and not the nominal recruitment status of the children (i.e. it was not based on the ‘gate’ through which they were recruited). We classified those who were referred by PTA as HI and those who passed PTA as hearing (or NHI), regardless of whether they were nominally recruited as cases or controls. The discordance between nominal case–control status and the PTA reference standard classification is summarised later.

The accuracy of each of the screening test results was assessed in relation to the PTA reference standard classification based on analyses at the level of individual ear and at the level of individual child. From the outset we considered the ear-level analysis to be primary to the objectives addressed in this chapter because it directly addresses the question of the accuracy of the tests for discriminating between hearing and impaired ears. We also present child-level analyses, however, because the resulting accuracy estimates from these are most relevant for the economic model reported in Chapter 8. For the child-level analyses hearing was considered impaired (HI) if they had at least one impaired ear on the PTA reference standard and considered to have been referred by a given screening test if at least one of their ears was referred by the test.

When examining the relationship between the screening test results and the reference standard, separate sets of analyses were carried out for two different definitions of hearing impairment status. Under the primary definition, impairment was considered present when the PTA reference standard threshold was ≥ 30 dB on at least one of the four frequencies examined (0.5 kHz, 1 kHz, 2 kHz and 4 kHz) and considered absent when the reference threshold was < 30 dB on all four frequencies. Secondary analyses were carried out under the alternative definition where impairment was considered present when the mean PTA threshold across the four frequencies was ≥ 30 dB and considered absent when the mean threshold was < 30 dB. The primary definition is the stricter of the two.

Separate sets of analyses relating the screening test result and the reference standard classification were also conducted based on the inclusion of different subsets of impaired ears (or children). In the primary analysis all HI ears (or children) were included regardless of whether nominally recruited as a case child or as a control child. In two further sets of secondary analyses we included only impaired ears (or children) for those nominally recruited as cases and then included only impaired ears (or children) for those nominally recruited as controls.

Cross-tabulation was used to report the test results (refer vs. pass) for the PTS and HC screen in relation to each other, separately for the groups of ears (or children) that were classified on the PTA reference standard as impaired (to compare sensitivity between tests), hearing (to compare specificity between tests) and missing. Sensitivity, specificity, positive likelihood ratio and negative likelihood ratio were reported for the PTS and HC screening tests for ear-level and child-level analyses and all scenarios defined by the above definitions of impairment. Sensitivity is the percentage of impaired ears (or children) that are referred by the test; specificity is the percentage of hearing ears (or children) that pass the test; positive likelihood ratio is the ratio of the percentages that are referred by the screening test between the impaired and hearing groups; and negative likelihood ratio is the ratio of the percentages that pass the screening test between the impaired and hearing groups. For the ear-level analyses, correlation between the results for ears of a given child was accounted for when reporting CIs for sensitivity and specificity by fitting null logistic regression models (i.e. with no covariates) to the binary test result outcome with information sandwich (‘robust’) estimates of standard error, with the resulting log odds and 95% CI converted to the proportions (percentage) scale. When estimating sensitivity the model was fitted using only impaired ears and when estimating specificity using only hearing ears. The test result outcome was coded 1 for refer and 0 for pass when estimating sensitivity and vice versa when estimating specificity.

We report the absolute difference in percentages between the PTS and HC screen for both sensitivity and specificity with 95% CIs and McNemar’s test p-value [using the mcc command in Stata software (version 13.1, StataCorp LP, College Station, TX, USA)]. The calculation of the CI and p-value recognise the pairing of the PTS and HC test results within ear (or within child). This was done for ear-level and child-level analyses and all scenarios defined by the above definitions of impairment. Unlike when estimating the accuracy of each test, when comparing accuracy between tests for the ear-level analysis there was no need to allow for the correlation between results related to ears from the same children because the estimated difference in sensitivity (and specificity) between the PTS and HC tests is based on information within ears. An alternative analysis in which the data structure was reshaped so that there were four rows per child (PTS result for the left ear, HC result for the left ear, PTS result for the right ear and HC result for the right ear) and logistic regression models fitted to the test result outcome (coded 1 for pass and 0 for refer) on the predictor test type (coded 1 for PTS and 0 for HC) to estimate the difference between test types in accuracy, taking account of the correlation of test results for ears that belong to the same child, provided identical CIs and p-values to those reported here.

Missing data

To be included in the main ear-level analysis the ear needed to have provided data on both the screening tests and provided hearing-level data on all four frequencies presented under the PTA reference standard. It was possible for a given child, therefore, to provide only one ear that was used in the ear-level analyses. To be included in the child-level analysis the child had to provide full data on the PTS and HC tests and the PTA reference standard for both ears. Analyses were carried out using Stata statistical software.

Participants

Nominal case children were recruited from 14 centres. The original eight centres in the East Midlands were: Nottingham, Leicester City, Leicester County, Sheffield, Derby, Lincoln, Mansfield and Chesterfield. Owing to recruitment difficulties a further six centres were added: Bradford, Rotherham, Huntingdon, Birmingham Children’s Hospital, Birmingham Heartlands Hospital and Doncaster. From 379 invitations sent by the audiologists, a total of 86 replies were received, a response rate of 23%. Eight children were not eligible, being outside the required age range. We were unable to contact one of the initial respondents, and we were unable to see a further two children due to researcher illness just before the close of recruitment. We recruited and tested the remaining 75 children. See the flow diagram (Figure 6) below. Details of recruitment by centre are given in Table 8.

FIGURE 6.

Numbers of case children in the diagnostic accuracy study.

Nominal control children were recruited from 51 of the 164 schools in the Nottingham area that were invited by post to take part (response rate of 31% at school level). Twenty-three of the participating schools agreed to take part only after a follow-up telephone call to discuss the information received.

The 51 schools, between them, gave information packs to the parents of 2787 children, of whom 291 (10.4%; median reply rate per school of 9.1%, IQR 4.6–14.9%, range 0–28.6%) replied, confirming they would like to participate. Seven of the schools that agreed to take part did not recruit any children. An additional 11 siblings of children who attended the appointment but who did not receive the invitation were in the correct age range and parents agreed for them to take part, bringing the number of children initially indicating that they wanted to participate to 302. Eight of these children were subsequently found to be ineligible for the study (one was too old, six already had hearing problems identified and one replied after recruitment closed), 11 changed their minds about taking part and we were unable to see 43 either because we could not make an appointment (mostly not contactable) or because they did not attend the arranged appointment.

The remaining 240 children were recruited as nominal controls and seen for study appointments (Figure 7).

FIGURE 7.

Numbers of control children in the diagnostic accuracy study.

Demographic characteristics

Table 9 summarises the demographic characteristics of children who attended appointments (315) by whether they were nominally recruited as cases (75 children) or controls (240 children). The groups were similar with respect to gender and age. Four-fifths of each group was categorised as white on ethnicity. The nominal case group had a higher percentage of Asians (15% vs. 4%) and a lower percentage of mixed ethnicity children (1% vs. 9%) than the nominal control group.

Time interval between screening test and reference standard for children nominally recruited as cases

Table 10 summarises the time interval between when the screening tests were administered and when the PTA reference standard was measured for the nominal cases, reporting the difference in weeks. Separate rows summarise the time interval for the 65 nominal cases for which reference standard data were available before the tests were administered, and for the 10 nominal cases who completed the screening tests before the reference standard. The absolute time interval is summarised for all 75 nominal cases. The median absolute time interval was 16 weeks. The reference standard was completed within the criterion period of 12 months (52 weeks) before to 3 months (13 weeks) after the screening tests for 73 children. For each of the remaining two children a reference standard measure was available prior to the 52-week cut-off (65 and 63 weeks before the screening test) and another was available for each at 24 weeks after the screening test. The two measures for each child showed very similar results and indicated a stable hearing impairment by both definitions and the children were therefore included in the analyses. As described in the methods section the children nominally recruited as controls completed the tests and the reference standard on the same day.

Missing data and indeterminate results

Of the 630 recruited ears, 600 (95.2%) provided full data on the PTS and HC tests and scores for all four frequencies of the PTA reference standard and were included in the main analyses. Sixty-two (82.7%) of the 75 children recruited as cases and 233 (97.1%) of the 240 children recruited as controls (total = 295/315, 93.7%), provided full data on the tests and reference standard. There were no indeterminate screening test or PTA results. For children recruited as cases it was usually the PTA that was part missing; however, one child refused to do the HC test, another child refused to do the PTS test and one child had no physical ear on one side. For those recruited as controls, PTA and PTS results were missing for two participants due to equipment failure. There were a further two children with autism who were unable to complete the tests and the remainder were children who refused to do or complete other tests mainly because of lack of concentration or finding the headphones too tight.

Concordance between nominal case–control status and pure-tone audiometry reference standard classification

A unique feature of the two-gate (‘case–control’) design used for this accuracy study is that the criteria used to recruit children as cases and controls were not the same as the reference standard classification (using the PTA) that was used to define presence of the target condition when evaluating the screening tests. In other words the nominal target condition status is not necessarily consistent with the status based on application of the reference standard. For children nominally recruited as cases, hearing status was based on audiology notes. The criterion for inclusion was a bilateral impairment up to an average of 60 dB HL or a unilateral impairment of any level. Consequently, although nominally recruited as case or control children, some of the case children had no hearing impairment when assessed on the PTA reference standard and some of the control children were categorised as having impairment in the child-level analyses. Furthermore, and of particular relevance to the fact that the primary analyses are conducted at the level of the ear, there were case children for whom the PTA reference standard defined them as having the target condition in one ear but not the other (i.e. had unilateral impairment). There were, therefore, ears that were categorised and analysed based on the reference standard as HI that belonged to children nominally recruited as controls and ears that were categorised and analysed as NHI that belonged to children nominally recruited as cases.

Figure 8 indicates the number of ears belonging to participants nominally recruited as cases and controls that under the reference standard PTA were classified as HI, classified as NHI or for which there were insufficient data to classify the ear (i.e. PTA data were missing for that ear). The figure is based on the PTA reference standard definition of impairment as having a HL of ≥ 30 dB on at least one of the four frequencies (the primary definition). Twenty-six of the 150 ears of children nominally recruited as cases were classified on the reference standard as hearing and 48 of the 480 ears of children nominally recruited as controls were classified as impaired. Similar data are summarised at the level of child in Figure 9. Two of the 75 children who were nominally recruited as cases were classified on the reference standard as hearing. The pure-tone audiograms provided by the child’s local audiology service did not match the inclusion criteria but the children were tested within the study because, as per the protocol, the research team did not access the PTA until after the screening tests were undertaken. Of the 240 children who were nominally recruited as controls 37 were classified as impaired. We suggest that parents who had concerns about their child’s hearing were more likely to take up the invitation to take part.

FIGURE 8.

Nominal recruitment status at the ear level and the reference standard classification by or of hearing impairment status based on a PTA score of ≥ 30 dB on at least one of the four frequencies.

FIGURE 9.

Nominal recruitment status at the child level and the reference standard classification by or of hearing impairment status based on a PTA score of ≥ 30 dB on at least one of the four frequencies for at least one ear.

Accuracy of the pure-tone screen and HearCheck screener for identifying hearing impairment at the level of the ear

Figures 10 and 11 present flow charts that describe the number of impaired ears (based on a PTA score of ≥ 30 dB on at least one of the four frequencies) and hearing ears that passed and referred on the PTS and HC tests, respectively. Table 11 summarises the relationship between the PTS test results and the HC test results separately for impaired ears, hearing ears and ears for which information on the reference standard was missing. The figures highlighted in green indicate the numbers that were used in the calculation of sensitivity and specificity. The relationship between the PTS and HC results is summarised for the other five scenarios (see Tables 47–51, Appendix 4).

FIGURE 10.

Pure-tone screen results at the ear level by hearing impairment status based on a PTA score of ≥ 30 dB on at least one of the four frequencies.

FIGURE 11.

HearCheck screener test results at the ear level by hearing impairment status based on a PTA score of ≥ 30 dB on at least one of the four frequencies.

Table 12 reports the sensitivity and specificity of the screening tests for the ear level. The sensitivity was 94.2% for the PTS and 89.0% for the HC screen. The 95% CIs for sensitivity indicate that we can be fairly certain that the true sensitivity is no lower than 89% for the PTS but could be as low as 83% for the HC screen. The McNemar’s test result (p = 0.02) indicates evidence that the true sensitivity is greater for the PTS than for the HC screener. The difference of 5% in sensitivity implies that for every 20 impaired ears tested the PTS would correctly identify an extra ear as having a hearing problem compared with using the HC screener. The corresponding values for specificity were 82.2% for the PTS and 86.5% for the HC screener, with evidence provided by McNemar’s test that the true specificity is higher for the HC screener than the PTS (p = 0.02).

Table 13 reports the sensitivity and specificity for each combination of definition of impairment (PTA score of ≥ 30 dB on at least one frequency vs. a mean PTA score of ≥ 30 dB across all four frequencies) and subset of impaired ears used to calculate sensitivity (all ears, ears of children nominally recruited as cases and ears of children nominally recruited as controls). The results for the primary analysis for which the reference standard is a PTA score of ≥ 30 dB on at least one frequency and all ears are used to calculate sensitivity is shown on the top row. The sensitivity is generally higher (especially for the HC screener) when impairment is defined based on average HL across the four frequencies. This might be expected, as the primary definition of impairment is more stringent and thus more likely to result in less severe impaired ears being included in the impaired group. For the same reason the specificity is lower for both tests when impairment status is based on average HL across the frequencies presented under the PTA. Restricting impaired ears in the analysis to only those belonging to children recruited as cases results in increased sensitivity relative to inclusion of all ears. Again, this would be expected as ears of such children would be expected to have more severe hearing loss. Restricting impaired ears in the analysis to only those belonging to children recruited as controls results in lower sensitivity. Although only a sensitivity analysis, this latter result is notable because the impaired ears of children nominally recruited as controls may be more representative of the spectrum of impairment in the type of child that the screen would predominantly want to identify in a school-based setting (i.e. children with less severe hearing impairment).

Accuracy of the pure-tone screen and HearCheck screener for identifying hearing impairment at the level of the child

Figures 12 and 13 contain flow charts that describe the number of hearing-impaired children (defined based on a PTA score of ≥ 30 dB on at least one of the four frequencies in either ear) and hearing children who passed and referred on the PTS and HC tests, respectively. Table 14 summarises the relationship between the PTS test results and the HC test results for hearing-impaired children, hearing children and children who did not provide full data on the reference standard. The numbers highlighted in green were used to calculate sensitivity and specificity. The relationship between the PTS and HC results is summarised for the other five scenarios (see Tables 52–56, Appendix 4). Table 15 reports the sensitivity and specificity of the screening tests at child level. The sensitivity estimates (95.9% for the PTS and 88.7% for the HC screener) were similar to those reported for the ear-level analyses. The McNemar’s test (p = 0.02) indicates that the true sensitivity is greater for the PTS than for the HC screener. The difference of 7% in sensitivity implies that for every 14 impaired children tested the PTS would correctly identify an extra child as having a hearing problem compared with using the HC screener. Specificity at the child level was higher for the HC screener than for the PTS (83.8% vs. 79.8%) but there was little evidence of a true difference (p = 0.18).

FIGURE 12.

Pure-tone screen test results at the child level by hearing impairment status based on a PTA score of ≥ 30 dB on at least one of the four frequencies in at least one ear.

FIGURE 13.

HearCheck test results at the child level by hearing impairment status based on a PTA score of ≥ 30 dB on at least one of the four frequencies in at least one ear.

Table 16 reports the sensitivity and specificity for each combination of definition of impairment (PTA score of ≥ 30 dB on at least one frequency vs. an average PTA score of ≥ 30 dB across all four frequencies) and subset of hearing-impaired children used to calculate sensitivity (all children, children nominally recruited as cases and children nominally recruited as controls). The results for the primary analysis in which the reference standard is a PTA score of ≥ 30 dB on at least one frequency for at least one ear and all impaired children are used to calculate sensitivity are shown on the top row. The pattern of results is similar to the ear-level analyses. The use of less stringent passing criteria for the reference standard (average PTA score of ≥ 30 dB across the four frequencies presented) generally results in higher sensitivity and markedly lower specificity. Restricting the sample to impaired children who were nominally recruited as cases resulted in higher sensitivity and restricting the sample to impaired children who were nominally recruited as controls resulted in lower sensitivity.

Strengths and limitations

The study has a number of strengths. It was carefully designed and conducted, and data analysis was rigorous and included a number of secondary (sensitivity) analyses across different definitions of impairment. A sample size calculation was conducted a priori and the recruited sample yielded narrower CIs (and thus less uncertainty) for the accuracy estimates than the estimates from the economic model in the 2007 report. 12

A weakness of the study is that it estimated and compared the diagnostic accuracy between the PTS and HC screener methods using a two-gate (‘case–control’) design. 32 Under this approach those with the target condition (HI) and those without (NHI) are sampled from separate sources. The impaired children were largely identified via audiology services and the hearing children were largely recruited through schools. The two-gate design contrasts with the single-gate (or cohort) design where both those with and without the target condition are sampled from a single source without any knowledge of their target condition status. The more traditional single-gate design would have been preferable, but the challenge with this approach is that, because the prevalence of hearing impairment is so low, a large number of children would need to be recruited to the study in order to ensure that there is a sufficient number with the target condition. Fewer than 1 in 2000 children screened at school entry will have sensorineural hearing impairment8 and 160,000 school children would need to be approached and tested in order to identify the target of 80 cases originally proposed in this study. The anticipated size of the single-gate design test accuracy study challenges both its feasibility and value for money in this context. This is not an issue for the two-gate case–control design used here. The two-gate design, however, is known to be more prone to biased estimates of accuracy largely resulting from the fact that those with and without the target condition are sampled from separate sources rather than drawn from a single defined population. This is problematic when estimating the accuracy of each test but is a less salient issue when comparing accuracy between hearing screening tests since any biases operating will be experienced equally for each test assessed. With respect to differential accuracy, conclusions on whether one hearing screen is better than the other should remain robust.

The criteria used to recruit children in each gate of the design were not used as the reference standard to categorise the children according to impairment status. A consequence of this is that some of the participants who were nominally recruited as controls were categorised as having a hearing impairment on the PTA reference standard. Cases were recruited as having a bilateral or unilateral hearing impairment and hence some ears were categorised as hearing on the PTA reference standard. In this event, this was a strength of the study as the HI group included children with established hearing impairment as well as children with no previously identified hearing impairment. The latter group may have levels of impairment severity that are more representative of the type of children who one might seek to identify with hearing impairment using a school entry hearing screen. The recruitment of such children enabled us to carry out sensitivity analyses based on whether impaired children were known cases (and therefore had a permanent hearing impairment) or were not previously identified with a problem (and therefore probably included children with transient hearing impairment).

A higher-than-anticipated number of children nominally recruited as controls were shown to have a hearing impairment on PTA measurement. We suggest this may be caused by a bias towards participation in those families who had some concerns about their child’s hearing and therefore saw the study as a means by which their child could be tested, even though Nottingham operates an open referral system offering an appointment without a referral from a GP.

For the children nominally recruited as controls (which made up most of the hearing group) the screening tests and the reference standard were administered on the same day. The children nominally recruited as cases (mostly children in the HI group), however, generally had the reference standard administered at a different time to the test; up to 62 weeks before the test and up to 24 weeks after the test. The HL of children with permanent hearing impairment, however, is likely to be stable or worsening and it is not possible for children who have permanent sensorineural hearing impairment at one point to not have that hearing impairment later on.

Results in comparison with other studies

Several studies assessing the accuracy of the PTS were identified in the 2007 HTA report12 (see Table 6) and no accuracy studies of the PTS were identified in the update review. The sensitivity from these earlier studies ranged from 82% to 100% and specificity from 65% to 99%. Thus, the accuracy obtained in this study, sensitivity 94.2% (95% CI 89.0% to 97.0%) and specificity 82.2% (95% CI 77.7% to 86.0%) is consistent with the previous findings, and shows no evidence of the overestimation of accuracy, which is theoretically predicted by using a case–control methodology. This provides reassurance that the accuracy estimate obtained for the HC screener, particularly the relative accuracy of the HC screener against the PTS, is also robust, as relative accuracy is less likely to be vulnerable to biases associated with study design. This is helpful as an additional accuracy study was identified for a HC device relative to reference standard of PTA in the update review. Gloria-Cruz et al. 17 estimated sensitivity as 23% (95% CI 11% to 39%) and specificity as 97% (95% CI 96% to 98%). The study was conducted in the Philippines in a convenience sample of Grade 1 elementary school children from three metropolitan schools with a mean age of 7.6 years. The prevalence of hearing loss was approximately 5% (39/821). The circumstances are thus very different from our consideration of SES in the context of the UK, which is likely to explain the marked difference in accuracy between the estimates. Additionally, the devices were not identical in each study. The device used by Gloria-Cruz et al. 17 (HC navigator) used three frequencies rather than two and a higher threshold to define hearing impairment, which would decrease sensitivity and increase specificity.

As well as impacting on the validity of our accuracy estimates, comparison with other studies also impacted on our decision-making about the choice of accuracy estimates to be used in the base case for the health economic evaluation. In the current protocol the 2007 HTA study estimates were identified as the source of estimates for our base-case analysis. This decision seems to be justified both in terms of the internal validity of the estimates and their applicability relative to alternative sources of accuracy data identified.

Literature review

No studies were identified that reported false-negative data for the school entry hearing screen. Four of the studies reported comparisons of two different screening tests without reference standards,44–47 two of the studies (identified from the search on negligence) reported no cases owing to missed diagnosis of hearing impairment,48,49 and three included general discussion of the value of hearing screens beyond the neonatal period. 8,50,51 These nine studies were not reviewed further.

Geal-Dor et al. 52 reviewed the results of neonatal screening at birth and behavioural screening at 7–9 months for 1545 children in Israel. They identified 58 children who had passed both screening tests or had passed the only screening test they underwent who were later referred for audiological assessment between the ages of 1 month and 4 years. Thirty-four had age-appropriate thresholds. Of these, 18 were discharged and 16 belonged to high-risk groups and were followed up. All of the remaining 24 children had a conductive impairment. No cases of permanent hearing impairment were identified.

In a further sample of 49 children identified with permanent hearing impairment, the results of the two screening tests were reviewed. Although eight children passed one screen and were referred by the other, only one child passed both screening tests, a conservative false-negative rate of 2% (1/49).

In an 8-year follow-up of the cohort of 21,279 babies enrolled in the Wessex (UK) trial of neonatal screening,3 two of the 31 children identified by the age of 8–10 years with a permanent bilateral hearing impairment ≥ 40 dB had passed the neonatal screen (false-negative rate 6%). Of the 28,172 children who did not undergo neonatal screening, 35 children were identified with a permanent bilateral hearing impairment ≥ 40 dB and six of these had passed the distraction screening test at 7–9 months (false-negative rate 17%). The authors note that seven children (who had failed early screens) had evidence of progression of hearing impairment and speculate that the eight children who passed the screens might also have a progressive impairment, implying that the negative results may not have been false negatives.

Lo et al. 53 compared a parental questionnaire with PTA results in 6- to 7-year-old Chinese children in Hong Kong. They report that parents were able to identify only 20% of the cases of OME (no permanent hearing impairments were identified).

A birth cohort of 49,335 children in Australia with no risk factors for hearing impairment at birth underwent the distraction test at 7–9 months and were followed up at age 6 years. 7 Of 45,078 children who passed the distraction test eight were later identified to have a bilateral hearing impairment > 40 dB HL and fitted with hearing aids. The authors report a false-negative rate of 0.02% (8/45,078). If we assume those who failed the distraction test had a hearing impairment, the true false-negative rate would be 0.19% (8/4257 + 8); however, this assumption cannot be validated.

A study of college students (> 18 years) in the USA43 noted that the false-negative rate was higher when the reference standard included all frequencies rather than the frequencies at which the screening was performed. However, they also note that the major reason for this was the presence of hearing impairment at high frequency (6 kHz) in these young adults, possibly caused by leisure noise. This is an important issue and is particularly relevant for this project when evaluating the HC screener (1 and 3 kHz) against gold standard PTA at 0.5, 1, 2 and 4 kHz.

Soares et al. 26 compared the results of a new AABR screener against PTA in children and young adults aged from 3 to 22 years in Japan. For the pre-school children aged 3–5 years the false-negative rate was zero but increased to 3% (3/108) for ages 6–17 years and 12% (2/17) for 18- to 22-year-olds.

Diagnostic accuracy study (see Chapter 3)

For control children in the diagnostic accuracy study, if a screening test (either PTS or HC) was passed but the PTA result indicated a hearing impairment, the result of the screening test is defined as a false-negative result. Sixteen ears of 16 children met this definition. Each ear underwent two screening tests, so the 16 ears underwent 32 screening tests. Of these, 22 tests were passed (false negatives) and 10 tests gave a refer results (true positives).

Of the 22 false-negative test results, eight passed only the HC, two passed only the PTS and six passed both screening tests (Figure 14).

FIGURE 14.

False negatives from diagnostic accuracy study (PTA average criterion: average of four frequencies is ≥ 30 dB. PTA at least one criterion: at least one of four frequencies is ≥ 30 dB).

Of the six ears that passed both screening tests, all had a PTA score of ≥ 30 dB on at least one of the four frequencies, 0.5, 1, 2, or 4 kHz, and three of those had a PTA score of ≥ 30 dB averaged across the four frequencies.

Of the eight ears that passed the HC but referred on the PTS, all had a PTA score of ≥ 30 dB on at least one of the four frequencies, 0.5, 1, 2, or 4 kHz, and two of those had a PTA score of ≥ 30 dB averaged across the four frequencies.

Of the two ears that passed the PTS but referred on the HC, both had a PTA score of ≥ 30 dB on at least one of the four frequencies, 0.5, 1, 2, or 4 kHz, and one of those had a PTA score of ≥ 30 dB averaged across the four frequencies.

All children for whom there was a refer result on the PTA (regardless of screening test results) were offered referral for further assessment. Of the 16 children with false-negative screening test results on one ear, one child was not referred because the result was considered to be because of a lack of concentration and one did not attend the appointment. Of the remaining 14, 10 were found to have no hearing impairment and discharged, presumably because a transient hearing impairment had resolved. Three children who passed the HC test but referred on the PTS test were found to have mild impairments and the one child who passed both screening tests was found to have a mild impairment.

Background to study sites

Site with a school screen (Nottingham)

The CHAC is a third-tier service and is part of Nottingham Audiology Services within Nottingham University Hospitals NHS Trust. There is no separate second-tier audiology service for children within Nottingham, therefore, all children referred are seen within the same service. CHAC provides services for all children up to the age of 16 years (19 years if in special education) within the catchment area of Nottingham City, Rushcliffe, Erewash, Ashfield, Broxtowe and Gedling. It also accepts referrals from surrounding areas for specialised services. The service is led by a band 8a Clinical Scientist (CB) and has six whole-time equivalent (WTE) audiological staff ranging from band 6 to 8a. CHAC has an open referral policy and therefore accepts referrals from parents as well as from all professionals. It works closely alongside the ENT Department at Nottingham University Hospitals and children move between the two services depending on their needs. The service has around 3400 new referrals a year with a total number of 5900 patient appointments on average.

The service assesses children’s hearing according to national and local protocols and provides counselling and advice to parents with any concerns regarding their child’s hearing. If a child is identified as having a transient conductive impairment, they are referred onto ENT as appropriate after monitoring. For those children identified as having a permanent impairment and those children with a transient conductive impairment whose parent(s) choose not to have surgical management, the service provides hearing aids and ongoing habilitation (Figure 15).

FIGURE 15.

Pathway for Nottingham patients. CHI, conductive hearing impairment.

All schools within the Nottingham City and County authorities carry out the school entry hearing screen as part of the school entry health screen.

Site with no school screen (Cambridge)

The Cambridge service is a second-tier service provided by Cambridge Community Paediatric Audiology Service, which is part of the Cambridgeshire Community Services (CCS) NHS Trust. It receives approximately 1800 new referrals per year with a total of approximately 2400 appointments annually. The service provides second-tier hearing assessment for children aged 7 months to 16 years (19 years if in special education) and information and education to carers and professionals. The catchment area covers children living in, attending school in or with a GP in Cambridge City, or South and East Cambridgeshire. The service is provided by two community paediatricians (total 1.3 WTE, one of which is clinical lead for the service), one WTE band 7 lead clinical scientist (Audiology) (JM), members from the Addenbrooke’s Paediatric Audiological Team who provide sessional coverage, and administrative support provided by members of the Clinical Support Team attached to Children’s Services within CCS.

The service is provided within family-friendly environments located in the community. A community paediatrician and an audiologist usually run each clinic by working together, using their own areas of expertise, to look at the whole child.

Referrals are made to the service from GPs, child and family nurse team/health visitors, speech and language therapists, paediatricians, education, social services, and the Newborn Hearing Screening Programme (NHSP). When a chronic or permanent hearing impairment is identified the service aim is to facilitate further assessment and management within third-tier services, that is, the tertiary audiology service at Addenbrooke’s Hospital, the ENT department at Addenbrooke’s Hospital, and speech and language therapy.

The role of the service is to assess children’s hearing, according to national and local protocols, for children referred due to carer or professional concern. It also includes assessment of children who have been identified to be ‘at risk’ by the NHSP despite clear responses on the screen. The main condition seen in the second tier is transient conductive hearing impairment associated with middle ear effusion or ‘glue ear’. The second tier provides carers and professionals with information and education regarding this type of hearing impairment and good strategies for supporting the child. If the middle ear effusion or conductive hearing impairment is persistent there are agreed written protocols for referring these children to the ENT Department at Addenbrooke’s Hospital. If a child is identified as having a sensorineural (unilateral or bilateral) hearing impairment, the child will be referred to the Tertiary Paediatric Audiology Service at Addenbrooke’s Hospital for diagnostic assessment and management; this includes any child where it has not been possible to exclude a sensorineural hearing impairment. Referrals to the second tier are often made in order to exclude a hearing impairment as a contributing factor to educational, behavioural and/or speech and language concerns. This includes children who may go on to be identified as being on the autistic spectrum. A diagrammatic representation of the pathway of care for Cambridge is shown in Figure 16.

FIGURE 16.

Pathway for Cambridge patients. CPAS, Community Paediatric Audiology Service; CHI, conductive hearing impairment; CUHFT, Cambridge University Hospitals Foundation Trust (third-tier hospital).

There has been no SES in Cambridge City, South and East Cambridgeshire since 1997, when the health visitor distraction testing was abolished in this area. Reasons for stopping included cost, variability of practice and lack of strong guidance from the Department of Health on what should be provided for SES and how it should be implemented. Community Paediatric Audiology in CCS was set up in 1997 with a strong campaign to first-tier services (GP, speech and language therapists, etc.) informing them that there would be no routine hearing tests in Cambridgeshire and emphasising the importance of referring to second-tier services if there are parental and professional concerns. This involved letters to all GP practices and health visiting teams, and speech therapy services outlining the changes in provision of routine hearing tests, and posters in GP practices and speech therapy clinics, highlighting the presenting symptoms of glue ear.

Data collection

Data were collected for children aged between 3 years and 6 years 364 days who were referred to Nottingham paediatric audiology or Cambridge audiology services by any source other than the UNHS. All referrals between 1 September 2012 and 30 June 2014 were included. Data on follow-up appointments that took place up to 30 September 2014 were included.

We originally proposed to explore retrospective data collected in Cambridge from 2007 to 2012. However, as those data were not collected with the objectives of the research in mind, much information was missing and it was decided that analyses would not add constructively to the project.

The collaborating audiologists for the areas of Nottingham (site with SES) and Cambridge City, and South and East Cambridgeshire (site without SES) collected data on referrals. Further data were collected via questionnaire from parents of children referred from the SES programme to the Nottingham service (see Chapter 6). Prospective data were processed using a database built by PenCTU.

Audiologists entered data from patient notes. In Nottingham, the waiting list co-ordinator identified eligible children, and audiologists within the service with permission to access patient records entered the data. In Cambridge assistance was also provided by one of the Nottingham researchers. Data were entered for the complete care pathway, from referral through to discharge (or 30 September 2014, whichever occurred first) including: date of birth; postcode; date of referral; location of clinic; referral source (GP, health visitor, school screen, etc.) date of appointment(s); type of assessment (e.g. visual reinforcement audiometry, play audiometry in the soundfield, PTA); result of assessment [normal bilateral, normal soundfield (better ear) thresholds, unilateral sensorineural, bilateral sensorineural, unilateral conductive, bilateral conductive, mixed bilateral, incomplete]; tympanometry results; hearing thresholds/minimal response levels; probable cause of impairment (if known); end of care (yes/no); and outcome [discharge, referral to ENT, referred for diagnostic confirmation (Cambridge only), hearing aids fitted (Nottingham only)].

Each child’s records were accessed by audiologists authorised to look at them, hence consent was not required from individual patients. Anonymised data were entered onto the database. These procedures received ethical approval. Referral data were checked and corrected for implausible values. Staff time to undertake a planned second data extraction check of 10% was severely restricted by the pressure of service delivery. Undertaking this data check was not possible with the staff resources available without causing a significant delay to the study reporting. Copies of the questionnaires were sent to PenCTU for second data entry and double entry data checking.

Statistical analysis

The yield, rate of referral and age at referral were compared between the site with a SES programme (Nottingham) and the site without a SES programme (Cambridge).

Yield was defined as the number of children between their third and seventh birthdays identified as having hearing impairment whose date of referral was between 1 September 2012 and 30 June 2014 (the study period) per 1000 person-years at risk. Hearing impairment included transient conductive and permanent sensorineural or conductive hearing impairments. The rate of referral was defined as number of referrals for suspected hearing impairment in the same period per 1000 person-years at risk. Some referrals resulted in more than one appointment. Children for whom the outcome of the last appointment was further referral or hearing aid were considered to have hearing impairment and included in the numerator in the calculations for yield; children discharged at the last appointment were considered to have no hearing impairment. In order to calculate the denominator for yield and the referral rate, the population size in each site was obtained from the Office for National Statistics mid-2013 estimates54 of the population who were aged 3, 4, 5 or 6 years in the study sites. The Nottingham site included referrals from the local authorities of Nottingham, Erewash, Ashfield, Broxtowe, Gedling and Rushcliffe; the Cambridge site included Cambridge, East Cambridgeshire and South Cambridgeshire. The number of person-years observed was calculated by multiplying the population size by the number of days during the study period (668 days) and dividing by the mean number of days in a year (365.25 days). The two sites were compared with respect to yield and referral rate using the rate ratio, reported with 95% CI and p-value.

The t-test was used to compare the mean age at referral between the Nottingham and Cambridge sites (1) for all initial referrals; and (2) for confirmed HI cases only.

We report the percentage of referrals that resulted in the identification of HI cases for: (1) Nottingham referrals that were via a school screen; (2) Nottingham referrals that were via any other source (e.g. GP, speech therapist); and (3) Cambridge referrals.

Finally, we report the median (IQR) level of hearing impairment in dB at each of four frequencies (0.5, 1, 2 and 4 kHz) in each ear and the source of referral (using numbers and percentages) for both Nottingham and Cambridge. These variables were summarised for all referrals and then for the subset of referrals that resulted in the identification of children with impaired hearing.

Analyses were carried out using Stata statistical software.

Referral rate and yield

There were 1702 referrals in Nottingham (21.9 referrals per 1000 person-years) and 1108 in Cambridge (34.4 referrals per 1000 person-years); the referral rate in Nottingham was two-thirds that of Cambridge (rate ratio 0.64, 95% CI 0.59 to 0.69; p < 0.001) (Table 18). Hearing impairment was confirmed in 195 children in Nottingham (yield of 2.51 cases per 1000 person-years) and 98 children in Cambridge (3.04 cases per 1000 person-years). There was little evidence that the yield is different between Nottingham and Cambridge (rate ratio 0.82, 95% CI 0.64 to 1.06; p = 0.12). Confirmed hearing loss cases made up 17.0% of referred children in Nottingham (25.2% of children who were referred via SES and 14.9% of children referred via other sources) and 10.6% of referred children in Cambridge (Table 19).

The mean age of referral was 4.7 years for both the Nottingham and Cambridge sites, but the mean age at referral for children who were subsequently confirmed as HI was higher in Nottingham than Cambridge (5.0 years vs. 4.5 years; mean difference 0.47 years, 95% CI 0.24 to 0.70 years; p < 0.001) (Table 20).

The characteristics are summarised in Table 21 for all referred children and in Table 22 for children confirmed as HI cases, separately for each of the Nottingham and Cambridge sites. In Nottingham 21.5% of all referrals and 30.8% of the confirmed HI cases were originally referred via SES. Other key sources of referral in Nottingham were ENT consultants (23.6% of confirmed cases), parents (11.8% of confirmed cases), GPs (10.8% of confirmed cases) and health visitors (10.8% of confirmed cases). In the Cambridge site the key sources of referral for confirmed HI cases were GPs (64.3%), health visitors (21.4%) and speech therapist (12.2%).

Strengths and limitations

Our study had a number of strengths. Data collection in both sites was comprehensive and actively monitored by a senior member of the clinical staff with responsibility for audiological services in each of the two areas. Both were members of the research team. An electronic database used in both sites was developed by staff of PenCTU to standardise data collection.

However, our study design of an observational comparison of two areas [one that operates a SES programme (Nottingham) and the other that does not operate a SES programme (Cambridge)] is subject to major methodological limitations, in spite of our best attempts to choose two sites that were similar to each other. We acknowledge that there may be epidemiological and social differences between the two geographical areas that are likely to confound our findings. Reassuringly, population estimates indicate the proportion of children aged < 16 years to be similar in Nottinghamshire (18.7%) and in Cambridgeshire (18.5%). 54 However, the index of socioeconomic deprivation indicates the city of Nottingham (rank 17) to be more deprived on a range of measures than the Cambridge district (rank 188). 55 Given the lack of availability of child-level data, we were unable to adjust our analyses to take account of these and other potential confounders. Furthermore, given that we consider only two geographical areas, our results may not be considered as generalisable.

Both sites mainly receive referrals from a defined geographical region but also accept referrals outwith that area. Equally, some children within the defined area may be referred elsewhere; the numbers are estimated to be few by the responsible audiologists. The referral catchment areas also do not exactly match the areas defined by the Office for National Statistics for the population estimates used and there may, therefore, be some minor imprecision in the population denominators used in the analyses.

Study population

Questionnaires (see Appendix 5) were distributed to parents of children referred to Nottingham CHAC from the school entry hearing screen between 1 September 2012 and 30 June 2014 (age range 4–6 years).

Questionnaire

The questionnaire captured data on aspects of the family’s experience of the child being referred for a hearing assessment. Questions included how they found out about the screen and further testing, parent opinion about having a school hearing screen, the time taken to attend appointments, the cost of travel to appointments, and impact on work, school and social activities of attending clinic appointments. Data on gender, ethnicity and school attended were also requested. The level of anxiety experienced by the parent on finding out that their child needed further testing and again when they attended the appointment was rated by the parent on a scale from 0 (not at all anxious) to 10 (extremely anxious). There was also the possibility of an option to follow-up responses by telephone, with those who gave their contact details. No follow-up was carried out as it was not considered that it would add anything to the data already collected on the questionnaire, especially taking into account the bias of recall over a long time period and the small data set.

Data collection

Questionnaires plus pre-paid return envelopes were sent by the audiologist to the parents of all children referred to the CHAC by the school screen, once the end of care (discharge, hearing aid provision or referral to ENT) had been reached, whether or not they had a hearing impairment. The audiologist kept a list of questionnaires sent and which participant ID these related to. Parents who chose to take part returned the completed questionnaire to the researchers at NHBRU. These returns could be anonymous but questionnaire responders could also choose to be entered into a prize draw to win vouchers of their choice to a value of £50, or to agree to a possible follow-up interview and hence supply contact details.

Nottingham researchers entered the data into the study database and copies of questionnaires (excluding the contact details page) were sent to PenCTU for second data entry and validation. A tracking log of the ID numbers of questionnaires returned was kept locally by researchers at NHBRU. A reminder and repeat questionnaire was sent after 3 months to non-responders identified from the list held by the audiologist.

Consent for the data collected in this study was implied by return of a completed questionnaire. It was explained to the parents completing the questionnaire that entry into the project was entirely voluntary and that the management and care of their child would not be affected by their decision on whether or not to take part.

Data analysis

The quantitative data were summarised using means with SDs or medians with IQRs.

To give structure to the qualitative data obtained from the questionnaires, a thematic analysis was carried out. Open comments from parents were assessed using a template analysis in which identified themes from each of the questionnaires were selected and ordered. Responses to the following questions were analysed using the template: (1) What are the good things about your child having their hearing checked at school?; (2) What are the not so good things about your child having their hearing checked at school?; and (3) Any further comments?

A five-step framework for analysing the questionnaire responses was used: familiarisation; identification of themes; indexing; charting; and interpretation. 56

1. Familiarisation: each questionnaire was read through six times to get a feel for the data.

2. Identification of themes: the broad research question was to explore what parents thought about the SES tests. Using an inductive approach, parents’ opinions were categorised into main themes that emerged across all the questionnaires. The researchers made notes on relevant points raised under each category. This process was repeated for all questionnaires and common themes were included in the final template. Ambiguous statements were not included as they were open to interpretation.

3. Indexing: subthemes based on similarities between all responses within the main identified categories were explored. Indexing involved generating more precise descriptions of the themes to make the analysis reader-friendly.

4. Charting: a template was drawn up for each category, theme and descriptor.

5. Interpretation: illustrative quotes were identified to provide the reader with examples of the parents’ experiences.

Quantitative data

Of the 60 children for whom questionnaires were returned, 32 (53%) were female, 44 (73%) were white, 3 (5%) were mixed ethnicity, 10 (17%) were Asian, 1 (2%) was black/African/Caribbean and 2 (3%) were of other ethnic group.

Forty-five (75%) parents knew that their child was having their hearing checked at school. Parents found out that their child needed further testing by means of a letter either taken home (n = 25) or sent in the post (n = 9); by telephone (n = 13) or by other means (n = 13) [already noticed at home (n = 5), told by school (n = 4), told by school nurse (n = 2), told by doctor (n = 1), do not know (n = 1)].

Parental anxiety was scored on a 0–10 scale where 0 indicated ‘not anxious at all’ and 10 indicated ‘extremely anxious’. The mean level of anxiety reported by the parent(s) on finding out that their child needed further testing was 5.3 (SD 2.1), and the median was 5 (IQR 4–7). At the clinic appointment the mean parental anxiety had reduced to 4.7 (SD 2.4); the median was 5 (IQR 3–6). Frequencies of anxiety are given in Table 23.

The same anxiety level on both occasions was reported by 36 out of 60 respondents (60%); 18 out of 60 (30%) reported feeling less anxious at the appointment and 6 out of 60 (10%) reported more anxiety at the appointment.

Most parents [51 out of 60 (85%)], strongly agreed that ‘children should have their hearing checked at school’, six (10%) agreed, one (2%) had no opinion, no one (0%) disagreed and one (2%) strongly disagreed (one missing).

Of the 60 questionnaires returned, parents reported that 25 children had only one appointment, 16 children had two appointments in total, nine had three appointments, five had four appointments, and five had five appointments. Altogether the 60 children had 129 appointments.

Time taken to get to an appointment was < 15 minutes for 24.8% (32/129) of appointments, about half an hour for 59.7% (n = 77) of appointments, about 1 hour for 14.0% (n = 18) of appointments and 1–2 hours for 1.6% (n = 2) of appointments. The length of appointment was < 30 minutes for 33.3% (43/129) of appointments, 30–60 minutes for 44.2% (n = 57) of appointments, about an hour for 13.2% (n = 17) of appointments, and 1–2 hours for 7.0% (n = 9) of appointments. Data on length of appointment were missing for three appointments.

For the journey to an appointment, 20.9% (27/129) included a bus or tram ride, 72.9% (n = 94) of journeys used a car, 5.4% (n = 7) included a taxi ride and one walked (0.8%). The median (IQR) total cost per appointment was £4.88 (£3.40–£8.80).

Parents were required to take either all or part of a day off work for 45.0% (58/129) of appointments; 29.5% (n = 38) of appointments were with parents who did not work and 25.6% (n = 33) of appointments did not require the parent to take time off work. The child was required to miss school for all or part of the day for 72.9% (n = 94) of appointments.

Twelve out of the 60 (20%) children missed other activities, including ballet and music lessons but eight of these referred to missing school. Thirteen (21.7%) of the 60 parents missed other activities, although eight referred to work – probably a misunderstanding of the questions. Eight of 60 (13%) families said appointments caused problems for other family members by requiring additional childcare for other children.

Qualitative data

Table 24 lists the themes that emerged from the data and provides illustrative quotes.

Main findings

The consequences for costs and the views of parents concerning referral for diagnostic evaluation were assessed by means of a questionnaire sent to parents of children referred to the Nottingham audiological service by the SES programme. The response rate was 24.4% (60/246). It is possible that those who did not respond are neutral, that is, they had no strong opinion with respect to the subjects in the questionnaire, but this cannot be assumed. If an adverse experience engendered non-response, then the low response rate could conceal an unsuspected level of ‘harm’ associated with SES.

Of the respondents, the majority had more than one appointment. Some of those with multiple appointments may be individuals in whom hearing impairment was confirmed (true positives), but others will be those in whom no hearing impairment was identified (false positives).

The mean parental anxiety score on finding out about the referral appointment was 5.3 on a 0–10 scale, with 0 indicating ‘not at all anxious’ and 10 ‘extremely anxious’. The mean score was 4.7 at the clinic appointment. Severe anxiety, indicated by scores of 10, was registered by only two (3%) parents. Thus, although there was some anxiety associated with referral in those who responded, it appeared moderate, albeit sustained during the waiting period.

Costs were associated with attending the referral appointment in the respondents in terms of time, travel and activities foregone. The median travel cost per appointment was £4.88, travel times to get to the appointment was about half an hour in 60% of appointments, and appointment times were 30–60 minutes in 44% of appointments. Activities foregone were generally work in the case of parents (45% of appointments involved missing work) and school in the case of children (73% of appointments involved missing school).

Despite the inconvenience, virtually all respondents were supportive of SES; 85% of parents strongly agreeing that ‘children should have their hearing checked at school’. Supportive statements and overall satisfaction were also evident in the open comments. These identified areas where care needs to be maintained in delivering SES, particularly attention to communication, thereby minimising concerns about the absence of parents at the initial test but also the potential for noise and other distractions in schools to reduce the value of hearing tests.

Strengths and weaknesses

The study was prospectively designed and carried out in accordance with a protocol without deviation. The most important challenge to the validity of the findings was the low response rate, with only 24% of parents to whom questionnaires were sent out providing responses. This was despite a small incentive for taking part and a reminder being offered. This may have been in part because of the length of time between the child being referred and receiving the questionnaire which was sent out after the child reached the end of care. However, for the specific purpose that this component of the clinical studies was designed, we do not believe this is a major shortcoming, but the conclusions drawn should be viewed in the light of a low and potentially biased response rate.

Children referred from SES may truly have a hearing impairment (true positives), may not have a hearing impairment, or have a hearing impairment at the time of screening which later resolves (false positives). We failed to identify information in the literature on what the consequences of false positives might be in a SES programme (see Chapter 4). As far as the original 2007 HTA report12 was concerned, the potential for disutility associated with false positives was not considered, almost certainly because there was no evidence to inform any assumptions. In this respect it would be useful to gauge whether there were likely to be substantial consequences associated with false positives, indicating that they might have a major impact on overall effectiveness and cost-effectiveness. However, as parents were given the option to return the questionnaires anonymously we were not able to collect data on outcomes for all those who responded. In Chapter 5 we report that there were 366 referrals from the SES to the Nottingham service, 60 (16.4%) of whom were confirmed cases. Nearly 90% of confirmed cases had a conductive impairment. We can assume, therefore, that the majority of the questionnaires were returned by the parents of children who did not have a permanent impairment (false positives).

There are clearly some adverse consequences of referral, but these appear to be small, judged externally or by the effect on parental satisfaction. Furthermore, it is reasonable to assume that the effects in non-respondents are unlikely to be greater than those observed in the respondents, so we evaluate that the adverse consequences of referral, particularly for those found not to have hearing impairment, are likely to be no greater on average than those observed in the respondents.

Results in the context of other studies

We emphasise that this is the first attempt to quantify the effect of the referral process in SES on parents and children. Inevitably, there is still much uncertainty, but we would argue that the uncertainty has been reduced. How the results of this study were considered in the economic model of this report is discussed in more detail in Chapter 8.

We explored the option to formally quantify the impact on health-related quality of life, particularly for children, through the Health Utilities Index (HUI) instrument but this was not pursued, both because the instrument is not responsive to specific issues related to hearing and on the grounds of cost of using the instrument under licence, a factor amplified by the need to repeat the assessment. Given that the response rate emerged as being problematic in the actual study, it is likely that an attempt to use a more formal instrument like the HUI may have further compromised the response rate.

Sample size

It was anticipated that a sample size of four schools would provide a convenience sample of about 180 children. The study size was not formally calculated.

Analysis

The mean and median time taken to complete the screening tests are presented for: (1) all children, (2) children for whom the PTS was administered first and (3) children for whom the HC screen was administered first. The mean time was compared between the PTS and the HC screener using linear regression models. As the distribution of time to complete the test was skewed, bias-corrected accelerated bootstrap CIs were constructed for the mean difference between the PTS and HC tests.

Time taken

The mean/median times taken for each screening test were similar for the two tests, at around 1.4 minutes per test, but the range of test times was wider for the PTS (to be expected as the test was not automated). The test time did not appear to vary with the order of the tests. The CI for the mean difference across all screens indicates that the PTS is unlikely to be more than 5 seconds quicker and unlikely to be more than 9 seconds longer on average to administer than the HC screen. That the CI includes zero indicates that it is plausible that there is no difference between the PTS and HC screener in mean time taken (Table 25).

Observations of researchers and nurses

The researchers observed that schools were often unsuitable for testing hearing because they were too noisy and a suitable alternative room was often not available. On some occasions the nurse had to give up with the hearing tests and return on a different day.

The nurses suggested advantages and disadvantages of each test (Table 26)

The three nurses scored each of the tests on a total of 20 occasions for 185 children (no tests at one session because no suitable room available; hearing testing abandoned at another session, after three children tested, owing to noise). All nurses scored all tests as 5 or above for ease of use, accuracy and future use. The mean, median and range of scores are shown in Table 27. The PTS test scores higher than the HC test on all measures but in terms of ease of use that might have been because the nurses were more familiar with using the PTS. They also rated accuracy more highly for the PTS than the HC screener. When asked how much they would want to use a test in the future they again rated the PTS higher, but commented that they could see the HC screener as being useful as a back-up in some situations.

For two of the nurses the scores all either decreased or stayed the same over time. For the third nurse the HC screener scores stayed the same and the PTS scores increased for ease of use and future use and decreased for accuracy over time.

Strengths and weaknesses

The study was prospective and undertaken in accordance with a protocol without deviation.

The target number of participants was achieved. The observations were carried out in familiar environments for the children allowing assessment of the operation of the screening tests without being compromised by the children feeling anxious – an issue that may have hindered the testing of control children at NHBRU in the diagnostic accuracy study.

Testing took place in a number of different schools throughout the school year, enabling the capture of seasonal changes that might affect hearing through colds, illness or hay fever.

The observations captured data on a range of children from different backgrounds. Several city schools were observed, enabling examination of how the hearing screens by school nurses might be affected by school size, behavioural challenges, support from teaching staff and school facilities, such as test room conditions.

We note the limitation that feedback on the two screening tests and the observations of testing involved only three nurses. However, all nurses undergo the same training and follow a set protocol while screening and therefore procedural differences should not affect test performance. With regard to verbal feedback provided by the nurses on the two tests, all three demonstrated inter-rater reliability, finding similar strengths and weaknesses for both the PTS and HC tests. Gaining opinion from a wider nursing community is unlikely to affect the conclusions drawn.

Results in the context of other studies

As far as we are aware this is the first study to systematically examine the practical issues associated with applying tests that might be used in SES with children in real-life circumstances.

Overall approach

Our approach was designed to estimate the cost-effectiveness of SES primarily from a health-care perspective but to consider other costs where data were available. In this report this was limited to consideration of transport costs associated with families attending diagnostic evaluation; thus the perspective adopted in this report was that of the NHS and the family. The study was designed to capture the costs and benefits of the two different methods of SES in order to inform the policy decision regarding the appropriate use of SES. Since SES is likely to have an impact on the costs and outcomes of all types of hearing impairment, the analysis is concerned with the identification of children with sensorineural or permanent conductive (SNPC) hearing impairment or transitory hearing impairment not diagnosed during the newborn hearing screen or the first 3 years of life (the final pre-school year which is the starting point for the economic model).

There are three key considerations that inform whether or not screening is cost-effective relative to no screening. First, there is the issue of whether or not screening at school entry improves the timeliness with which children are referred to diagnostic evaluation and, if indicated, to management, thus generating a more prolonged improvement in quality of life than would otherwise occur. Second, cost-effectiveness will depend on whether or not the diagnostic accuracy of the test makes a difference in the time at which children are identified, referred and managed. If a test has a high level of false negatives, children who should have been picked up by the screening test will experience a delay in their diagnosis and management. Third, to the extent that there are false positives (either from screening or other modes of identification), there is a potential associated negative impact on the child and family of stress and lost time in school or work, as well as additional unnecessary costs to the health system.

The following sections outline the key components of our modelling approach. This includes the rationale and important assumptions required to estimate the cost-effectiveness of the PTS and HC tests and provides some background to the evolution of the modelling approach since the 2007 HTA report. 12

Decision-analytic structure

The screening question was addressed using a decision-analytical approach to evaluate the cost-effectiveness of school entry hearing screening programmes. The model estimates the costs and consequences of a hypothetical cohort of 10,000 children with a given prevalence of hearing impairment receiving the PTS, HC testing or no screening. The basic structure is presented in Figure 18. Figure 19 shows the route followed from screening to diagnosis of hearing impairment while Figures 20 and 21 illustrate the management follow-up for transitory and SNPC hearing impairment, respectively. As the decision tree representation does not explicitly capture the time element in the model, this aspect is detailed in the following sections.

FIGURE 18.

Basic decision tree structure.

FIGURE 19.

Decision tree: screening arm.

FIGURE 20.

Transitory hearing impairment.

FIGURE 21.

Sensorineural or permanent conductive hearing impairment.

Initial model development

Although a version of the original model from the 2007 HTA report12 [built in TreeAge Pro 2005 (TreeAge Software Inc., Williamstown, MA, USA)] was provided for the current research, the updated model was developed in Microsoft Excel to provide flexibility in the modelling approach. Part of the initial work on this project was to simplify the decision tree model developed for the 2007 HTA report. 12 Based on consultations with the Project Steering Group, a flexible modelling approach was adopted which would retain the logic of the original model and could be represented by a decision tree format, but also capture the way in which hearing impairment is identified over a period of time. This is relevant not only when SES is unavailable and hearing impairment is identified exclusively by other means, but also in the presence of screening, when these other routes of identification may also be important. This chapter describes the initial development of the model in anticipation of the results of the clinical studies described elsewhere in this report (see Chapters 3, 5–7) and subsequent to the data becoming available, at which point we were able to generate final estimates of cost-effectiveness for the new technology (HC screener) and the existing technology (PTS) compared with no screening. For the two screening tests, the structure of the evaluation is identical.

Based on information in the 2007 HTA report12 and correspondence with the lead modeller (Professor Linda Davies), we reconstructed the model used to generate the cost-effectiveness results presented in that report. Before updating the model, as specified in the original protocol, the modelling approach was reviewed to ensure that it was still fit for purpose. The alternative of a more detailed simulation model was considered but it was felt that it would add minimal benefit in terms of answering the research question. It was concluded that a Markov model or discrete-event simulation would not greatly improve the representation of screening while adding to the complexity of the modelling by increasing its data requirements (e.g. the use of transition probabilities). At the same time, this would have presented a considerable challenge in terms of data availability and would have introduced additional parameter uncertainty. As part of the validation of the initial model we developed, we were able to reproduce, within reasonable limits, the original base-case results from the 2007 HTA report.

This initial model was populated with the parameter values used in the original modelling exercise conducted for the 2007 HTA report12 and was driven by the diagnostic accuracy of the screening tests. For the screening arm of the model, this generated a number of referrals for suspected hearing impairment in the first year (true positives and false positives). As Figure 19 indicates, children who are referred by the screening test (positive result) are sent for a DEA for a definitive assessment (DEA being assumed to be 100% accurate). It was then assumed that all remaining cases of hearing impairment (false-negative results of the screen) would be identified over the following 2 years (i.e. up to age 7) by referral for a DEA as a result of concern by parents, teachers or other professionals. An assumption was required about the timing of these additional cases. In the absence of other evidence, they were evenly spread over the subsequent 2 years of the model. Similarly, an assumption was required about the rate of identification in the no screening arm. Under no screening, it is assumed that those with a suspected hearing impairment (on the basis of concern by parents, teachers or other professionals) are referred for a DEA. The background rate of identification was set so as to generate an even flow of diagnosed cases over a 3-year period.

Those referred by the screen or referred for a DEA on the basis of the concerns of parents, health professionals, teachers and others (in the presence or absence of screening) will either be found not to have a hearing impairment (false positives) or will be confirmed as having transitory or SNPC hearing impairment in either one or both ears (true positives). Management strategies received by children with hearing impairment include active observation, non-surgical and surgical interventions, with the type of intervention offered varying depending on the severity of hearing impairment.

Model development incorporating new clinical data

Although the basic model structure illustrated in Figure 18 has remained essentially unchanged in the current version of the model, there have been substantial revisions compared with the model described above to take account of the clinical observations reported in previous chapters. The present model has been informed by the two-gate (‘case–control’) study investigating the diagnostic accuracy of the two screening methods (see Chapter 3), a comparison of a site with a SES programme (Nottingham) and a site without a SES programme (Cambridge) (see Chapter 5), a study exploring the impact of referral for diagnostic evaluation on parents and children (see Chapter 6) and data on practical implementation in schools (see Chapter 7).

For both transitory and SNPC hearing impairment, the model takes account of the benefits of management for those found to have mild, moderate or severe hearing impairment. The analysis runs for a 4-year time period, starting in the year before school entry. By the end of the 4-year time period, evidence on referrals from the studies undertaken in Nottingham and Cambridge indicates that all cases of hearing impairment are likely to have been diagnosed regardless of whether children at age 4 years are screened or not screened. In the absence of screening, identification of hearing problems occurs as a result of the concerns of parents, teachers or other professionals, sources which can also generate referrals when a screening programme is in place. As discussed in the following sections, the results of the model are primarily driven by the total numbers of referrals with and without screening, and the numbers referred for a DEA in each year.

Given the key influence of referrals on the results of the analysis, the analysis has explored the effect of varying the number and pattern of referrals over time. This analysis has been reported as a threshold analysis following the presentation of the base-case results. A probabilistic sensitivity analysis (PSA) was not carried out, as it was not considered that probability distributions could usefully be attributed to the numbers of referrals or their timing based on referrals data drawn from two areas. It is argued here that PSA may not always be the most informative technique for exploring parameter uncertainty, as it can result in attention being focused on those variables for which probability distributions are most easily attributed rather than those that have the greatest influence on the results. In this case, while it is recognised that there will be variability in the numbers and rate of referrals with and without hearing screening, this uncertainty is particularly difficult to quantify. Threshold analysis was therefore performed on these variables.

Total referrals and cases of hearing impairment

Table 28 presents the key items of data obtained from the clinical studies that were used in obtaining estimates of the prevalence of hearing impairment, numbers of referrals and results of the screening tests.

The total number of cases of hearing impairment to be identified (with or without screening) is based on the prevalence of hearing impairment using the number of confirmed cases of hearing impairment in Nottingham (n = 195) as a proportion of the base population (N = 42,553). This gives a prevalence of approximately 46 cases in a population of 10,000. These cases were divided into SNPC and transitory cases of hearing impairment in accordance with the assumption used in the 2007 HTA report. 12 In the base case, the number of referrals has been assumed to be the same in both screening and non-screening arms of the model and is given by the number of children referred in Cambridge (n = 1108) as a proportion of the base population (N = 17,624), or around 6.3%. This is equivalent to 629 referrals in a population of 10,000. Using the rate of referrals observed in Nottingham does not have a material impact on the cost-effectiveness results. The estimates of test sensitivity are the child-level estimates reported in the diagnostic accuracy study. The implications of these data in terms of screening results (true positive, false positives, true negatives and false negatives) are presented in Table 29.

The three groups of most interest are the true positives, false positives and false negatives since no further intervention is required in the true-negative group. True positives and false negatives are generated by the observed diagnostic sensitivity of the two tests, given the prevalence of hearing impairment. Total referrals are made up of true positives, false positives and false negatives, the last of these groups being referred by means other than the screening test so that all cases of hearing impairment are ultimately identified.

Owing to the constraints imposed by the numbers of referrals and prevalence of hearing impairment, the resulting number of false positives implies a substantially higher test specificity (the probability of a child without hearing impairment testing negative) than that found in the new diagnostic accuracy data from Nottingham (Table 30). Moreover, these constraints mean that the implied specificity is the same under either screening method in the base case and cannot be varied between the PTS and HC screener.

However, it was possible to explore the impact of varying the numbers of false-positive referrals between the screening and no screening options by varying the total number of referrals. This is a particularly relevant area of uncertainty to consider as, in the base case, total referrals are assumed to be the same with and without screening, whereas evidence from the service comparison study suggests that the rate of referrals may be higher in the absence of screening than when a screening programme is in place.

Although the significantly higher referral rate in Cambridge than Nottingham (34.4 vs. 21.9 per 1000 children per year) does not lead to a significantly higher yield of confirmed cases in Cambridge (3.04 vs. 2.51 per 1000 children per year), the possibility that a screening programme will reduce the number of referrals needs to be considered. Given the differences between the characteristics of the populations in the two areas, it is unclear what the difference in referral rates might be in any given area in the presence or absence of screening. Nevertheless, a higher rate of referrals in the no screening arm could result in a substantial increase in costs relative to the screening option (given the unit cost of diagnostic evaluation relative to screening) and thus have an important influence on the cost-effectiveness of screening. We therefore vary the referral rate between screening and no screening arms of the model in sensitivity analysis.

Distribution of referrals over time

In the current model, the previous assumptions about the identification of hearing impairment over time have been superseded by data on referrals obtained from the screening area (Nottingham) and the no screening area (Cambridge). Table 31 gives the distribution of referrals by age at last birthday in Cambridge and Nottingham. Referrals at age 3 years are taken to apply to the pre-school entry year (year 1 of the model) while referrals at ages 4–6 years are taken to apply to the first 3 years of school (years 2–4 of the model). In addition to the distribution over time, the model also takes account of the different sources of referral in the presence and absence of screening, based on the available data (Table 32). It is worth noting the importance in the screening area of sources of referral other than screening.

Management of hearing impairment

Once cases of hearing impairment are confirmed by a DEA, their management will depend on the type of hearing impairment and its severity. The available management options are essentially unchanged from the 2007 HTA report. 12 We simply note here that management of hearing impairment results in an improvement in quality of life, which translates into QALYs. While the total QALY difference between screening and no screening will include the QALYs experienced by those with no hearing impairment, we exclude these from our results, as we assume that the characteristics of the modelled populations are the same in respects other than their exposure to type of screening (or no screening). The reported QALYs in screening and no screening groups represent the QALYs accruing only to those with managed hearing impairment who thereby experience an improvement in quality of life. The assumptions underlying the QALY calculations and the other parameters in the model are reported in the following sections.

Sources of parameter inputs

Values of input parameters for the model are based on the original HTA model, the literature, the observational studies described in greater detail elsewhere in this report (see Chapters 3, 5–7) and standard reference sources. A limited literature review was carried out to update health state utilities and costs. For probabilities of different severities of hearing impairment and utilisation of management options, the model used the data on which the original HTA report12 was based as the steering group felt that these were still relevant.

An overview of sources for the main broad categories of data input is given in Table 33.

Reviews of the literature

A short literature review was undertaken to search for any updates to the data required by the model that were not expected to be included in the results of the clinical studies. These were utility values of hearing impairment, the prevalence of hearing impairment (using data from the NHSP) and costs of management.

The scope for searches was publications since 2007, to capture new research published since the 2007 HTA report. 12 Search ranges were from January 2007 to July 2014 and were carried out on 10 July 2014.

The following databases were searched:

• MEDLINE

• EconLit

• SocINDEX

• PsycINFO.

Searches were also run through Google Scholar.

The following charity and support group websites were checked for useful publications (grey literature) that may not show up in searches of conventional databases:

• Action on Hearing Loss

• National Children’s Bureau

• National Deaf Children’s Society.

Referrals

The clinical studies have, to date, reported on samples of 1108 children referred to audiology services in the no screening area (Cambridge) and 1702 children referred in the screening area (Nottingham).

Number of children with hearing impairment

Determining the number of children who need to be identified is a critical assumption within the model and the majority of the analysis relies on this calculation. The original HTA report12 on which this analysis draws assumed that the prevalence of hearing impairment was 78 of 1000 children. This was split into approximately 3.5 of 1000 children with SNPC hearing impairment and 74.5 of 1000 children with transitory hearing impairment through the probabilities that populate the model. The SNPC number is supported by research used, and subsequently published, for the 2007 HTA report,12 which gives a figure of 3.65 of 1000 children with SNPC hearing impairment. 8,12

However, although this figure is likely to be an accurate representation of the total prevalence of SNPC hearing impairment in the child population, it is also likely to be a sizeable overestimate of the numbers of children who have SNPC hearing impairment yet to be diagnosed at the age of 3 years (prior to the start of the model). This is because cases of hearing impairment in children are generally identified either through the NHSP or by parents before school age. Moreover, while our main interest is in children with SNPC hearing impairment, the vast majority of the children who are detected as a result of screening will have transitory hearing impairment.

The prevalence of both transitory and SNPC hearing impairment in the modelled population has been estimated on the basis of the numbers with a confirmed diagnosis of hearing impairment as a proportion of the base population in the two areas studied. Based on last appointment, we obtain estimates of 46 per 10,000 children in Nottingham (195/42,553) or 56 per 10,000 children in Cambridge (98/17,624). The former has been used for the purposes of the model and divided between transitory and SNPC hearing impairment in the ratio 96% : 4% as applied in the 2007 HTA report. 12 This gives approximately 2 children per 10,000 with undiagnosed SNPC hearing impairment and 44 with undiagnosed transitory hearing impairment.

Diagnostic accuracy

The sensitivity of the two screening tests used in the model was based on the child-level analysis of the case–control study using all children whether nominally recruited as cases (HI) or controls (NHI), where hearing impairment was defined as a PTA score of ≥ 30 dB on at least one of the four frequencies. Estimates of diagnostic accuracy have been reported (see Table 30).

Both screening methods generate a small number of false-negative cases, and, in these instances, the children should be picked up via other referral methods such as through GPs or speech therapists. Differences in sensitivity between the two screening methods will give rise to different numbers of false negatives which have been distributed over time in the model in the manner illustrated in Table 34. The distribution draws on the distribution of total referrals but adjusts for false negatives of the screening tests occurring only in year 2 onwards (rather than all 4 years of the model).

Probabilities of hearing impairment by severity

Table 35 reports the proportions of cases of transitory and SNPC hearing impairment that are unilateral or bilateral and the distributions of unilateral and bilateral SNPC hearing impairment by severity. The source of these probabilities is the 2007 HTA report. 12

Management probabilities

Probabilities of each management type for unilateral and bilateral transitory hearing impairment and the different severities of unilateral and bilateral SNPC hearing impairment were sourced from the 2007 HTA report12 and are listed in Table 36.

Resource use and costs

The following sections report the sources underlying the calculation of costs over the 4-year modelling period. The cost base year is 2012–13. Costs in the first year are undiscounted and, in subsequent years, are discounted at an annual rate of 3.5%. This is the rate recommended by the UK Treasury Green Book,57 based on the rate at which individuals discount future consumption over present consumption.

Screening costs

All children in the screening arm incur the costs of screening. Children who are referred by the test also incur the cost of a DEA and the subsequent cost of management is incurred by children who are referred by the screening test and are diagnosed with hearing impairment by a DEA.

The length of time to perform the test was collected (see Chapter 7). A mean duration of 1.4 minutes (measured to be the same for both methods) and an hourly rate of £73 per hour from the relevant 2012–13 Reference Costs60 (N05OGS – School-Based Children’s Health Other Services – Group Single Professional) give a staff cost associated with the screening tests of £1.69. This is reported in Table 37 while unit costs for other sources by which hearing impairment can be identified are presented in Table 38. The cost per child of screening was calculated by dividing the total costs (total costs of screening tests and total capital costs) by the total cohort of children screened over 5 years (10,000). The cost of each diagnostic evaluation was £150 (NHS Reference Costs 2012–1360).

Cost of travelling to appointments

Sixty respondents to a questionnaire asking about travel costs to attend screening appointments in Nottingham reported a total of 129 appointments and expenditure of £788.64, or £13.14 per child.

Unit costs of the pure-tone screen and HearCheck screener

Unit costs of each method of screening are presented in Table 37. Where costs extend beyond 1 year, these have been discounted accordingly in the model. Equipment suppliers have provided the costs of devices and consumables. Although the device costs for the PTS are higher than those for the HC screener (£898.80 vs. £139.20), the total costs for the HC screener are higher, primarily owing to the costs of ear cups. Table 38 reports the unit costs associated with other means by which hearing impairment is identified.

Management costs

Management costs were compiled from NHS Reference Costs for 2012–13,60 representing national unit costs, and are reported in Table 39. Where cost estimates were available for unilateral impairment only, these were multiplied by two to give the corresponding costs associated with bilateral impairment. This is a conservative assumption to account for the costs of providing and maintaining hearing aids for two ears. In the case of surgery, expert opinion suggests that this approach may overestimate costs. No follow-up costs or postoperation observation (active observation) were taken into account with surgical interventions. As with NHS reference costs generally, figures relating to paediatric services or the under 18 years age group were used where available. However, for hearing aids, the reference costs are not broken down for children and adults separately.

Utilities

A search to update the utilities from the 2007 HTA model12 was undertaken and the results were used to inform the updated parameter sheet (Table 40). Where possible, a distinction was made between unilateral and bilateral hearing impairment; otherwise, the utility associated with hearing impairment was determined primarily by its severity. In the first year (the pre-screening year), we assume that children entering the model do so evenly over the course of the year. Total utilities in the first year are therefore reduced by 50% (in undiscounted terms) compared with subsequent years. QALYs have been calculated only for children with a managed hearing impairment who consequently achieve a quality-of-life improvement. As QALYs accruing to children without hearing impairment do not affect the incremental cost-effectiveness ratio (ICER), they have been excluded from the calculations. Limited updating of the parameter values used in the 2007 HTA report was possible. For example, utilities for cochlear implants came from Summerfield et al. ,62 and utilities for grommet surgery came from Bisonni et al. 63 These studies provided intervention-specific utility data to supplement the evidence on utility by severity of hearing impairment.

Modelling the potential costs and consequences of false-positive results

Chapter 6 presents the outcomes of a questionnaire given to parents whose children were referred from SES for a DEA appointment. Survey responses were obtained in respect of 60 children over 129 appointments.

The questionnaire asked a range of questions concerning parents’ views of the screening programme and the impact on themselves and their children. The travel costs associated with screening and diagnostic visits were based on this survey. One impact that is commonly discussed in the context of screening programmes, although not straightforward to value in monetary terms, is the anxiety associated with false-positive results. This was an issue addressed by the questionnaire.

Anxiety

Parents were asked to rate their anxiety level on a scale from 0 to 10 as a result of finding out that their child needed further testing. Approximately 60% of respondents listed their anxiety as ≤ 5 out of 10, with only one parent rating their anxiety > 8 out of 10. Between finding out that their child needed further testing and attending the clinic visit, mean parental anxiety score fell slightly, from a score of 5.3 to a score of 4.7. The survey was felt to provide insufficiently compelling evidence to make an adjustment to the model, either by incorporating anxiety into the QALY or as a monetary disbenefit. On the basis of current evidence, it is unknown whether or not there is a health impact on parents of further hearing tests that would outweigh the benefits gained by children receiving a correct diagnosis; this is an issue on which further research may shed some light.

Number of referrals to diagnostic evaluation and numbers diagnosed with hearing impairment

Tables 41 and 42 show the number of children referred for a DEA and the numbers diagnosed with hearing impairment in each model arm over the 4 years of the model. Under the base case, a hypothetical population of 10,000 children has been used, of which the number referred for diagnostic evaluation in the counterfactual and the two intervention arms is 629, and the number with hearing impairment is approximately 46. The numbers referred in each year are determined by the data on referrals at different ages in the screening and no screening sites and have an important bearing on the cost-effectiveness of screening compared with no screening.

School entry hearing screening versus no screening: costs and quality-adjusted life-years

Tables 43 and 44 present the undiscounted and discounted costs broken down into the costs of identification (whether by screening or other means), diagnostic evaluation and management over 4 years for each arm of the model.

Diagnosis and management of hearing impairment has a positive impact on children’s quality of life as measured using the QALY. The figure for QALYs gained outlined in Table 45 is an estimate of the QALYs gained from children being diagnosed and managed (children moving from hearing impairment to no hearing impairment/managed hearing impairment) over the 4 years of the model. The QALYs accruing to children with no hearing impairment are not included here. The results show that, in the base case, not having a screening programme results in more QALYs than either the PTS or HC screen, which is associated with the lowest QALY gain of the three options.

In order to determine the cost-effectiveness of each method of screening compared with no screening, the ICER needs to be calculated. The ICER presents the ratio of the marginal gain of the intervention over the counterfactual in terms of both costs and benefits. It is calculated as:

Table 46 presents incremental costs and QALYs for the two screening approaches relative to no screening and for the PTS compared with the HC. In the base case, it is not appropriate to report an ICER, as no screening dominates (is more effective and less costly than) either screening approach. In the context of our research question, the results indicate that SES is not cost-effective compared with no screening.

Sensitivity analysis

The key data from the clinical studies that influence the magnitude and direction of the cost-effectiveness results primarily relate to the number and timing of referrals with and without a SES programme. While there is uncertainty about the applicability of the findings from Nottingham and Cambridge to a more general assessment of the costs and benefits of a screening programme as opposed to no screening programme, it is difficult to put boundaries on this uncertainty. This is in part because of the sociodemographic differences between the populations in Cambridge (the counterfactual) and Nottingham (screening site), and to the differences in service configuration between the two sites. A factor of particular significance is the difference in the rates of referrals between the two areas (the question of whether a screening site has more or fewer referrals than a non-screening site was an explicit objective of the study). In the light of clinical data suggesting that the referral rate in Cambridge is higher than that in Nottingham, the impact of a higher referral rate in the absence of screening was felt worthy of exploration. Increasing the referral rate when no screening programme exists increases the costs of this option while leaving QALY benefits unchanged, as these depend on the timing rather than the number of referrals. Increasing referrals sufficiently will render no screening more costly than screening and enable the trade-off between costs and benefits to be investigated.

In order to determine whether an intervention is cost-effective, the National Institutes of Health and Care Excellence (NICE) recommends comparing the ICER with a benchmark value of between £20,000 and £30,000 per QALY gained. 58 Technologies with an ICER of < £20,000 are generally considered to be cost-effective while, for those with an ICER > £20,000, reference needs to be made to other factors when considering value for money (and the case needs to be made increasingly strongly in relation to these factors when the ICER is > £30,000). Using £30,000 per QALY as the cut-off point for cost-effectiveness, the referral rate in the absence of screening would need to increase by ≥ 36% for no screening to cease being cost-effective relative to the PTS. This gives an upper limit on the extent to which referrals can be increased in the absence of screening without this option becoming excessively costly relative to its QALY benefits.

We are also interested in the circumstances under which screening becomes more effective than no screening. Leaving the baseline level of referrals unchanged, we investigated the extent to which referrals would need to be brought forward with screening compared with no screening. As we lacked clear bounds to place around the proportions of children referred at different ages, we again conducted a threshold analysis. It was found that the benefits of the PTS test would be increased sufficiently for the ICER to fall below £30,000 per QALY gained compared with no screening if the proportion of children referred in the first year of school (the screening year) increased by 5.9 percentage points or more.

Compared with the distribution and total numbers of referrals, other variables had relatively little impact on the conclusions of the analysis. For example, raising the sensitivity of the screening tests to 100% increased the QALY benefits under the PTS and HC but not sufficiently to make screening more effective than no screening. Altering the prevalence of hearing impairment had no impact on the results.

Discussion of key assumptions

Several key modelling assumptions require further discussion. Good use of assumptions is crucial to economic modelling. Done correctly, assumptions simplify the modelling approach, allowing for targeted models to be built using the best quality of data. A model with large parameter demands often has to make compromises and assumptions that can end up weakening the model.

One of the main assumptions in the model concerns the rate of referral to diagnostic evaluation. For the base-case scenario, it has been assumed that the number of referrals in the screening and non-screening areas is the same. From the comparative data presented in Chapter 5, the rate of referral in Cambridge (area without SES) was higher than that of Nottingham (area with SES). However, after careful consideration of the issue, these comparative data were not used for the base case, as Cambridge may not be reflective of all non-screening areas. Assuming that the rates of referral would be higher when no screening programme is in place makes an implicit assumption that other methods of referral, such as referral via GP, health visitors and speech therapists have low specificity. That is, it assumes that these other methods require a higher number of referrals than are required with a screening programme to identify the same number of cases of hearing impairment when, in practice, it is not known what the true increase in rate of referrals for non-SES areas relative to SES areas would be.

In the base case, screening is more costly than no screening. However, if referrals are increased in the no screening option, there is a point when no screening becomes more costly than screening. If referrals are increased further, the no screening option will eventually become too costly to justify its additional QALY benefits over screening relative to conventional cost-effectiveness benchmarks such as the £30,000 figure used by NICE. Sensitivity analysis considered the ‘tipping point’ in terms of referral numbers under the no screening option at which no screening would no longer be cost-effective. It was found that the cost per QALY ratio of no screening relative to the PTS test increased to £30,000 if the rate of referrals was 36% higher in the absence of screening compared with the PTS. In this case, screening is the cost-effective (if less effective) option.

An alternative way in which screening could become cost-effective is if, despite being more costly than no screening (as in the base case), it was also more effective. This could come about if screening was associated with more rapid detection of cases of hearing impairment than no screening. A threshold analysis on the pattern of referrals suggested that, under screening, referrals would need to increase by around 5.9 percentage points in the screening year (increasing the proportion of referrals taking place either in the pre-screening year or the screening year from 59% to 64.9%) in order to reduce the cost per QALY of the PTS relative to no screening to £30,000. While there are grounds for believing that the number of referrals is likely to be higher without screening, the potential for screening to achieve timelier referral and management of HI children is less clear.

Strengths and weaknesses

The model developed has a number of features that support the validity of its results.

It builds on an existing model (2007 HTA report12), which allowed a systematic consideration of areas where the original model could be improved and incorporates these into the updated model. This was greatly assisted through the involvement of the architect of the original model who contributed to the project as a consultant (Professor Linda Davies).

The model was conducted by an experienced multidisciplinary team of researchers who had been involved in the development of economic models, and models concerning hearing impairment in particular, prior to this project.

The model also drew on the experience of the project steering group, both in terms of content knowledge to advise on the design of the model and the input of a health economics specialist who fed back on the model design and early results.

The model was underpinned by a protocol outlining the key features and limiting the opportunity for results to be data driven. In the event, changes were made to the structure of the model so that it could capture aspects of the impact of SES that were not originally anticipated. These changes have been fully documented relative to the original model plan, so reducing the opportunity for bias.

The model was conducted in parallel with a series of clinical studies that were designed to improve information on key parameters where high levels of uncertainty had been identified in the 2007 HTA report model (see Chapters 3, 5–7). 12

There were also some limitations. The greatest limitation was that, despite attempts to reduce uncertainty, the data collection in the accompanying clinical studies was unable to overcome this uncertainty completely. For instance, the study designs for accuracy were chosen on the basis of feasibility and used a diagnostic case–control study, which is known to exaggerate accuracy, particularly where the controls are healthy subjects. Similarly, a randomised comparison between SES and non-SES areas would have been desirable to assess the impact of SES on referrals and yield. Instead we had to employ an observational two-centre comparative study design subject to major potential limitations, including confounding and lack of generalisability. In retrospect we could have invested research in quantifying the accuracy of processes used to refer children for a DEA where SES was not in place, but the importance of this emerged only when the results of the clinical studies were reported.

The inevitable lack of availability of the results of the clinical studies until late in the research programme limited the time available to perform sensitivity analyses in the economic model. These were thus prioritised in consultation with the research and steering group and we remain confident that all key aspects have been covered and that the cost-effectiveness findings remain robust.

Findings in comparison with other health economic evaluations

The 2007 HTA report12 identified virtually no health economics literature on SES, and the update search for this report identified no new health economics literature since the 2007 report. Thus the main point of comparison for the new economic model is the economic model from the 2007 HTA report. 12

The most important difference between the two reports is a change in view about the likely cost-effectiveness of SES from possibly cost-effective (albeit with considerable uncertainty in 2007) to probably not cost-effective in this report. This change is primarily because of the use of observations on the timing of referrals as the basis for the updated model. The 1-year results from the 2007 report, showing a favourable cost-effectiveness ratio for screening, are consistent with a substantial advantage for screening in terms of the timeliness of referrals compared with no screening. This is also implied by the assumptions used in initial attempts to replicate the 2007 results. In comparison, the observational data incorporated into the model on which the findings presented here are based suggest that screening does not result in a more rapid rate of referral and that other methods used in the absence of screening may be more effective in this regard.

Systematic review of diagnostic accuracy (see Chapter 2)

The updated review of diagnostic accuracy studies confirms the conclusion from the 2007 HTA report12 that research to date demonstrates significant variability in the design, methodological quality, and results. Robust conclusions about the performance of individual test types for use in SES cannot be drawn. Summarising the review reported in the 2007 HTA report and this update we conclude that:

• Parental questionnaires had the poorest diagnostic accuracy compared with all other tests.

• The findings from the new audiometry-based studies evaluating computer-based devices and the HC screener reported higher and more consistent specificity but lower and widely varying sensitivity estimates compared with the sweep PTA studies included in the original report.

• Studies evaluating TEOAE reported variable sensitivity with wide CIs, while specificity estimates were relatively high and more consistent.

• The study evaluating AABR reported high sensitivity and specificity.

The review included studies from countries with and without an established UNHS system and with very different systems of health-care delivery. The generalisability of the findings to other situations, including the UK NHS system, is likely to be limited.

Diagnostic accuracy study (see Chapter 3)

The findings of our diagnostic accuracy study indicate that the PTS and HC devices have a high level of sensitivity (PTS ≥ 89%, HC ≥ 83%) and an acceptably high level of specificity (PTS ≥ 78%, HC ≥ 83%) for identifying hearing impairment at the level of the ear. The PTS test has greater sensitivity than the HC screener and the HC screener has greater specificity than the PTS.

These conclusions appear robust, with the child-level analyses indicating similar levels of sensitivity and specificity for the screening tests to those seen for the ear-level analyses and for all different definitions of impairment.

Assessment of false negatives (see Chapter 4)

Assessment of false-negative rates is challenging for screening evaluations of a condition that may fluctuate, progress or be of later onset. From our review of the existing literature and data from the diagnostic accuracy study, we are unable to quantify the effect of false-negative results from the PTS or HC screening, but were able to confirm that the rate was extremely low. Of the 16 ears of children in our diagnostic study (total N = 630) which passed one or both of the screening tests but were referred by the PTA measure, only four were confirmed to have a hearing impairment at diagnostic evaluation and all were mild (4/630, 0.63%).

Comparison of school entry hearing screening and non-school entry hearing screening services (see Chapter 5)

There was strong evidence that the rate of referral for hearing problems is lower when a SES programme is present. The referral rate was 36% lower in Nottingham (SES) relative to Cambridge (no SES) (rate ratio 0.64, 95% CI 0.59 to 0.69; p < 0.001).

There was little evidence that the yield of confirmed cases differs between areas with and without a SES programme but a higher proportion of referred children were subsequently confirmed to be HI in the area with a SES programme (17.0% in Nottingham vs. 10.6% in Cambridge).

The mean age of referral was nearly identical between the two sites when looking at all referrals but for children who were subsequently confirmed as having a hearing impairment there was strong evidence that children in sites with a screen are older at referral (mean age difference 0.47 years, 95% CI 0.24 to 0.70 years; p < 0.001).

It appears that the site with a SES programme is referring fewer children for suspected hearing problems than the site with no SES programme, but there was little evidence that the yield of children with confirmed hearing impairment was different. Children with confirmed hearing impairment are older at referral in the SES site.

Survey of parents (see Chapter 6)

We found from our survey of parents of children referred by the SES programme in Nottingham that the consequences of the referral process for parents and children, including false positives, are minor. They certainly do not appear to be sufficient to undermine parental views about the value of SES. However, it should be noted that the referral process, including the possibility of false positives, occurs both in a system with SES and one relying on ad-hoc referral based on concern alone and hence the data on anxiety and costs could apply to both service implementations. The difference for parents whose child is referred by the SES programme is that they may have had no concerns prior to the screening test.

Practical implementation (see Chapter 7)

We demonstrated minimal differences between the PTS and HC tests in terms of time taken to conduct each examination and practical issues. Testing covered a range of schools throughout the school year and thus we suggest the findings might be generalisable beyond the Nottingham schools.

Cost-effectiveness of school entry hearing screening (see Chapter 8)

Our economic modelling showed that SES is unlikely to be cost-effective and, using base-case assumptions, including the assumption that the number of referrals is the same in the presence or absence of screening, is dominated by a no screening strategy. This is consistent with the observed results of the clinical studies (see Chapter 5), which suggest that cases of hearing impairment are identified in similar numbers but at a younger average age in the absence of SES.

Two situations where SES might be cost-effective were identified. In the first situation, SES may be considered cost-effective if there are fewer referrals associated with SES or, conversely, if there are more referrals without screening. This assumption might be supported by the observation from our clinical study (see Chapter 5) that referral rates (and by assumption, potential false positives) were less in the site where SES had been in place for many years. However, in order for this to be the case the reduction in referrals would need to be attributable to SES and there is considerable uncertainty about this. The model is also sensitive to a second set of assumptions in which referrals occur more quickly with screening than is observed from our study comparing SES and non-SES sites.

Implications for practice

Although our finding that the lack of cost-effectiveness of SES may be considered as a reason to withdraw SES where it is currently being practised, we would highlight aspects of the results that suggest caution. First, we have not completely excluded the possibility that SES is cost-effective and we have shown that there are at least two scenarios in which it may be cost-effective. Second, our conclusions are highly dependent on findings in the two specific areas (Nottingham and Cambridge) that were used here, and may not be generalisable to other areas. Third, the cost-effectiveness of SES depends on how effective (or ineffective) the ‘no SES system’ is. This, in turn, is highly dependent on the effectiveness of ad-hoc identification and referral for a DEA, which is not only largely unknown, but likely to be variable. It seems plausible that SES might have greater potential to be cost-effective where ad-hoc identification and referral is less well developed than in a system where it is well established. If withdrawal of the SES service is to be considered it needs to be carefully managed to ensure that the ad-hoc referral system is working effectively. Health professionals, school and nursery staff, and parents who would then be responsible for referral of children about whom there were concerns in the school entry year might need to be reminded to be more vigilant for signs of hearing impairment.

Implications for research

There are opportunities for further research.

We have highlighted the continuing evolution of evidence on the accuracy of tests for screening for hearing impairment in children of school entry age, and think that it would be useful to do this in an ongoing manner, particularly for the general value of the information to health-care workers testing hearing in childhood. Cochrane now includes reviews of test accuracy, and so would be an ideal vehicle for such an ongoing systematic review. We thus suggest that even though it might not directly impinge on a decision about SES, systematic reviews of the accuracy of devices that might be used to measure hearing in children at around school entry age should continue to be pursued, as this performance will undoubtedly influence the performance of any ad-hoc referral system prompted by suspicion of hearing loss.

Our economic model has highlighted the need for careful consideration of the alternatives to SES. It is not appropriate to consider a comparison with SES where there is absolutely no screening/identification activity. Although less well described and potentially subject to variation, the alternative to SES in the UK is a system in which parents, or those involved in the care of children, raise concerns. Where the concerns are substantiated this is followed by referral from a health-care professional, such as a GP, for a DEA. This system will also be taking place where SES is provided, not just in the years before or after school entry, but also in the year of SES. This is an important general finding from this work. Success in identifying hearing impairment in children in the UK is heavily dependent on the effectiveness and cost-effectiveness of the ad-hoc system of referral by a wide range of agents (parents, education, social services and health care) for formal evaluation on suspicion of hearing impairment. This appears to have been very little explored. Thus, in the future, characterising and then measuring the cost-effectiveness of different approaches to the ad-hoc system, with a view to optimising it should receive as much, if not greater priority, than further evaluation of SES.

A related issue is the process by which concern, or referral from SES, is converted into DEAs. It is clear from this project that there is considerable additional triage as the actual number of referrals for a DEA in Nottingham (a service with a SES programme) is much less than the number of referrals that would be expected from the false-positive rates seen in the accuracy estimates (see Chapter 3). This may, in part, be because of the fact that, in the real-world implementation of the SES, a child who is referred by a screening test will be re-tested some weeks later before an onward referral for a DEA is made. In this respect, more research on what determines programme specificity (as opposed to test specificity) would be useful.

Understanding why the referral rate in Nottingham was less than in Cambridge (a service without a SES programme) and whether or not this is related to the presence of SES would be of specific interest. Whether or not the difference and cause of any differences is generalisable to other areas would be the wider objective of further research. Further observational studies similar to our comparison between Nottingham and Cambridge could be undertaken, albeit recognising the difficulty of matching the geographical areas.

Further research to better quantify the impact of referral, particularly with respect to anxiety, and whether or not all referrals are affected to the same degree as respondents in our study may be required, particularly if it appears that overall effectiveness and cost-effectiveness could be critically dependent on the costs and disutilities experienced by those testing false positive. We note also that our research has not explored children’s perspectives on testing for hearing impairment and so this could be an important target for further research.

Expanding the research on implementation to more sites could contribute further information on differences in provision and further opinion from nurses applying the tests within those different systems.

If withdrawal of SES is contemplated in particular settings, this could be used as an opportunity for further data collection. Particularly where the pattern of referrals and cases was known over many years in the run up to withdrawal, any change in pattern of referrals/cases could be very useful evidence confirming the lack of effectiveness and cost-effectiveness of SES, or challenging it. More formally if SES cessation is being contemplated in many areas, a randomised trial of withdrawal of SES programmes could be designed using referrals and hearing impairment cases identified as outcomes.

Objectives

This report aimed to determine answers to the following three questions:

• What is current practice for the SES in the UK?

• What is known about the accuracy of alternative screening tests and the effectiveness of interventions?

• What is known about costs, and what is the likely cost-effectiveness of the SES?

Methods

A national postal questionnaire survey was addressed to all leads for the SES in the UK, considering current practice in terms of implementation, protocols, target population and performance data. Primary data from cohort studies in one area of London were examined. A systematic review of alternative SES tests, test performance and impact on outcomes was carried out. Finally, a review of published studies on costs, plus economic modelling of current and alternative programmes was prepared.

Results

The evidence from the national survey of current practice is that:

• the SES is in place in most areas of England, Wales and Scotland; just over 10% of respondents have abandoned the screen; others are awaiting guidance in the light of the national implementation of newborn hearing screening

• coverage of the SES is variable, but is often over 90% for children in state schools

• referral rates are variable, with a median of about 8%

• the test used for the screen is the pure-tone sweep test but with wide variation in implementation, with differing frequencies, pass criteria and retest protocols; written examples of protocols were often poor and ambiguous

• there is no national approach to data collection, audit and quality assurance, and there are variable approaches at local level

• the screen is performed in less than ideal test conditions

• resources are often limited and this has an impact on the quality of the screen.

The evidence from the primary cohort studies is that:

• the prevalence of permanent childhood hearing impairment continues to increase through infancy

• of the 3.47 in 1000 children with a permanent hearing impairment at school screen age, 1.89 in 1000 required identification after the newborn screen

• the introduction of newborn hearing screening is likely to reduce significantly the yield of SES for permanent bilateral and unilateral hearing impairments; yield had fallen from about 1.11 in 1000 before newborn screening to about 0.34 in 1000 for cohorts that had had newborn screening, of which only 0.07 in 1000 were unilateral impairments

• just under 20% of permanent moderate or greater bilateral, mild bilateral and unilateral impairments, known to services as 6-year-olds or older, remained to be identified around the time of school entry.

The evidence from the systematic review of the alternative tests and of the effectiveness of interventions is that:

• no good-quality published comparative trials of alternative screens or tests for school entry hearing screening were identified

• studies concerned with the relative accuracy of alternative tests are difficult to compare and often flawed by differing referral criteria and case definitions; with full pure-tone audiometry as the reference test, the pure-tone sweep test appears to have high sensitivity and high specificity for minimal, mild and greater hearing impairments, better than alternative tests for which evidence was identified

• there is insufficient evidence to draw any conclusions about possible harm of the screen

• there were no published studies identified that examined the possible effects of SES on longer-term outcomes.

The evidence from the cost-effectiveness study is that:

• no good-quality published economic evaluations of SES were identified

• a universal SES based on pure-tone sweep tests was associated with higher costs and slightly higher quality-adjusted life-years (QALYs) compared with no screen and other screen alternatives; the ICER for such a screen is around £2500 per QALY gained; the range of expected costs, QALYs and net benefits was broad, indicating a considerable degree of uncertainty

• targeted screening could be more cost-effective than universal SES

• lack of primary data and the wide limits for variables in the modelling mean that any conclusions must be considered indicative and exploratory only.

A national screening programme for permanent hearing impairment at school entry meets all but three of the criteria for a screening programme, but at least six criteria are not met for screening for transient hearing impairment.

Conclusions

The lack of good-quality evidence in this area remains a serious problem. Services should improve quality and audit screen performance for identification of previously unknown permanent hearing impairment, pending evidence-based policy decisions based on the research recommendations.

Recommendations for research

Further research is highlighted in the following areas:

• evaluation of an agreed national protocol for services delivering the SES to make future studies and audits of screen performance more directly comparable

• development and evaluation of systems for data monitoring so that robust data on screen performance are available

• determination with greater certainty of the prevalence of congenital unilateral hearing impairment, and permanent mild and minimal hearing impairment at school entry, that could be identified by a suitable quality-assured screen protocol

• a comparison of the effectiveness, efficacy and efficiency of alternative approaches (reactive services, formal surveillance, targeted screening and universal screening at school entry age) to the identification of permanent hearing impairment post newborn screen

• controlled studies of the effectiveness of hearing screening and subsequent interventions for later outcomes in children with permanent mild, minimal and unilateral hearing impairment identified at school entry

• determination of the distribution of detection thresholds for pure tones in the population at school entry.

Ovid MEDLINE(R)

Date range of search: January 2005 to July 2014.

Date of search: 10 July 2014.

EMBASE

Date range search from: January 2005 to July 2014.

Date of search: 10 July 2014.

Cumulative Index to Nursing and Allied Health Literature

Date range searched from: January 2005 to July 2014.

Date of search: 10 July 2014.

PsycINFO

Date range searched: January 2005 to July 2014.

Date of search: 10 July 2014.

Education Resources Information Center (Cambridge Scientific Abstracts)

Date range searched: January 2005 to July 2014.

Search date: 10 July 2014.

Science Citation Index (Web of Knowledge)

Date range searched: January 2005 to July 2014.

Search date: 10 July 2014.