Developing routinely recorded clinical data from electronic patient records as a national resource to improve neonatal health care: the Medicines for Neonates research programme

Electronic patient records

Electronic patient records have been used increasingly over the last two to three decades and represent a rich data resource. Successive UK governments have recognised the potential of NHS clinical data to improve patient care and outcomes. 1–3 However, these aspirations have been slow to be adequately realised.

The challenges faced in harnessing the power of clinical data in health care are perhaps exemplified by the lessons of care.data and other high-profile UK projects. These include limited population coverage, weak clinical engagement, unknown data quality, regulatory uncertainty and public disquiet. Confidence in the concept of clinical data as a resource to improve patient care, despite an ambition to use these to improve standards, quality of care, accountability and patient choice, has been further damaged by escalating costs, critical media reports, breaches of data security and loss of public confidence by reports that personal data would be ‘sold’ to commercial organisations.

Neonatal specialised services

In the UK, neonatal specialised services (i.e. services for newborn infants requiring care over and above normal care) are currently provided by neonatal units operating in a series of mature clinical networks, each comprising around six to eight neonatal units. Neonatal networks were introduced as part of the restructuring of neonatal services in response to a report by the Department of Health and Social Care in 2002. Each neonatal network was to be largely self-sufficient in providing care across the complete range of intensities (the traditional intensive, high dependency and special care levels). As a result, large numbers of infants were transferred between neonatal units providing different levels of care in accordance with their care requirements, with ultimate ‘repatriation’ to a neonatal unit closest to home in preparation for discharge. The concurrent need for clinical information to be readily transferable between NHS provider trusts was a cardinal driver for the introduction of EPR technologies into neonatal units.

Prior to the restructuring of neonatal services into networks, the British Association of Perinatal Medicine (BAPM) had commenced developing a ‘minimum’ Neonatal Data Set. 4 Neonatologists had long recognised the benefits of a uniform approach to recording clinical information, including the the ability to evaluate outcomes consistently at a national level. Over the period of the restructuring and subsequently, successive BAPM working parties made refinements to the ‘minimum’ Neonatal Data Set. 4

Neonatal electronic patient records

Over the last decades, a UK-based commercial firm had developed a technical platform for neonatal data in close consultation with neonatal clinicians. This platform has evolved, with successive versions introduced into use over the years. Electronic systems were introduced across all NHS provider trusts from 2005 onwards. The EPR system in most widespread use includes fixed-choice and free-text items, with the NHS number as the principal identifier. Data are recorded daily throughout the neonatal inpatient stay. Clinician-entered diagnoses are converted to the International Classification of Diseases, Tenth Edition (ICD-10) codes. 5

Thus, the reorganisation of neonatal services into managed clinical networks, an established, commercially available technical platform with a user front-end developed in close collaboration with neonatal clinicians, and a ‘minimum’ Neonatal Data Set established by a professional organisation, provided the three prime underpinning requirements on which to develop electronic clinical data for secondary purpose, including research and evaluation. Members of the Medicines for Neonates investigator group had been involved in several of the developments in relation to neonatal data described above, and electronic records more widely (e.g. as members of successive BAPM data working parties) and, hence, brought a wealth of experiential knowledge to the programme.

Systematic review methods

We carried out an electronic search on MEDLINE (via Ovid), EMBASE (via Ovid), and CINAHL (Cumulative Index to Nursing and Allied Health Literature; via Athena), of publications covering the period 1 January 2000 to 15 March 2015. We applied language restrictions, including only English, French, German, Italian, Russian and Spanish articles. We employed the following search terms: ‘intensive care units, neonatal/’ OR ‘intensive care, neonatal/’ OR ‘neonatal intensive care units’ OR ‘NNU’ OR ‘NICU’ OR ‘neonatal ICU’ AND ‘infant/’ OR ‘neonat$’ AND ‘database$’ or ‘registry’ OR ‘registries’ OR ‘dataset$’ OR ‘data set$’ OR ‘vital statistics’. The literature search strategy is summarised in Figure 1.

FIGURE 1.

Literature search strategy.

We carried out grey literature searches on the Web of Science and the Ovid Maternity and Infant Care Databases using the free-text terms ‘neonatal intensive care unit’ AND ‘infant’ AND ‘database’.

We exported results, including abstracts, into EndNote X7 [Clarivate Analytics (formerly Thomson Reuters), Philadelphia, PA, USA]. Two researchers reviewed titles and abstracts to identify relevant publications and remove duplicate results. We retained publications that mentioned databases of patient-level information (administrative or clinical) and specified that data covered populations of infants from more than one neonatal unit. We reviewed full-text articles, references and websites. We entered extracted predefined information into Microsoft Excel^® 2013 (Microsoft Corporation, Redmond, WA, USA) (Table 1).

Creating the Neonatal Data Set

We built on and extended the BAPM ‘minimum’ Neonatal Data Set (data items used to derive daily level of care, the currency underpinning the commissioning of neonatal specialised care services) and the mandated National Critical Care Minimum Data Set (NCC-MDS) (used for deriving neonatal Healthcare Resource Groups) to build a national ‘Neonatal Data Set’. This comprised basic demographic details (e.g. date of birth, birthweight), clinical interventions captured daily (e.g. respiratory support, type of feeds, surgical procedures, high-cost drugs), clinical outcomes and diagnoses. Each data item is clearly defined in an accompanying metadata set, and mapped to existing national standards as well as ICD codes. There was a preliminary assessment of the compatibility of Neonatal Data Set items for conversion to Snomed computed tomography (CT) terminology (international medical nomenclature); the conclusion was that the Neonatal Data Set is compatible, but conversion would require clinical and technical resourcing.

With the support of the NHS Information Standards Board (now NHS Digital), we submitted the Neonatal Data Set for approval as a national NHS standard. Following initial submission, the Information Standards Board issued an ‘advance notice’ of the Neonatal Data Set standard. In the process to becoming a national standard, the Neonatal Data Set evolved through changes that came about following public consultation, review by terminology experts at the NHS data dictionary, and alignment to other national data sets. As a result, 25 data items were added to the revised Neonatal Data Set and an existing 28 items were recoded to reflect data dictionary terminology or other national criteria. Full approval of the standard was obtained in December 2013. The stages leading to approval are shown in Table 2. The current approved Neonatal Data Set (SCCI1595) for standard items and age 2 years items are provided as Appendix 3.

Systematic review

Search results

The search yielded 2037 unique papers. Following a review of the titles and abstracts, 415 papers met our prespecified criteria. From these, we identified 82 databases and, for 52, data were recorded specifically for the database. In 21 papers, data were obtained from a primary administrative source and in nine papers data were obtained from a clinical source (Figure 2). Five countries accounted for the location of more than half (47/82) of all identified databases: the USA (n = 24), Canada (n = 11), the UK (n = 7) and Australia/New Zealand (n = 5). We provide details of the databases identified in Table 3.

FIGURE 2.

Data flows into the Neonatal Data Analysis Unit.

TABLE 3 - Details of databases identified

a, administrative; c, clinical; NICHD, National Institute of Child Health and Human Development; o, other; r, research; ROP, retinopathy of prematurity; s, surveillance; VLBW, very low birthweight; WHO, World Health Organization.

Number	Name	Primary purpose	Country	Scope	Population limits	Data source	Time period covered by publication (database still enrolling patients)	Maternal variables	Infant variables	Funding source
1	Alberta Perinatal Health Program Database6	A	Canada	Regional (Alberta)	No limitations, entire region included	Extracted from administrative data source	2002–4 (yes)	Mother’s age, mode of delivery	Birthweight, sex, multiplicity	Support from public body
2	Alere or Matria Health care/Paradigm7,8	C	USA	National	Admissions or births in enrolled hospitals	Recorded specifically for database	2003–7 (yes)	Mother’s ethnicity, mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention	Hospital subscription
3	Arizona Newborn Intensive Care Program9	A	USA	Regional (Arizona)	Admission to neonatal units, gestational age and/or birthweight cut-off point	Extracted from administrative data source	1994–8 (yes)	Mother’s ethnicity, mother’s age, mother’s education, mode of delivery	Gestational age in weeks, birthweight, sex, identifier, intervention, ROP, post-discharge information	Support from public body
4	Asian Network on Maternal and Newborn Health10	R	Asia (Malaysia, Japan, Hong Kong and Singapore)	International	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	2003–6 (yes)	Mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, diagnoses, abdominal X-rays, cranial ultrasound	Support from public body
5	AUDIOPOG Sentinel Network11	C	France	National	Admissions or births in enrolled hospitals	Recorded specifically for database	1994–2008 (yes)	Mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier	No current funding support identified
6	Australia and New Zealand Neonatal Network12	C	Australia/New Zealand	National	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	1994–2012 (yes)	Mother’s ethnicity, mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention, blood cultures, ROP, post-discharge information	Mixed funding, including support from public body
7	Better Outcomes Registry and Network13	A	Canada	Regional (Ontario)	No limitations, entire region included	Recorded specifically for database	2006–10 (yes)	Mother’s age, mother’s education, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention, ROP, post-discharge information	Support from public body
8	California Patient Discharge Linked Birth Cohort Database14,15	A	USA	Regional (California)	Admissions or births in enrolled hospitals	Extracted from administrative data source	1999–2004 (yes)	Mother’s ethnicity, mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity	Support from public body
9	California Perinatal Quality Care Collaborative16	C	USA	Regional (California)	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	2005–11 (yes)	Mother’s ethnicity, mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention	Support from public body
10	Canadian Institute for Health Information Discharge Abstract Database17	A	Canada	National	No limitations, entire region included	Extracted from administrative data source	2002–10 (yes)	Mother’s ethnicity, mother’s age	Gestational age in weeks, birthweight, sex, identifier, intervention, ROP, post-discharge information	Support from public body
11	Canadian Neonatal Follow-Up Network18	C	Canada	National	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	2010–11 (yes)	Mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention, diagnoses, blood cultures, ROP, post-discharge information	Support from public body
12	Canadian Neonatal Network19	C	Canada	National	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	2013–14 (yes)	Mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention, diagnoses, blood cultures	Support from public body
13	Canadian Paediatric Surgery Network20	C	Canada	National	Admissions or births in enrolled hospitals	Recorded specifically for database	2013–14 (yes)	Mother’s ethnicity, mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, identifier, intervention, diagnoses, abdominal X-rays, cranial ultrasound	Support from public body
14	Children’s Hospital Neonatal Database21	C	USA	National	Admissions or births in enrolled hospitals	Recorded specifically for database	2010–11 (yes)	Mother’s ethnicity	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention	Hospital subscription
15	Colorado Birth Certificate Database22	A	USA	Regional (Colorado)	No limitations, entire region included	Recorded specifically for database	2007–12 (yes)	Mother’s ethnicity	Gestational age in weeks, birthweight, sex, multiplicity, identifier	Support from public body
16	Consortium of Safe Labor Database23	R	USA	National	Admissions or births in enrolled hospitals	Extracted from clinical data source (electronic health records)	2002–8 (yes)	Mother’s ethnicity, mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention	Support from public body
17	Croatian Intensive Care network24	C	Croatia	National	Admissions or births in enrolled hospitals	Recorded specifically for database	2004–5 (yes)	No variables identified from those sought	Identifier	Support from public body
18	Danish Medical Birth Registry25	A	Denmark	National	No limitations, entire region included	Recorded specifically for database	1997–2008 (yes)	Mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier	Support from public body
19	Danish Neonatal Clinical Database (NeoBase)26	C	Denmark	Regional (North And South Jutland)	Admission to neonatal units, gestational age and/or birthweight cut-off point	Extracted from clinical data source (electronic health records)	2005–6 (yes)	No variables identified from those sought	Gestational age in weeks, birthweight, sex, identifier, intervention	Support from public body
20	Emilia-Romagna Health Agency27	A	Italy	Regional (Emilia-Romagna)	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	2002–9 (yes)	No variables identified from those sought	Gestational age in weeks	Support from public body
21	EPICure28	R	UK	National	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	2006–7 (no)	Mother’s ethnicity, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, intervention, ROP, post-discharge information	Support from public body
22	Erie County Register29	A	USA	Regional (Erie County, New York)	No limitations, entire region included	Recorded specifically for database	2006–8 (yes)	Mother’s ethnicity, mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, identifier, intervention	Support from public body
23	EuroNeoNet30	C	European	International (Austria, Belgium, Croatia, Finland, France, Germany, Greece, Italy, Poland, Portugal, Russia, Slovenia, Spain, Switzerland, the Netherlands, Turkey and the UK)	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	2006–11 (yes)	Mother’s age, mother’s education, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention, ROP, post-discharge information	Support from public body
24	Florida birth registry31	A	USA	Regional (Florida)	No limitations, entire region included	Recorded specifically for database	2009–10 (yes)	Mother’s ethnicity, mother’s age, mother’s education, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier	Support from public body
25	Intermountain Health care32	C	USA	Regional (Utah)	Admissions or births in enrolled hospitals	Extracted from clinical data source (electronic health records)	2003–5 (yes)	No variables identified from those sought	Birthweight	Hospital subscription
26	Israel National VLBW Infant Database33	C	Israel	National	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	1995–2003 (yes)	Mother’s ethnicity, mother’s age, mother’s education, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention	Support from public body
27	Japanese Vital Statistics34	A	Japan	National	No limitations, entire region included	Recorded specifically for database	1999–2008 (yes)	Mother’s age	Gestational age in weeks, birthweight, sex, multiplicity	Support from public body
28	Kaiser Permanente Medical Care Program35,36	C	USA	Regional (Northern California and Boston, Massachusetts)	Admissions or births in enrolled hospitals	Extracted from clinical data source (electronic health records)	1995–6 (yes)	Mother’s age, mother’s education, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention	Hospital subscription
29	Kids’ Inpatient Databases37	R	USA	National	Admissions or births in enrolled hospitals	Recorded specifically for database	2003–12 (yes)	Mother’s ethnicity	Birthweight, sex	Support from public body
30	Linked Emergency Management and Research Institute – Department of Health and Family Welfare, Government of Gujarat38	A	India	Regional (Gujarat)	Admissions or births in enrolled hospitals	Extracted from administrative data source	2008–9 (yes)	Mother’s age, mode of delivery	Gestational age in weeks	Support from public body
31	London Neonatal Transfer Service39	C	UK	Regional (London)	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	2005–11 (yes)	No variables identified from those sought	Gestational age in weeks, birthweight	Support from public body
32	Massachusetts Community Health Information Profile (MassCHIP) and PELL40,41	A	USA	Regional (Massachusetts)	No limitations, entire region included	Extracted from administrative data source	2002–10 (yes)	Mother’s ethnicity, mother’s age	Gestational age in weeks, multiplicity, intervention	Support from public body
33	Malaysian National Neonatal Registry42	C	Malaysia	National	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	2006–7 (yes)	Mother’s age, mother’s education, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention, blood cultures, abdominal X-rays, cranial ultrasound	Support from public body
34	Medicaid Analytic eXtract43	A	USA	National	Health insurance enrolment	Extracted from administrative data source	2006–8 (yes)	Mode of delivery	Identifier	Support from public body
35	Memorial Care Medical Centres: Perinatal database, Quality Improvement Database44	C	USA	Regional (California)	Admissions or births in enrolled hospitals	Recorded specifically for database	2002–3 (yes)	Mother’s ethnicity, mode of delivery	Gestational age in weeks, identifier, intervention	Hospital subscription
36	Michigan Linked Records45	A	USA	Regional (Michigan)	No limitations, entire region included	Extracted from administrative data source	2003–4 (yes)	Mother’s ethnicity	Gestational age in weeks, birthweight, sex, multiplicity, identifier	Support from public body
37	National Centre for Health Statistics linked live birth and infant death cohort file46	A	USA	National	No limitations, entire region included	Extracted from administrative data source	1998–9 (yes)	Mother’s ethnicity, mother’s age, mother’s education, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity	Support from public body
38	National Collaborative Perinatal Neonatal Network47	C	Lebanon	Regional (Greater Beirut)	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	2009–10 (not identified)	Mother’s age, mother’s education	Gestational age in weeks, birthweight, sex, multiplicity	Support from public body
39	National Institute for Health and Welfare: Medical Birth Register48	A	Finland	National	No limitations, entire region included	Recorded specifically for database	2012–13 (yes)	Mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention	Support from public body
40	National Neonatal Database SEN150049	C	Spain	National	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	2002–5 (yes)	Mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention, diagnoses, blood cultures, abdominal X-rays, cranial ultrasound	Support from public body
41	National Neonatal Perinatal Database50	C	India	National	Admissions or births in enrolled hospitals	Recorded specifically for database	2002–3 (yes)	Mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, intervention	Support from public body
42	NNRD51	C	UK	National	Admission to neonatal units, all infants included	Extracted from clinical data source (electronic health records)	2009–11 (yes)	Mother’s ethnicity, mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention, diagnoses, blood cultures, abdominal X-rays, ROP, cranial ultrasound, post-discharge information	No current funding support identified
43	National Perinatal Information Centre52	A	USA	National	Admissions or births in enrolled hospitals	Recorded specifically for database	2004–8 (yes)	Mother’s ethnicity, mother’s age, mode of delivery	Gestational age in weeks, sex, identifier	Support from public body
44	National Perinatal Registry of Slovenia53	A	Slovenia	National	No limitations, entire region included	Recorded specifically for database	2012–13 (yes)	No variables identified from those sought	Gestational age in weeks	Support from public body
45	National Perinatal Registry, the Netherlands54	C	The Netherlands	National	No limitations, entire region included	Recorded specifically for database	2003–7 (yes)	Mother’s ethnicity, mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier	Support from public body
46	National Perinatal Data Collection55	A	Australia/New Zealand	National	Admission to neonatal units, gestational age and/or birthweight cut-off point	Extracted from administrative data source	2001–5 (yes)	Mother’s ethnicity, mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier	Support from public body
47	National Registry of Respiratory Distress Syndrome in Romania56	C	Romania	National	Admissions or births in enrolled hospitals	Recorded specifically for database	2011–12 (yes)	No variables identified from those sought	Gestational age in weeks, birthweight, sex, identifier, intervention	Support from public body
48	Neonatal Intensive Care Outcomes and Research Evaluation57	C	UK	Regional (Northern Ireland)	Admission to neonatal units, all infants included	Extracted from clinical data source (electronic health records)	1999–2000 (yes)	Mother’s ethnicity, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention, blood cultures, abdominal X-rays, cranial ultrasound	Support from public body
49	Neonatal Research Network of Japan58	R	Japan	National	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	2003–8 (yes)	Mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, intervention, blood cultures	Support from public body
50	NEOSANO’s Perinatal Network in Mexico59	A	Mexico	Regional (Mexico City, Tlaxcala City and Oaxaca City)	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	2006–9 (yes)	Mother’s age, mother’s education, mode of delivery	Gestational age in weeks, multiplicity	Support from public body
51	New Jersey Perinatal Linked Data-Set60	A	USA	Regional (New Jersey)	No limitations, entire region included	Extracted from administrative data source	1997–2005 (yes)	Mother’s ethnicity, mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, identifier	Support from public body
52	New South Wales Newborn and Paediatric Emergency Transport Service61	C	Australia/New Zealand	Regional (New South Wales)	Admissions or births in enrolled hospitals	Recorded specifically for database	1992–2001 (yes)	No variables identified from those sought	Gestational age in weeks, birthweight, sex, intervention	Support from public body
53	New York State-wide Perinatal Data System62	A	USA	Regional (New York)	Admissions or births in enrolled hospitals	Extracted from administrative data source	1996–2003 (yes)	Mother’s ethnicity, mother’s age, mother’s education, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention	Support from public body
54	Newfoundland and Labrador Provincial Perinatal Program Database63	C	Canada	Regional (Newfoundland and Labrador)	Admissions or births in enrolled hospitals	Extracted from administrative data source	2001–9 (yes)	Mother’s age, mother’s education, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity	Support from public body
55	NICHD Neonatal Research Network Generic Database64	R	USA	National	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	1998–2009 (yes)	Mother’s ethnicity, mother’s education, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention, ROP, post-discharge information	Support from public body
56	Nova Scotia Atlee Perinatal Database65	A	Canada	Regional (Nova Scotia)	No limitations, entire region included	Extracted from administrative data source	2002–11 (yes)	Mother’s ethnicity, mother’s age, mother’s education, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention	Support from public body
57	NSW Pregnancy and Newborn Services Network66	C	Australia/New Zealand	Regional (New South Wales And Australian Centralised Territory)	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	1997–2006 (yes)	Mother’s ethnicity, mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, intervention	Support from public body
58	Pediatrix BabySteps Clinical Data Warehouse67	C	USA	National	Admission to neonatal units, all infants included	Extracted from clinical data source (electronic health records)	1996–2010 (yes)	Mother’s ethnicity, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention, blood cultures	Hospital subscription
59	Perinatal and Neonatal Surveys in Saxony68	C	Germany	Regional (Saxony)	No limitations, entire region included	Recorded specifically for database	2001–5 (yes)	No variables identified from those sought	Gestational age in weeks, birthweight, sex, multiplicity, identifier	Support from public body
60	Perinatal database of Middlesex Country, Canada69	C	Canada	Regional (Middlesex County, Ontario)	Admissions or births in enrolled hospitals	Recorded specifically for database	2002–11 (yes)	Mother’s ethnicity, mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex	Support from public body
61	Perinatal Revision South70	C	Sweden	Regional (Southern Sweden)	Admissions or births in enrolled hospitals	Recorded specifically for database	1995–6 (yes)	Mode of delivery	Gestational age in weeks, birthweight, identifier	Support from public body
62	Perinatal Services British Columbia71	A	Canada	Regional (British Columbia)	Admissions or births in enrolled hospitals	Extracted from administrative data source	2004–14 (yes)	Mother’s age, mother’s education, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention	Support from public body
63	Population Health Research Data Repository72	A	Canada	Regional (Manitoba)	No limitations, entire region included	Extracted from administrative data source	2004–9 (yes)	Mother’s age	Gestational age in weeks, birthweight, sex, multiplicity, identifier	Support from public body
64	Scottish Administrative Linked Data73	A	UK	National	No limitations, entire region included	Extracted from administrative data source	1981–2007 (yes)	Mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier	Support from public body
65	Seguro Medico para una Nueve Generacio74	A	Mexico	National	Health insurance enrolment	Recorded specifically for database	2008–9 (yes)	No variables identified from those sought	Gestational age in weeks, sex, identifier	Support from public body
66	Swedish Neonatal Quality Register75,76	C	Sweden	National	Admission to neonatal units, all infants included	Extracted from administrative data source	2001–2 (yes)	Mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention, blood cultures, ROP, post-discharge information	Support from public body
67	Swiss Neonatal Network77	C	Switzerland	National	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	1996–2008 (yes)	Mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, intervention, blood cultures, ROP, post-discharge information	Hospital subscription
68	Taiwan’s National Health Insurance Research Database78	A	Taiwan	National	Health insurance enrolment	Extracted from administrative data source	1998–2001 (yes)	No variables identified from those sought	Gestational age in weeks, birthweight, sex, intervention, ROP, post-discharge information	Support from public body
69	Tennessee Hospital Discharge Data System79	A	USA	Regional (Tennessee)	Admissions or births in enrolled hospitals	Extracted from administrative data source	2003–5 (yes)	Mother’s ethnicity	Birthweight, sex, identifier, intervention	Support from public body
70	The National Neonatology Database80	C	The Netherlands	National	Admission to neonatal units, all infants included	Recorded specifically for database	2003–5 (yes)	No variables identified from those sought	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention	Support from public body
71	The Neonatal Survey81	C	UK	Regional (East Midlands and Yorkshire)	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	2008–10 (yes)	Mother’s ethnicity, mother’s age, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, infant identification, intervention, diagnoses, laboratory samples, ROP, post-discharge information	Support from public body
72	The WHO’s Global Survey for Maternal and Perinatal Health82,83	S	International [Africa (Angola, Democratic Republic of Congo, Algeria, Kenya, Niger, Nigeria and Uganda), Latin America (Argentina, Brazil, Cuba, Ecuador, Mexico, Nicaragua, Paraguay and Peru) and Asia (Cambodia, China, India, Japan, Nepal, Philippines, Sri Lanka, Thailand and Vietnam)]	International	No limitations, entire region included	Recorded specifically for database	2004–8 (no)	Mother’s age, mother’s education, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier	Support from public body
73	Vermont Oxford Network84	C	International (Australia, Brazil, Canada, China, Columbia, Finland, Germany, Hungary, Ireland, Italy, Kuwait, Malaysia, Namibia, Poland, Portugal, Qatar, Romania, Saudi Arabia, Singapore, Slovenia, South Africa, Spain, Switzerland, Taiwan, Turkey, United Arab Emirates, the UK and the USA)	International	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	1990–2012 (yes)	Mother’s ethnicity, mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention, blood cultures, abdominal X-rays, cranial ultrasound	Hospital subscription
74	Victorian Perinatal Data Collection Unit85	R	Australia/New Zealand	Regional (Victoria)	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	1979–97 (no)	No variables identified from those sought	Birthweight, sex, identifier, ROP, post-discharge information	Support from public body
75	West Midlands Perinatal Institute86	C	UK	Regional (West Midlands)	No limitations, entire region included	Recorded specifically for database	2008–9 (yes)	No variables identified from those sought	No variables identified from those sought	No current funding support identified
76	Wisconsin Linked Birth Record File87	A	USA	Regional (Wisconsin)	Health insurance enrolment	Extracted from administrative data source	2001–2 (yes)	Mother’s ethnicity, mother’s age, mother’s education, mode of delivery	Birthweight, sex, multiplicity, identifier	Support from public body
77	AOK National Insurance Entries88	A	Germany	National	Health insurance enrolment	Recorded specifically for database	2002–6 (yes)	No variables identified from those sought	No variables identified from those sought	Insurance
78	Regional Census Data89	A	Germany	Regional (Westfalen Lippe)	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	1990–6 (yes)	No variables identified from those sought	Blood cultures	Support from public body
79	Neonatal Quality Assurance System90	A	Germany	Regional (Baden Wuertemberg)	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	2003–4 (yes)	No variables identified from those sought	No variables identified from those sought	Support from public body
80	Bourgogne database91	A	France	Regional (Bourgogne)	Admissions or births in enrolled hospitals	Extracted from clinical data source (electronic health records)	2000–1 (yes)	Mode of delivery	Gestational age in weeks, birthweight, sex, multiplicity, identifier, intervention	Support from public body
81	Multicentre national database92	R	France	National	Admission to neonatal units, gestational age and/or birthweight cut-off point	Recorded specifically for database	2005–6 (yes)	No variables identified from those sought	Gestational age in weeks, sex, intervention	Support from public body
82	Hessen Neonatal Register93	A	Germany	Regional (Hessen)	No limitations, entire region included	Recorded specifically for database	1989–2012 (yes)	No variables identified from those sought	No variables identified from those sought	Support from public body

Regulatory approvals

We obtained National Research Ethics Service approval in 2010 to establish a NNRD from extracts from EPRs, undertake projects within the Medicines for Neonates Programme, and employ the NNRD for NHS service evaluations and other research studies (REC reference number 10/H0803/151; provided as Appendices 5 and 6). We obtained approval in 2010 from the Confidentiality Advisory Group of the Health Research Authority [formerly Ethics and Confidentiality Committee of the National Information Governance Board; reference number ECC 8–05(f)/2010; provided as Appendix 4] to receive specific patient identifiers for the purpose of linking to Hospital Episode Statistics (HES) data.

Under standard operating procedures for Research Ethics Committees, site-specific approval is not required for studies conducted using research databases. There is no requirement for specific ethics approval for data collection centres that provide data because, under the Research Governance Framework, data collection centres are not regarded as research sites. NHS trust approval (‘R&D’ approval) is only required from the NNRD host institution (i.e. not from each data collection centre). However, we were informed that ‘local collaborators at Data Collection Centres within the NHS will require internal permission from their NHS care organisation to collect and supply data relating to NHS patients’. 94 We addressed this requirement by seeking Caldicott Guardian and Lead Neonatal Clinician approval from every NHS trust providing neonatal specialised care, to receive a defined extract from their neonatal EPRs, to hold these in the NNRD and to use these in NHS service evaluations and Research Ethics Committee-approved research studies. We obtained approvals incrementally and all NHS trusts in England, Wales and Scotland have now granted approval.

The National Neonatal Research Database

The data items constituting the Neonatal Data Set (NND) are extracted from EPRs created by clinical staff on all admissions to neonatal units in England, Wales and Scotland. Neonatal units in Northern Ireland utilise the same EPR platform but, to date, the regulatory approvals governing data transfer into the NNRD have not been sought. Following receipt of the necessary approvals, retrospective data extraction was undertaken so that the NNRD contains data from 2007 to the present. The NNRD is updated quarterly, and to date it contains data on approximately half a million infants and > 5 million care days. All neonatal units across England, Wales and Scotland have approved the release of Neonatal Data Set data items for inclusion in the NNRD (the total number of neonatal units is approximately 200, which has fluctuated over the course of the Medicines for Neonates programme as neonatal units have merged or reorganised).

An NHS approved supplier, Clevermed Ltd (Edinburgh, UK), provides a web-based data capture platform known as Neonatal.Net or BadgerNet. Data held by Clevermed Ltd are stored on a secure N3 server and transmitted to the Neonatal Data Analysis Unit (NDAU) where they are used to create the NNRD after merging and cleaning of files. The NNRD is held on the NHS servers of Chelsea and Westminster NHS Foundation Trust. Data flows are shown in Figures 2 and 3.

FIGURE 3.

Data flows to create the NNRD. ONS, Office for National Statistics.

Data management

At the NDAU, all data extracted from the neonatal EPRS are interrogated to identify duplicate, missing and potentially erroneous entries. Items are considered potentially erroneous if they fail a series of out of range, internal logic, and internal inconsistency checks.

A web-based portal was created to notify neonatal unit lead clinicians of missing or potentially erroneous entries. If clinical teams amend errors or complete missing fields, this is done in the baby’s EPR and this is sent to the NDAU at the next download. Initially, this process was confined to the data items (approximately 60 items) used for analyses for the National Neonatal Audit Programme. In addition, as the NNRD has become used for research studies, if key data items are required for specific projects then these are also subjected to the feedback loop process.

At the NDAU, patient episodes across multiple neonatal units are also merged to create a single file for each infant.

Clinician engagement

We termed NHS neonatal units contributing data to the NNRD the ‘UK Neonatal Collaborative’. Of note was that, although only site-specific approval and NHS approvals are required for the NNRD host institution, we would adopt a policy of seeking the approval of each NHS trust’s lead neonatal clinician for their data to be included in research studies. We adopted this practice in order to grow clinician engagement with the concept of the NNRD as a national resource and in recognition of their crucial contribution to acquiring the data.

Parent information leaflet

We collaborated with the Royal College of Paediatrics and Child Health, the national charity Bliss and the parents of newborn babies receiving specialised neonatal care to develop a parent information leaflet (‘A Guide for Parents and Carers’) that explains the multiple uses of the NNRD. This was approved by the National Research Ethics Service and the Ethics and Confidentiality Committee of the National Information Governance Board.

If the parent or carer of the infant does not wish for EPR data on their infant to be extracted for the NNRD, then they can notify the neonatal unit staff who will then notify the data entry system supplier to prevent the flow of the data. To date, no parent or carer has asked that their infant’s data not be extracted.

Uses and outputs of the National Neonatal Research Database to date

The use of the NNRD to support a wide range of outputs grew rapidly over the course of the Medicines for Neonates Programme, and continues to expand. Examples of the multiple outputs from the NNRD are shown in Figure 4.

FIGURE 4.

Examples of multiple outputs from the NNRD.

Approvals and agreements

We sought and obtained research ethics approval from the National Research Ethics Service (Dulwich Research Ethics Committee; reference number 11/LO/1430) and inclusion into the UK Clinical Research Network Portfolio (ID 11853). We invited the participation of all neonatal units in England. We sought agreement from the local UK Neonatal Collaborative lead clinicians to utilise data from their neonatal unit held in the NNRD on all live-born infants admitted over the complete 2-year period 2012–13.

In order to promote maximal engagement with the neonatal community and optimise data quality and completeness, we asked local clinical staff to ensure that the following data items were recorded in each infant’s EPR: birthweight, gestational age, sex, mother’s race, antenatal steroids, and clinical and abdominal X-ray (AXR) findings for infants in whom abdominal signs were being investigated. We excluded infants of mothers that were unbooked, booked in non-English networks, or for whom network of booking was unknown.

Identifying babies with severe necrotising enterocolitis in the National Neonatal Research Database

We defined ‘severe NEC’ as NEC confirmed at surgery or post-mortem or resulting in death (death certification and/or verified by neonatal team). We initially intended to capture outcomes on a specific section of the neonatal EPRs, the screen used to record details of abdominal X-rays taken to investigate clinical signs consistent with gastrointestinal pathology. However, despite regular quarterly feedback of data completeness to neonatal units, we found that only 25% of infants who proceeded to NEC surgery had completed this screen; hence, sole use of these data would underestimate the incidence of severe NEC. Therefore, we used data from the NNRD to identify infants who may have received surgery for NEC or died from NEC, and verified these outcomes with clinicians at neonatal units. In addition, outcomes were sought for infants who received NEC surgery or who died at the four stand-alone paediatric surgical centres which do not use the BadgerNet neonatal EPRs (Great Ormond Street, Sheffield, Alder Hey and Birmingham Children’s Hospitals). Here, we describe the steps taken to identify and verify infants with severe NEC.

Other data extraction from the National Neonatal Research Database: data management

We extracted the following data for all babies from the NNRD: booking network, gestational age in completed weeks and days, birthweight, fetus number, antenatal steroids, maternal pyrexia in labour, whether or not mother received antibiotics, mode of delivery, maternal chorioamnionitis and maternal infection. We calculated birthweight SDS, standardised for sex and gestational age from UK World Health Organization (WHO) reference data. 113 We considered a birthweight SDS of < –4 or > 4 to be erroneous, and we treated these as missing values.

Analyses

Data validation

We compared data from the NNRD with data from paper medical notes as part of a project evaluating a quality improvement project conducted by the East of England Neonatal Networks. 116 In brief, between 2011 and 2013, we assessed and fed back completeness and accuracy of NNRD data to participating neonatal teams involving 17 neonatal units. The study lead at each neonatal unit extracted a selection of data items from medical notes for two randomly selected infants discharged in the previous month. These data were sent to the NDAU and compared with NNRD data.

Incidence and numbers of cases of severe necrotising enterocolitis

We extracted data on a total population cohort of 118,073 infants (Figure 5). We identified 531 infants with severe NEC. Table 4 shows the proportion of infants with severe NEC who had surgery and survived to discharge from neonatal care, who had surgery and died, and who died without surgery, by gestational age bands. Of the total number of infants with severe NEC 79.7% (423/531) had surgery; of those who had surgery 32.9% (139/423) died; 20.3% (108/531) of infants with severe NEC had died without surgery (Figure 6).

FIGURE 5.

Flow chart showing derivation of the study population.

FIGURE 6.

Flow chart showing the population with severe NEC.

Comparing ONS data with NNRD data from 2012 shows that of the infants who were born alive in England between the gestational ages of 25 and 31⁺⁶ weeks, 96–99% were admitted to a neonatal unit (Figure 7); the corresponding figures are 92% and 70% of infants born at 24 and at 23 weeks’ gestation, respectively. The percentage of infants who were admitted to a neonatal unit starts to fall after 32 weeks’ gestation. Therefore, as we have population data for infants born before 32 weeks’ gestation, we present the incidence for these infants; but for infants who were born after 32 weeks’ gestation, we present only the raw numbers.

FIGURE 7.

Percentage of ONS-reported live births in the NNRD.

The incidence of severe NEC was inversely related with gestational age (p < 0.001, test for trend). The highest incidence of severe NEC occurred in infants born at 24 weeks’ gestation. The incidence per 1000 infants for all infants born before 32 weeks’ gestation is 31.5 per 1000 (95% CI 28.7 to 34.3 per 1000), and by gestational age is as follows: 23 weeks, 96.4 (95% CI 66.8 to 125.9); 24 weeks, 112.4 (95% CI 90.0 to 134.8); 25 weeks, 86.9 (95% CI 68.4 to 105.5); 26 weeks, 58.9 (95% CI 45.1 to 72.7); 27 weeks, 41.0 (95% CI 30.6 to 51.3); 28 weeks, 39.0 (95% CI 30.1 to 48.0); 29 weeks, 10.9 (95% CI 6.5 to 15.3); 30 weeks, 9.5 (95% CI 5.8 to 13.2); and 31 weeks, 5.3 (95% CI 2.9 to 7.7). Figure 8 shows that there is a sharp decline in the incidence of severe NEC at 29 weeks’ gestation.

FIGURE 8.

Incidence of severe NEC, infants born before 32 weeks’ gestation. Incidence derived from raw data; bars are 95% CI; gestation category labelled 23 weeks represents 22 and 23 weeks (14 infants born at 22 weeks, none with severe NEC).

Of the 531 infants with severe NEC, 462 (87%) were born before 32 weeks’ gestation, of whom 366 received surgery. Table 5 shows the postnatal and postmenstrual age at NEC surgery for infants < 32 weeks’ gestation. There is an inverse relationship between gestational age at birth and postnatal age at surgery (log-rank test < 0.001). The most immature infants born, before 26 weeks’ gestation, receive surgery for NEC around the third to fourth week of life; in contrast infants who are born at 30–31 weeks’ gestation undergo surgery in the second week of life.

Factors associated with severe necrotising enterocolitis for infants born before 32 weeks’ gestation

We compared the patient characteristics of the 462 preterm infants who developed severe NEC against the 14,216 preterm infants without severe NEC. Infants who developed severe NEC were more immature, with a mean gestational age of 26.2 weeks compared with 28.5 weeks (p < 0.001). Gestational age, birthweight, birthweight SDS, fetus number, whether or not the mother received antibiotics in labour, and mode of delivery were significantly different between the two groups (see Appendix 1, Table 53).

Using univariate logistic regression for infants born before 32 weeks’ gestation (n = 14,678), we identified fetus number, whether or not the mother had received antibiotics, mode of delivery, gestational age (completed weeks) and birthweight SDS to be significantly associated with severe NEC. After multivariable logistic regression analysis, only gestational age and birthweight SDS were significant independent predictors of severe NEC. We investigated the effect of including antenatal steroid exposure in the multivariable model and found that the conclusions were unchanged. As antenatal steroids are an important determinant of survival, we chose nonetheless to include this is our model. Appendix 1, Table 54, shows parameters for the final multivariable model that include gestational age, birthweight SDS and antenatal steroid exposure.

Incidence of severe necrotising enterocolitis by neonatal network

The incidence of severe NEC per 1000 infants born before 32 weeks’ gestation, by network of booking, is shown in Table 6. The highest incidence was 47.4 cases per 1000 babies (95% CI 32.5 to 62.4); the lowest incidence was 19.8 (95% CI 9.1 to 30.4). The rate of severe NEC in England is 3.15% (462/14,678). We calculated adjusted NEC rates (adjusted for gestational age, birthweight SDS, antenatal steroid exposure) for infants born before 32 weeks’ gestation. The adjusted funnel plot in Figure 9 illustrates that the network-level variation in severe NEC rates is consistent with the pattern we would expect given that the population rate of NEC is 3.15% (i.e. we identified no strong evidence that any neonatal network differed from the mean national incidence of severe NEC).

FIGURE 9.

Funnel plot for infants born before 32 weeks’ gestation, showing percentage of observed to predicted NEC cases estimated from the multivariable logistic regression model (adjusted for gestation, birthweight SDS and antenatal steroids) relative to average percentage of NEC cases in England. 1 = Bedfordshire and Hertfordshire; 2 = Cheshire and Merseyside; 3 = Eastern; 4 = Greater Manchester; 5 = Kent; 6 = Lancashire and South Cumbria; 7 = London North Central; 8 = London North West; 9 = London South East; 10 = London South West; 11 = Midlands Central; 12 = Midlands South West; 13 = Midlands North; 14 = North Trent; 15 = Northern; 16 = Peninsula South West; 17 = South Central (North); 18 = South Central (South); 19 = Surrey and Sussex; 20 = Trent; 21 = Western; 22 = Yorkshire; and 23 = London North East.

For comparison between networks, we selected a reference network with the largest number of infants born before 32 weeks’ gestation (London North East with 1014 infants) to minimise the standard errors. We identified a statistically significant difference in the incidence of severe NEC between the reference network and all the other 22 networks combined (overall p-value of < 0.001). We therefore proceeded to perform pairwise comparisons between each network and the reference network (see Appendix 1, Table 55). Following correction for multiple testing, we found no statistically significant difference in the incidence of severe NEC between any network and the reference network.

Comparison of National Neonatal Research Database against East of England medical notes

There was high agreement (> 95%) for sex, gestational age, birthweight and discharge destination (see Results). Agreement for the length of parenteral nutrition improved from 80–90% to > 90% over the 26-month study duration (p < 0.029). The agreement for central line days was consistently > 80%, with no significant change over time. Agreement on type of discharge feed improved over time, from 50–60% to 70–80% (p < 0.009). Agreement for numbers of AXR and blood cultures was consistently low, at around 50%. Further results of this work are available in the published manuscript. 116

Statistical analysis

Direct standardisation was used to control for population differences, as this permits comparison of rates over time, unlike indirect standardisation. Survival rates were directly standardised for risk. Infants were grouped into 10 categories based on the probability of death predicted by the regression model. The thresholds for the 10 categories were calculated such that each group had an equal number of predicted deaths. The directly standardised rate for each period is the weighted sum of the survival rates in each risk group, with the weights determined by the proportion of infants in each risk group in the whole study cohort. Sensitivity analyses were performed using 5 and 15 risk categories.

Prediction model

We used multivariable logistic regression to model the probability of death before discharge from neonatal care. Variables included in the regression model were gestational age, birthweight, sex, multiplicity of pregnancy (singleton/multiple) and administration of any antenatal steroids (no/yes). Previous research has shown these variables to be significant predictors of mortality. We used spline terms to model gestational age and birthweight and their interaction, using simpler functions if the fit was comparable. We included interactions between multiple birth, gestational age and birthweight, as the influence of these variables on survival is different for singleton and multiple pregnancies. As outcomes for babies from the same pregnancy are likely to be correlated, we used generalised estimating equations (GEEs) to account for the lack of independence.

We excluded observations if > 1% were missing. Otherwise, missing outcome and covariate data were imputed 25 times using multiple imputation with chained equations based on all other variables in the prediction model. Sensitivity analysis using complete cases was performed.

We carried out modelling using Stata^® 12 (StataCorp LP, College Station, TX, USA). We present results as regression coefficients with standard errors (SEs). We produced isosurv graphs135 using the ggplot2 package in R 2.13.2 (The R Foundation for Statistical Computing, Vienna, Austria) to show contours of survival probability by gestation and birthweight. We added birth year to the model used to generate the graphs so that predictions would be calibrated to the most recent year.

Model performance

We checked discrimination (ability to differentiate between babies that survived and those who died) by calculating the area under the receiver operating characteristic curve (AUC). We calculated the Brier score as a measure of overall model fit (range 0–1; 0 means better fit). We checked calibration (comparability of actual and predicted survival) using Cox’s calibration in gestational age subgroups (< 28⁺⁰ weeks and ≥ 28⁺⁰ weeks). If the model predicts perfectly across all survival probabilities, the intercept α will equal 0 and the slope β will equal 1. As model performance was assessed on the same data set used to build the model, these measures were corrected for optimism using 200 bootstrap samples.

Comparison with existing models

We compared model performance with three previously published models to predict death before discharge in preterm babies admitted to neonatal units: (1) the Clinical Risk Index for Babies II (CRIB II), a frequently used model based on data from infants born in 1998–9 and admitted to 35 UK neonatal units; (2) a more recent UK model using data from neonatal units in the East Midlands and Yorkshire region (The Neonatal Survey); and (3) the National Institute of Child Health and Development Neonatal Research Network model based on infants born at 22⁺⁰–25⁺⁶ weeks’ gestation in 1998–2003 who were admitted to 19 hospitals in the USA. The relevant subset of the NNRD cohort was used to match the population characteristics of the comparator model. The predicted survival rate, AUC, Brier score and the intercept and slope from Cox’s calibration were compared between the NNRD and comparator models.

Time trend analysis

For survival to discharge and to 28 days, trends over time were analysed using joinpoint regression applied to quarterly periods using Joinpoint 4.2.0 software (National Cancer Institute, Bethesda, MD, USA). This method allows detection of changes in trend when the number and location of the changes are unknown. Rates were log-transformed so trends are presented as annual percentage change, which is the annual rate of change of the survival rate. Heteroskedastic errors were allowed using weighted least squares, with weights inversely proportional to the variance. As the number of contributing neonatal units increased over time, we repeated all time trend analyses using data from complete neonatal networks only as a sensitivity analysis. Differences in the time of death across years were tested using quantile regression.

Variation by region and Index of Multiple Deprivation quintile

We restricted this analysis to data from 2011 onwards as lower population coverage in earlier years may bias regional estimates. Infants were assigned to one of the four NHS commissioning regions (London, Midlands and East of England, North of England and South of England) based on LSOA of mother’s residence. Crude and directly standardised rates of survival to discharge and associated 95% CIs were calculated for each region. Trends in crude survival were estimated and compared for each region using joinpoint regression; standardised trends by region were not calculated because of the low quarterly numbers in each risk group.

To examine whether or not survival experiences differ by socioeconomic deprivation, crude and directly standardised rates of survival to discharge were calculated for the highest and lowest IMD quintile and compared using RR. NHS commissioning region (categorical) and IMD decile (continuous) were added in the risk adjustment model to test whether or not there was evidence of residual variation across regions.

Validation with Office for National Statistics data

For validation purposes, the number of deaths before 28 days of infants born in England and Wales at 22 to 31 weeks’ gestation in 2012 was compared in the NNRD data (denominator is neonatal unit admissions) and published ONS data (denominator is live births). 136 Data were compared for 2012 only, owing to comparability of sources.

Comparison with previous national data

The EPICure studies examined survival and morbidity outcomes for all infants born 22⁺⁰ to 25⁺⁶ weeks’ gestation during 10 months of 1995 in the UK (EPICure) and all infants born at 22⁺⁰ to 26⁺⁶ weeks’ gestation in 2006 (EPICure 2) in England. 124,137 Survival outcomes were reported separately for infants admitted to neonatal units, giving a population comparable to the NNRD cohort. We compared data on survival to discharge of admitted infants born at 22⁺⁰ to 25⁺⁶ weeks’ gestation in England in EPICure, EPICure 2 and this study cohort using joinpoint regression to see whether or not the rate of improvement has changed.

Mortality by Operational Delivery Network

We obtained data from the NNRD on infants born in 2013 and 2014 at ≤ 31⁺⁶ weeks’ gestation and admitted to neonatal care for whom the neonatal network of booking was known. Death was defined as death before discharge from neonatal care. We used multiple imputation (applying the mi routine in Stata, version 12) to impute missing outcome and covariate data; analysis of complete-case data was also performed. We present standardised mortality ratios (SMRs) for each ODN, assigning infants to the network of booking. Crude and adjusted SMRs are presented for 2013 and 2014 combined.

The SMR was calculated as the observed number of deaths divided by the expected number of deaths. The observed number of deaths was averaged over the imputed data sets so that infants with missing outcomes were included. For the unadjusted SMR, the expected number of deaths was calculated as the total number of infants multiplied by the overall mortality rate across all networks. For the adjusted SMR, the expected number of deaths was calculated by estimating the probability of death for each infant using logistic regression, and adding up the probabilities to obtain the expected number of deaths. The 95% CIs for the SMRs were calculated using Byar’s approximation138 with correction for multiple testing, controlling the false discovery rate at 5%. 115

The logistic model used to estimate the probability of death was derived using data from babies born at ≤ 31⁺⁶ weeks’ gestation in England in 2008–14. Multivariable logistic regression was used with survival to discharge from neonatal care as the outcome. Predictor variables were gestational age (typically the best obstetric estimate from antenatal ultrasound), birthweight, sex, multiplicity of pregnancy (singleton/multiple), administration of any antenatal steroids (no/yes). Spline terms were used to model gestational age and birthweight and their interaction. The association between gestation and mortality is known to be different among singletons and multiples;139 a similar interaction effect has been shown for birthweight. 140 Interactions between multiple birth and gestational age/birthweight terms were therefore included.

As outcomes for infants from the same pregnancy are likely to be correlated, we used GEEs to account for the lack of independence. We used funnel plots to illustrate the variation in SMR. Funnel plot limits were drawn corresponding to 2 and 3 standard deviations (SDs) from the target SMR of 1, assuming the observed deaths follow a Poisson distribution. The limits were adjusted for multiple testing141 controlling the false discovery rate at 5%.

Population

Data were available for 71% of neonatal units from the beginning of 2008, 80% in 2009, 86% in 2010, 97% in 2011, 99% in 2012, and 100% in 2013 and 2014. There were 50,467 infants born between January 2008 and December 2014 at 22⁺⁰ to 31⁺⁶ weeks’ gestation whose mothers were resident in England and who were admitted to a contributing neonatal unit. Thirty-eight infants were excluded owing to implausible birthweight for gestation. A further 317 observations (0.6%) were excluded because birthweight, sex or multiple birth status was missing, leaving 50,112 infants included in the study. Population characteristics were fairly similar across all 5 years (Table 7), although some differences were statistically significant. There was a slight increase in the proportion of infants born 30⁺⁰ to 31⁺⁶ weeks’ gestation, from 39.6% in 2008 to 42.9% in 2014. The proportion of babies delivered by caesarean section increased every year, from 55% in 2008 to 59% in 2014, but this outcome was missing for around 9% of infants. Infants admitted to neonatal units tend to be from more deprived areas than the general population, and this became more marked over the study period: the 20% most deprived LSOAs contribute > 30% of the study population (increasing from 29% to 33% over the period), whereas the 20% least deprived LSOAs contribute only 13% (decreasing from 14% to 12%).

Predictive model

Parameter estimates from the logistic regression model are shown in Appendix 1, Table 56. Gestational age was modelled with a five-knot spline and birthweight was modelled as birthweight and birthweight2 with interactions between linear birthweight with all gestational age terms, and birthweight2 with linear gestational age included. Appendix 2, Figures 28–35, show isosurv plots for survival prediction. After correcting for optimism, the AUC was 0.84 and the Brier score was 0.07. The model was well calibrated for both gestational subgroups, with optimism-corrected slopes of 1.01 and 0.97, and intercepts of 0.01 and –0.1 for infants born at < 28⁺⁰ weeks’ and ≥ 28⁺⁰ weeks’ gestation, respectively. Table 8 shows the results comparing performance with comparator models. The NNRD model was better calibrated than the other models, with calibration intercepts and slopes closer to the target values of 0 and 1, respectively. The AUC was higher than the National Institute of Child Health and Human Development (NICHD) model, but still fairly low at 0.70. There were no other differences in the AUC or the Brier score.

Survival to discharge from 2008 to 2014

Of the 48,422 admitted infants for whom outcomes were known, 43,444 (89.7%) survived to discharge. There was no evidence of autocorrelation in any analyses (no change when altering the autocorrelation parameter), so results are presented without autocorrelation. There was an increase in the percentage of admitted infants who survived to discharge from 88% in 2008 to 91.3% in 2014. Survival increased with gestational age, from 35% for 22⁺⁰ to 23⁺⁶ weeks’ gestation to 98% for 30⁺⁰ to 31⁺⁶ weeks’ gestation. Appendix 1, Table 57, shows the associations between survival and infant characteristics for the whole cohort, based on complete data only. Crude survival rates were lower for boys, infants whose mothers did not receive antenatal steroids and infants born by vaginal delivery. Infants born to younger mothers, mothers who smoked and mothers from more deprived areas had lower crude survival rates.

The annual percentage change (APC) for crude survival was 0.51% (95% CI 0.35% to 0.67%; p < 0.001), and 0.46% (95% CI 0.30% to 0.62%; p < 0.001) after direct standardisation for risk of death. Results were similar when the only neonatal networks where all hospitals contributed data for the whole period were examined [crude APC 0.56% (95% CI 0.35% to 0.77%); adjusted APC 0.53% (95% CI 0.33% to 0.73%)]. Sensitivity analysis of complete-case data and standardising for 5 and 15 categories gave very similar results.

Survival to 28 days

Fifty deaths were established by linkage with ONS, of which 20 were within 28 days of birth. There was an increase in the percentage of infants who survived to 28 days, from 91.4% in 2008 to 93.5% in 2014. Survival increased with gestational age from 48.4% for 22⁺⁰ to 23⁺⁶ weeks’ gestation to 98.2% for 30⁺⁰ to 31⁺⁶ weeks’ gestation. The APC for crude 28-day survival was 0.3% (95% CI 0.15% to 0.45%; p < 0.001), and 0.27% (95% CI 0.11% to 0.44%; p = 0.002) after direct standardisation for risk of death. Results were similar when only the neonatal networks in which all hospitals contributed data for the whole period were examined (crude APC 0.35%, 95% CI 0.19% to 0.52%; adjusted APC 0.3%; 95% CI 0.14% to 0.47%).

Time of death

A total of 24% of deaths occurred within 24 hours, 28% between 25 hours and 7 days, 26% between 8 days and 28 days and 23% beyond 28 days. There was no evidence of a change in the median and 25th percentile time of death, whereas the 75th percentile reduced from 2008 (27.2 days) to 2013 (20.8 days), but rose to 24.3 days in 2014 (estimated annual decrease 2008–14: 0.92 days, 95% CI 0.2 to 1.7 days; p = 0.02).

Trends in survival to discharge by gestational age

Figure 10 shows the joinpoint regression analysis for survival to discharge by gestational age group. Improvements were less marked with increasing gestation, ranging from an APC of 6.03% (95% CI 2.47% to 3.53%; p = 0.002) in infants born 22⁺⁰ to 23⁺⁶ weeks’ gestation to no change in infants born 30⁺⁰ to 31⁺⁶ weeks’ gestation (APC 0.01%, 95% CI –0.08% to 0.09%; p = 0.9).

FIGURE 10.

Joinpoint regression analysis for crude rates of survival to discharge for admitted infants born at (upper plot) 22⁺⁰ to 25⁺⁶ weeks’ gestation and (lower plot) 26⁺⁰ to 31⁺⁶ weeks’ gestation by birth year. APC, average percentage change. NNU, neonatal unit.

Variation by region and Index of Multiple Deprivation quintile using data from 2011 onwards

Crude survival varied from 89.3% (95% CI 88.6% to 89.9%) in the Midlands and the East of England to 91.1% (95% CI 90.3% to 91.8%) in London; after direct standardisation, the range was 89.2% (95% CI 87.3% to 91.1%) to 91.6% (95% CI 89.1% to 94.2%). Only London and the South of England showed improvements in crude survival (Figure 11).

FIGURE 11.

Joinpoint regression analysis for crude rates of survival to discharge for admitted infants born in 2011–14 at 22–31 weeks’ gestation by NHS commissioning region. A, adjusted survival (%) over years 2011–14; C, crude survival (%) over years 2011–14. NNU, neonatal unit.

Infants from the most deprived quintile had lower survival rates than those from the least deprived quintile [89.5% (95% CI 88.9% to 90.1%) versus 91.1% (95% CI 90.2% to 92.1%)]; little difference remained after standardisation [89.8% (95% CI 87.9% to 91.5%) versus 90.1% (95% CI 87.1% to 93.2%)]. Inclusion of IMD decile in the risk adjustment model did not change results for each region, with evidence of residual variation across regions (p < 0.001).

Comparison with Office for National Statistics and EPICure data

The number of deaths before 28 days among admitted infants born 22⁺⁰ to 31⁺⁶ weeks’ gestation recorded in the NNRD for England and Wales in 2012 was 801. This represents 81% (801/989) of the deaths recorded among live births for the same gestation range in England and Wales by the ONS. Most of the discrepancy occurred at earlier gestations: there were seven deaths among infants born 22 weeks’ gestation in the NNRD, compared with 154 in the ONS.

There was no evidence of a change in the rate of improvement since the first EPICure study. Improvements in survival to discharge of infants born at 22⁺⁰ to 25⁺⁶ weeks’ gestation and admitted to neonatal care in 1995 (EPICure),124 2006 (EPICure 2)135 and 2008–14 (NNRD)137 have continued at a similar rate (Figure 12).

FIGURE 12.

Survival to NNU discharge from 1995 to 2015 based on data from EPICure (triangle), EPICure 2 (cross) and NNRD (circles), for infants born at 23 (blue), 24 (green) and 25 (black) weeks’ gestation. NNU, neonatal unit.

Mortality by Operational Delivery Network

The number of infants were born in 2013–14 at ≤ 31⁺⁶ weeks’ gestation and admitted to neonatal care for whom the neonatal network of booking was known was 15,255; 8.9% of those for whom the outcome was known (1327/14,837) died before discharge. The outcome was missing for 2.7% of infants. Infants with a missing outcome tended to be more vulnerable based on other neonatal characteristics. Antenatal steroid entries were missing for 0.5% of infants, and sex and multiple birth status SDSs for < 0.01%. The prediction model fit the data well, giving an area under the receiver operating characteristic (ROC) curve of 0.83 (95% CI 0.82 to 0.84).

The SMRs are shown on funnel plots, both unadjusted (Figure 13) and adjusted (Figure 14), with neonatal networks numbered (Table 9 contains the key). Analysis of complete cases gave very similar results (largest absolute difference in SMR of 0.04).

FIGURE 13.

Funnel plot for unadjusted SMR for babies live-born ≤ 31⁺⁶ weeks’ gestation in 2013–14 and admitted to neonatal care, by neonatal network of booking; control limits show 2 and 3 SDs from the mean after correction for multiple testing, assuming observed deaths follow a Poisson distribution; numbers correspond to neonatal networks in Table 9.

FIGURE 14.

Funnel plot for adjusted SMR for babies live-born ≤ 31⁺⁶ weeks’ gestation in 2013–14 and admitted to neonatal care, by neonatal network of booking; control limits show 2 and 3 SDs from the mean after correction for multiple testing, assuming observed deaths follow a Poisson distribution. Adjusted for gestation, birthweight SDS, sex, antenatal steroids accounting for correlation within multiple birth sets and missing data; numbers correspond to neonatal networks in Table 9.

Survival between 2008 and 2014

We demonstrate that survival of very preterm infants admitted to neonatal units in England improved between 2008 and 2014, with the greatest improvement seen among infants born at the lowest number of weeks of gestation. However, survival did not improve consistently across the four NHS commissioning regions, with London and the South of England performing better than the Midlands, the East of England, and the North. Survival was lower for infants from more deprived areas, but regional differences in survival persisted after adjustment for socioeconomic differences.

A key strength of the study is the data. Over 50,000 very preterm infants were included, representing the vast majority of neonatal unit admissions in the country during the study period. Assurance on the quality and completeness of the data was provided through comparison with ONS data. Furthermore, neonatal units had the opportunity to validate survival outcomes and clinical characteristics for infants in the study population born after 2012 as part of work by the NDAU. The risk adjustment variables used in the study were limited to key, unambiguous clinical characteristics to minimise the risk of incomplete or inaccurate data; < 3% of records had any missing data, and only 38 records had implausible birthweight for gestation. Several steps were taken to limit or investigate potential bias in the analysis. The improvements in survival remained when we examined only neonatal networks contributing data throughout the period, changed the number of categories used in risk adjustment and looked at infants with complete data only.

A limitation is that the population comprises infants admitted to a neonatal unit, thus excluding live-born infants who died before admission; this is because data capture in the EPRs is triggered by neonatal unit admission. However, this is the relevant population for neonatal services, and comparison with ONS data showed that > 80% of known deaths in this gestational age range were captured in the NNRD, with the shortfall at the earliest gestations likely to represent deaths before admission. Furthermore, analysis of live births does not guarantee a consistent population as there is variation across England in whether or not infants born at < 24 weeks’ gestation are registered as live births. We have imputed missing outcomes on the assumption that the missingness presents no additional information beyond the other neonatal characteristics. This may not be so, as some missing outcomes might reflect infants who were transferred to specialist surgical providers that are not standard neonatal units and do not contribute data to the NNRD. However, no patterns were seen (e.g. with gestational age) for infants according to the reason for transfer, and the proportion of infants with missing outcomes was small.

Improvements in survival of very preterm infants have been demonstrated in other countries, but most examine survival of live-born infants rather than neonatal unit admissions. A population study from New South Wales and the Australian Capital Territory showed improved survival of admitted babies at 24 (50–60%), 25 (60–75%) and 26 (80–85%) weeks’ gestation between 2000–1 and 2007–11, which is similar to results seen here. 142

Mortality by Operational Delivery Network

We combined 2 years’ data to provide improved power to detect significant deviation from average national performance. The average number of expected deaths in a network is 60. If a neonatal network with a patient case-mix leading to 60 expected deaths had a true underlying SMR of 1.3, the probability of the network’s observed data falling above the 2 SD upper control limit is around 60% (i.e. there would be 60% power to alert the network as having potentially unusual performance). This is before widening the limits to allow for multiple testing, which reduces power further.

Note that if the SMR for a neonatal network lies outside the funnel, it will not necessarily have a CI that excludes 1. This is because they have different interpretations: the CI reflects uncertainty about the true SMR for that particular network, whereas the funnel plot limits reflect the variability we would expect to see in the SMR for similar neonatal networks. More specifically, the CI is the range in which we are 95% confident that the true SMR for the neonatal network lies, whereas the funnel plot limits represent the range in which we expect 95% of neonatal networks to lie.

Data

Neonatal EPR data obtained from the NNRD were compared with those recorded independently on trial CRFs and held in the PiPS database. Items for comparison were selected either because the definitions were identical or because data in the NNRD could be used to derive the item as defined for PiPS trial requirements. Variables were selected to represent a broad range of patient characteristics, processes and outcome measures.

The 15 baseline patient data items comprised (1) expected date of delivery, (2) gestational age (weeks and days), (3) month and year of birth, (4) birthweight (g), (5) sex, (6) Apgar score at 5 minutes, (7) whether inborn or transferred, (8) singleton or multiple, (9) birth order, (10) maternal year of birth, (11) maternal ethnicity by NHS category, (12) LSOAS, as derived from postcode, (13) any antenatal corticosteroid given, (14) mode of delivery (vaginal vs. caesarean) and (15) instrumental delivery (Table 10).

The 13 processes or interventions during admission were (1) surgery for PDA, (2) medical treatment of PDA, (3) retinopathy of prematurity (ROP) treatment by laser or cryotherapy, (4) central venous line days, (5) intensive care days, (6) high-dependency care days, (7) whether or not transferred to another hospital, (8) discharge month and year (Table 11) and, in the first 14 days, (9) day of first milk feed, (10) type or types of milk (maternal milk, donor milk or formula) given on the first day of feeding, (11) a summary of all types of milk received over all of the days on which feeds were reported on both databases, (12) duration of exposure to any antibiotic or (13) duration of exposure to any antacid (Table 12).

The nine outcome data items were (1) worst stage of ROP in either eye, (2) bronchopulmonary dysplasia (BPD), defined by whether or not the infant required supplementary oxygen at 36 weeks’ postmenstrual age, (3) mechanical respiratory support at 36 weeks’ postmenstrual age, (4) cranial ultrasound findings, (5) survival to discharge, (6) any diagnosis of perforated NEC, (7) any abdominal surgery for NEC, (8) any gastrointestinal perforation and (9) length of stay (Table 13).

Changes to the original protocol

When this study was designed it was anticipated that recruitment to the PiPS trial would begin in 2009. The analysis could be conducted only once PiPS data for the individual infant were complete and any queries had been addressed. Although it was always appreciated that the order in which the data for trial recruits were signed off would not be sequential, it was nonetheless expected that it would be possible to receive data in batches every few months. The original protocol involved comparison of PiPS and NNRD data from the first 200 infants recruited, as well as feedback of rates of minor and major discrepancies to neonatal units, with the final analysis undertaken on the next 200 infants. There were delays to the start of PiPS recruitment, which did not begin until July 2010 and which did not achieve its target rates for over 1 year. There were further delays to PiPS data being signed off as complete because of problems with introducing a new automated data query system. It became clear that we could not provide feedback to neonatal unit staff after 200 cases and that we were likely to receive any PiPS data at the NDAU only towards the end of PiPS trial recruitment. It was agreed within the MfN Steering Committee that we would request an initial download of all available complete data for piloting the database merger and would then perform the comparative analysis on the final PiPS data set, including all recruits, and this was received in October 2013. There is a constant process of data scrutiny and feedback aimed at improving data completeness and accuracy in the NNRD. It was therefore agreed that we should examine the whole data set for changes in discrepancy rates over time, in order to address the second aim listed above.

Preparation of data for comparison

Data acquisition differs between the PiPS trial and the NNRD, the former being recorded specifically for trial purposes and the latter containing data extracted from point-of-care EPRs, designed to provide a complete clinical record. In this section, we will describe how the data sets were prepared to ensure that the comparisons related to the same infant and the same episode of care.

Neonatal units in NHS hospitals in England function as clinical networks. Neonatal units vary in the care that they provide, in that some provide intensive care only in an emergency, whereas others that do provide ongoing intensive care might not do so for extremely preterm infants. A small number of neonatal units provide specialist services, such as surgery and cardiology. In so far as is possible, infants are looked after in the neonatal unit closest to the family home. Those infants needing specialist care may require transfer to a neonatal unit offering the required expertise; hence the entire course between birth and eventual discharge home may comprise a series of ‘episodes of care’ of varying durations in different neonatal units. The variables that were compared for each infant between the PiPS database and the NNRD comprise ‘once only’ data (e.g. baseline demographic items such as birthweight), ‘episodic’ data (e.g. processes and interventions during a defined episode of care) and ‘infant-level’ data (e.g. outcomes summarised from multiple episodes of care).

Infants born between 23⁺⁰ and 30⁺⁶ weeks’ gestational age and who were < 48 hours old were eligible for recruitment to the PiPS trial. Infants with a lethal congenital anomaly or any known gastrointestinal malformation known at birth, or with no realistic chance of survival, were excluded.

The CRF data collection was paper based, with four main collection forms: form 1 (entry), form 2 (daily data), form 3 (transfer/discharge) and form 4 (abdominal pathology). Form 1 (entry) is completed within 7 days of recruitment and returned to the National Perinatal Epidemiology Unit (NPEU), and it contains ‘once only’ information regarding the infant and maternal history [e.g. infant sex, expected date of delivery (EDD), maternal and obstetric details and infant’s condition at birth). Form 2 (daily data) requires daily recording for the first 14 postnatal days from the day of birth of the type of milk received, the total daily volume of milk (ml/kg/day), the antibiotics by type, the antifungals and the antacids. Form 3 (transfer/discharge) is completed for each episode of care, terminating at discharge (whether to another hospital or home or at death). It contains only events occurring during that episode (e.g. diagnoses, procedures and treatments received, including, if the admission covers 36 weeks’ postmenstrual age, whether or not the baby is still receiving supplementary oxygen or mechanical ventilatory support). Form 4 (abdominal pathology) is completed for any episode of proven or suspected abdominal pathology including NEC, for which the severity is staged using modified Bell criteria.

On receipt at the PiPS trial office at the NPEU, validation included a series of range, logic and missing data checks to identify inconsistencies within and across forms for the same baby. Some queries were resolved in-house according to predefined protocols; those that could not be resolved were reconciled between the staff in the trial office, the PiPS trial research nurses, the chief investigator and principal investigators with reference to the clinical notes, and documented accordingly. Data were double-entered onto a dedicated trial database.

The NNRD is organised into different files, including static ‘once only’ data, ‘episodic’ data for each admission to a different neonatal unit, ‘daily data’ recorded on a daily basis, and ‘if and only’ data recorded only if applicable (e.g. ad hoc abdominal X-ray forms, blood cultures).

Episode numbering and matching

Episode number 1 on the PiPS database is the admission to the neonatal unit where the infant was recruited to the trial [Form 1 (entry)] and may not be where the infant was born. In contrast, the first episode on the NNRD is always at the hospital of birth (i.e. episode 1 for PiPS may be episode 2 for the NNRD if a baby was transferred from the hospital of birth to a second hospital where PiPS recruitment took place).

It emerged that during processes of addressing missing data and queries, some PiPS episodes had been renumbered and did not appear on the database sequentially. An additional problem arose because data for PiPS were recorded for all episodes, including those spent on paediatric wards and in specialist surgical or cardiac centres that do not use the neonatal EPRs and, thus, these episodes were not present in the NNRD. Consequently, it was necessary to check the dates, anonymised patient ID and hospital name of each episode on each database and renumber, when necessary, to ensure that comparisons were indeed for the same episode. This was further complicated because of inconsistencies of the names of hospitals and NHS trusts entered as free text on the PiPS database. By contrast, the names of hospitals and NHS trusts are standardised in the EPRs through the use of drop-down menus.

Sources of items within the databases

Although many of the data points recorded for the PiPS trial are also present in the NNRD, there are differences in how some data were obtained. In general, data for PiPS were obtained by asking a direct question (e.g. ‘during this episode of care did the infant have surgical ligation for a PDA?’); by contrast, information on whether or not an infant has surgery for a PDA can be entered into the NNRD by a range of routes including variables in the ‘daily data’, ‘discharge diagnoses’ and ‘procedures’ EPR fields.

Preparation of data sets for linkage

We compared data items only for episodes present in both the PiPS and the NNRD databases. We excluded episodes that could not be linked.

Methods of comparison

Statistical methods for assessing agreement

We assessed items that were identical, and items with minor and major discordance, using predefined criteria based on clinical judgement and set a priori by KC and CB to mitigate bias.

We calculated for each variable of interest the proportion of babies for whom the NNRD and PiPS trial data differed and the 95% CIs for the proportion. For variables with discordance rates of < 5%, we used the Poisson approximation to the binomial to calculate CIs; otherwise we used the Agresti and Coull method145 for binomial CIs, as this method has better coverage properties. 146 As observations from the same hospital are not independent, we calculated the CIs for discordance using generalised linear models with variances estimated to allow for within-hospital correlation. 147 To test whether or not the discordance rate had changed over the course of recruitment, we calculated discordance rates for sequential quarters for five key variables (i.e. antenatal steroids, mode of delivery, day of first milk feed, type of first milk and central line days) and tested a time trend using linear regression with weights, to allow for a varying number of observations at each time point and adjusting for clustering by hospital. We assessed autocorrelation using residual plots and the Breusch–Godfrey test and accounted for this using the Prais–Winsten procedure, if necessary. To investigate whether or not discordance varied by recruitment site, we calculated discordance rates separately for each hospital for five key variables (i.e. antenatal steroids, mode of delivery, birthweight, EDD and central line days).

To check case ascertainment for binary clinical outcomes, we calculated sensitivity and specificity, treating PiPS data as the gold standard. For continuous variables, we calculated mean and median differences and 95% limits of agreement for the differences.

Sensitivity, specificity and positive predictive values

For binary outcome variables, we calculated the sensitivity, specificity and PPVs of NNRD data in comparison with PiPS data.

In the context of this data comparison, we have taken the prevalence of an outcome as the proportion of infants on the PiPS database with that outcome reported. The following definitions have been used:

• Sensitivity is the ability of the NNRD database to correctly classify an individual as ‘diseased’ as indicated by the gold standard PiPS database.

• Specificity is the ability of the NNRD database to correctly classify an individual as disease free as indicated on the PiPS database.

• Positive predictive value is the percentage of individuals who are identified on the NNRD as having the disease who actually do have the disease as indicated on PiPS database.

Sensitivity and specificity are characteristics of the test and are, in contrast to the PPV, unaffected by the prevalence of the outcome.

Regulatory issues

The establishment of the NNRD and the PiPS trial had each been approved by a Research Ethics Committee (REC) (10/H0803/151 and 09/H0604/30 respectively; patient information leaflet is provided in Appendix 5). Advice was sought from the REC chairperson regarding whether or not an additional application either as a stand-alone project or as an amendment to the existing approval for the PiPS trial was required before undertaking this analysis. We were advised that no such application was necessary.

A data-sharing agreement was then put in place between the NPEU, University of Oxford, where the PiPS trial data are held, and Imperial College London for the transfer of PiPS trial data. These data were stripped of identifiers, other than the Badger ID, and were sent to the NDAU. Database merging and all subsequent analyses took place at the NDAU.

Linkage

A total of 1315 babies were recruited into the PiPS trial; the parents of five babies withdrew consent including consent for the use of any data and, therefore, data for 1310 babies were available for analysis. Clevermed Ltd was able to provide Badger ID for 1280 (98%) infants [no EPR data could be identified for 30 (2%) recruits into the PiPS trial]. This resulted in data for 2360 episodes of care being available on the PiPS database (Figure 15).

FIGURE 15.

Records from PiPS CRF and the NNRD.

Of the second episodes on the NNRD, 81 were the first episode on PiPS because the baby was recruited at a different hospital from that of birth. There were 103 episodes on PiPS from 22 infants who could not be reliably matched on the NNRD, because they were in a ward or a hospital that was not entering data onto BadgerNet, because of duplicate reports of the same episode on the NNRD or, in the case of two infants, because they could not be found in the NNRD. All data for these 22 infants were excluded from the analyses, leaving 2257 episodes of care from 1258 infants eligible for comparison (see Figure 15).

Infant and maternal characteristics

We compared baseline infant and maternal baseline characteristics for 1258 infants. The numbers of infants with missing data for each variable in both databases are reported along with the minor and major discordances calculated using the predefined criteria (Table 14). Gestational age on the PiPS database is calculated from the EDD, whereas gestational age in weeks and days is recorded directly on the EPR and, hence, the NNRD.

Missing data on the PiPS database were few (< 3%): EDD was missing for 9.2% of infants in the NNRD, LSOA for 12.1% and maternal ethnicity for 5.1%. Rates of ‘any discordance’, defined as ±3 days, were 9.9% and 20.4% for EDD and gestational age, respectively. Major discordance rates were < 10% for all variables except for maternal ethnicity (10.2%) and maternal LSOA (16.5%).

Processes

We were able to compare 2257 linked episodes for 1258 infants (Table 15) using the predefined criteria. Major discordances, defined as ± 5 days, were 10.2% and 11.2% for the duration of high-dependency care and central venous lines, respectively. Discordance for medical treatment of a PDA was 6.0%. For all other variables, discordancies were < 5%.

Sensitivity and specificity

We report sensitivity, specificity and PPVs for NNRD data in Table 18. In the conventional context, the sensitivity would be the probability of a test that correctly identifies an individual with a disease as ‘diseased’; in this context, it is the probability of being ‘NNRD disease positive’ when disease is present. Sensitivity for outcomes other than survival to discharge, which is 100%, ranges between 50% and 87%. There is a 50–87% chance that infants with the disease are ‘NNRD disease positive’. Therefore, there is under-reporting of disease in the NNRD. Specificity was > 85% for all outcomes, with the majority being > 90%. Infants without the disease have a high probability of being ‘NNRD disease negative’, NNRD correctly identifying infants without disease. With the exception of BPD and medical treatment for PDA, which have a prevalence of 49.0% and 20.3% respectively, the prevalence of these adverse outcomes is low, at < 6%.

The PPV of all outcomes with the exception of treated ROP (71.4, 95% CI 56.7 to 83.4), perforated NEC (66.0, 95% CI 51.2 to 78.8) and a range of details of cerebral ultrasound scans was > 75.

By hospital analysis

We compared major discordance rates for five variables (i.e. antenatal steroids, mode of delivery, birthweight, EDD and days with a central line) across the 24 PiPS recruiting hospitals (Table 19). The discordance rates of mode of delivery and central line days, which in general have higher major discordance, show striking variation at different hospitals, with mode of delivery varying from 0.0% to 18.5% and central line days from 2.7% to 28.6%.

Trends over time

There were no significant changes in discordance rates over time for any of the five selected variables: any discordance in antenatal steroids (–0.5%, 95% CI –1.4% to 0.4%; p = 0.27) or mode of delivery (–0.5%, 95% CI –2.9% to 1.8%; p = 0.64), major discordance in day of first milk (–0.2%, 95% CI –2.0% to 1.5%; p = 0.78), major discordance in type of milk on first day (0.8%, 95% CI –5.2% to 6.8%; p = 0.78) and central line days (–1.2%, 95% CI –6.2% to 3.8%; p = 0.64).

Overview

Around 6000–7000 children who were born very preterm (< 32 weeks’ gestation) are admitted to NHS neonatal units each year. They are at substantial risk of adverse neurodevelopmental outcomes. Severe disability rates of between 5% and 56% are reported155 and long-term studies show that the adverse consequences of preterm birth are still apparent in adolescence and adulthood. 156,157

Information on the later neurodevelopmental outcomes of preterm infants is necessary for several reasons. For an individual child, outcome assessment is needed to ensure that disability, when present, is identified and timely intervention is provided. Professionals require up-to-date outcome information to counsel, advise and support parents. Neonatal unit and population-based outcome data are essential for service planning, benchmarking and evaluation of the impact of neonatal specialised services and their cost. Neurodevelopment is also a common outcome measure in epidemiological, observational and clinical research.

Most neonatal services attempt to provide follow-up assessments up to around 2 years of age. Ceasing systematic assessment at an earlier age would risk confounding by transient neurological dystonia, which mimics cerebral palsy but which improves or resolves completely during the first year. 158 The literature suggests that a reliable early diagnosis of moderate to severe cerebral palsy can be made by 18 months corrected age, and a reliable early diagnosis of mild cerebral palsy can be made by 24 months corrected age. 159 Although assessment tools, such as the Bayley Scales of Infant Development (BSID), provide standardised mental (cognitive) scores from as early as 12 months of age, the correlation of the early mental scores with subsequent IQ at school age is unclear. Two recent cohort studies reported moderate to substantial agreement between BSID, second edition (BSID-II), Mental Development Index (MDI) at the age of 2 years and full-scale IQ at the age of 5 years among infants born before 30 weeks’ gestation or with very low birthweight (VLBW) (i.e. birthweight of < 1500 g). 160,161 Conversely, Hack et al. 162 described a considerable reduction in the proportions of extremely low-birthweight infants (i.e. birthweight of < 1000 g) who were diagnosed with cognitive impairment (defined as standardised cognitive scores < 70), from 39% at 20 months to 16% at 8 years of age, when the children were tested sequentially. Applying the same diagnostic criteria, Roberts et al. 163 also found a reduction in the proportions of very preterm (< 27 weeks’ gestation) and extremely low-birthweight infants with cognitive impairment, from 27.3% at the age of 2 years to 19.3% at the age of 8 years.

All follow-up programmes, whether for clinical or research purposes, incur significant costs related to the employment of trained staff, interim assessments, long-term tracking, data management and analysis, and the need for financial and logistic support to be sustained long term. This is often the main constraint on maintaining follow-up assessments. In the UK, there are currently no nationally agreed, implemented and funded policies for very preterm follow-up.

Types of neurodevelopmental outcome measures

Cerebral palsy is the most commonly quoted outcome in neonatal follow-up studies. It is an umbrella term used to describe a group of non-progressive permanent disorders of movement and posture that occur following damage to the developing fetal or infant brain. It is most commonly described based on the nature of the neurological abnormality (e.g. spastic, dyskinetic or dystonic) and the topography of limb involvement.

Even in the absence of cerebral palsy, preterm infants experience abnormal patterns of motor development and neuromotor dysfunction. 164 Several authors have described the presence of transient dystonia, which may mimic cerebral palsy, in the first year of life in almost one-third of VLBW cohorts. 158,165 A meta-analysis of studies of children who were born very preterm (≤ 32 weeks’ gestation) reported motor scores of between 0.57 and 0.88 SDs behind their term-born peers. 166

The most common disability among preterm children is developmental or cognitive delay. 167,168 Cognitive ability can be described using developmental quotients (DQs) or intelligence quotients (IQs) derived through standardised developmental or intelligence tests. Conventionally, a standardised DQ or IQ > 2 SDs below the population mean is used to define impairment or disability, as it represents the lowest functioning 2.3% of the population. The prevalence of developmental or cognitive impairment exists as a gradient that is inversely related to gestational age. 169 A population-based comparison of school-age children who were born before 28 weeks’ gestation or with a birthweight of < 1000 g with term-born controls revealed a 0.7 SD reduction in IQ points in the preterm children, after adjusting for sociodemographic factors and exclusion of children with neurosensory impairment. 170 In the EPICure 2 study, which followed infants born before 27 weeks’ gestation in 2006 in England, 35% of survivors assessed at the age of 3 years had cognitive scores (predicted MDI) that were > 1 SD below the normative mean. 28

Preterm infants have delays in receptive language processing,171 expressive language acquisition,172,173 articulation and phonological short-term memory. 172,174,175 A meta-analysis of 12 studies published by Barre et al. 176 in 2011 reported that very preterm (< 32 weeks’ gestation) infants perform between 0.38 and 0.77 SDs below their term-born counterparts in areas of expressive and receptive language. A metaregression of six studies for the difference in language scores between very preterm infants and term-born controls against the age at assessment between 3 and 12 years suggested that the deficit in language function deteriorated with increasing age. 177

Published data in the past 20 years estimate that hearing impairment affects between 1.5% and 9% of infants born very preterm, although < 1% had severe bilateral sensorineural hearing loss uncorrectable with hearing aids. 28,167,178–181

Retinopathy of prematurity resulting from disordered retinal vascular development is a major threat for vision loss in preterm infants and high-risk groups receive regular screening ophthalmic screening examinations. 182 In the UK, ROP affects approximately 17% of infants born very preterm and/or VLBW183 and it accounts for around 3% of all childhood vision loss. 184

There is an increased risk of attention deficit hyperactivity disorder (ADHD) and emotional and social disorders, including ASD, among very preterm/VLBW children, compared with the general population. 185–188 Case–control studies have indicated a twofold to threefold increase in the risk of ADHD in very preterm/VLBW infants, compared with term-born controls. 188 The estimated prevalence of ASD has been reported to be 5% in children with a birthweight of < 2000 g189 and 8% in children who were born at < 26 weeks’ gestation. 190 This represents an approximate tenfold increase over the 2–9 per 1000 prevalence estimate in the general population. 191,192

Nine out of 12 case–control studies published in 1980–2001 and included in a meta-analysis reported an increase in internalising behaviour among the very preterm/VLBW cases at ages 5–12 years; 9 out of 11 studies also reported an increase in externalising behaviour. 169 However, in a more recent meta-analysis based on parents’ and teachers’ ratings, the difference in internalising behaviour scores reported between preterm/VLBW cases and full-term controls was small (preterm cases’ scores were < 0.28 SDs below the scores of the term-born controls), and for externalising behaviour the difference was negligible. 193 The EPICure 1 study reported that extremely preterm children were 3.5 times more likely to have anxiety disorders than their term-born classmate controls. 194

Standardised developmental and neuropsychological tests

Classification of neurodevelopmental outcomes

Neonatal follow-up programmes in the UK

Neonatal follow-up programmes are not universal in the UK. Some neonatal networks have set up regional projects,231 but the cost of setting up and running a follow-up programme, which includes training staff, is considerable. 232 Most neonatal units offer routine clinical follow-up for infants born very preterm, but the proportion that actually receive the assessment is unknown. In addition, the approach to the assessment of neurodevelopment during routine clinical follow-up varies widely. Children may be assessed by a neonatal or community paediatrics consultant, staff grade doctor, associate specialist, trainee doctor, advanced neonatal nurse practitioner or an occupational or developmental therapist.

In the UK, there is an established surveillance programme to monitor the health and development of all children. 198,199 In the 1990s, several studies investigated the extent to which data recorded during routine service delivery can be used to report the outcomes of survivors of neonatal intensive care. 224,233,234 The Trent Neonatal Follow-up Project reported that most of the data required to meet the NPEU/Oxford minimum data set could be extracted from routinely available information systems. 233 However, the quality of the data was variable and there was no standardisation in the interpretation or documentation of clinical assessments. An exercise on data linkage between the neonatal register and the community child health surveillance database produced ‘error-free’ linkage (using the identifiers date of birth, birthweight and gestation) in only 53.9% of children who had received neonatal intensive care. Modi and Carpenter235 reported similar problems when they reviewed the use of district and regional child health database in the North Thames Region to ascertain the 2-year health status of children who were born at < 29 weeks’ gestation. They were able to retrieve child health surveillance records for only 2 out of 80 children surviving to 2 years. When Johnson and King234 used the routine child health information system to compile a list of children with motor or sensory disability, they failed to identify 162 out of 446 (36.3%) children listed on the coexisting population register of cerebral palsy, sensorineural deafness and severe vision loss. Since 1992, several reports have highlighted the need for data collection on the later morbidity of survivors of neonatal intensive care. 236–239 The Audit Commission237 proposed that all neonatal units collect data in a nationally agreed format. The Department of Health and Social Care (DHSC)’s report Changing Childbirth recommended the development of a system of data collection to enable meaningful comparison of perinatal statistics. 239

The British Association of Perinatal Medicine (BAPM) first published standards for hospitals providing neonatal services in 2001, recommending that the later health status of survivors at particular risk of disability should be ascertained up to at least a corrected age of 2 years and standardised guidelines for the definition of disability should be used. 240

Despite these recommendations, routine outcome reporting of health outcomes following preterm births remains largely unavailable. In 2007, the National Audit Office reported that evidence of neonatal outcomes, other than the traditional indicator of mortality rates, was still sparse. 241

The National Institute for Health and Care Excellence (NICE), in 2010, published a list of statements that define high-quality specialist neonatal care. 242 This included evidence of processes to enable collection of health outcome data on babies who receive specialist neonatal care. The NICE guideline for developmental follow-up of preterm infants is also currently being developed and is anticipated to be published in full in 2017. Some neonatal networks have included a target for 2-year assessment of very preterm infants in the CQUIN payment framework to encourage follow-up and data collection.

Parent-completed questionnaires

Parent-completed questionnaires have been developed as a low-cost alternative to developmental tests to identify children with disabilities. The level of agreement between parental perceptions and paediatrician assessments is inconsistent243–245 and may be influenced by parent sociodemographic factors. The validity of the revised Parent Report of Children’s Abilities-Revised (PARCA-R),246,247 the Parent’s Evaluation of Developmental Status (PEDS),248 the Functional Status II (FS-II) questionnaire,249 the Ages and Stages Questionnaire (ASQ)250 and a questionnaire adapted from the Griffiths Developmental Scales251 had been evaluated in the preterm population. In particular, the PARCA-R was found to have good diagnostic utility for moderate to severe cognitive and language impairment when validated against the BSID-II (reported sensitivity 85%; specificity 87%)247 and the Bayley-III (sensitivity 75–94%; specificity 79–89%),252 and had been used for outcome reporting in neonatal studies. 253,254 Although the typical response rates to postal questionnaires were reported to be between 52% and 61%,255 Field et al.,256 when testing parent-completed questionnaires as a source of outcome data at 2 years following neonatal discharge, recorded a 90% response rate by maintaining contact with the families in the form of Christmas and birthday cards.

Electronic patient records

In the UK, most community child health services hold clinical information from child health surveillance programmes on electronic information systems, although these systems vary from one NHS trust to another and the data are not routinely passed back to neonatal units. In the past decade, all neonatal units in the UK have moved towards routinely recording clinical information in an EPR to facilitate shared care within neonatal networks. The BadgerNet platform is most widely used (www.clevermed.com/). In 2007, a standardised format for the recording of 2-year neurodevelopmental and health status, adapted from the NPEU/Oxford classification of disability, was developed by the Thames Regional Perinatal Group Outcomes Group. This was incorporated into the EPR in 2008. Since 2009, the National Neonatal Audit Programme, delivered by the Royal College of Paediatrics and Child Health, has been using data held in the NNRD for audit purposes, including 2-year health status of children who were born at < 30 weeks’ gestation. The programme has promoted outcome data recording, with an increase in the number of participating neonatal units documenting any 2-year outcome data on the eligible infants from 51 out of 170 units (30%) in 2009 to 158 out of 179 units (88%) in 2013.

Approvals and registration

Approval was received from the National Research Ethics Committee (REC 10/H0720/35) and the NHS Research and Development Department of each study site. The study was adopted onto the UK Clinical Research Network Portfolio (ID 8626). The systematic review meta-analysis was registered on PROSPERO, an international prospective register of systematic reviews (CRD42012002168).

Study sites

Study sites were selected to provide representation of infants from a wide range of ethnic and socioeconomic backgrounds as well as include neonatal units where clinicians of different grades and specialties provide follow-up assessments of preterm infants. Study sites were restricted to within Greater London and Cambridge for practical reasons. The lead consultant responsible for post-discharge follow-up at each hospital was invited to be the local collaborator for the study. No hospital declined participation.

Participants

Eligible participants were children who were born at < 30 weeks’ gestation and attended the routine NHS follow-up assessments at participating hospital sites between the corrected ages of 20 and 28 months (age adjusted for prematurity) during the recruitment period (June 2010 to July 2012).

To prevent ‘practice effect’ bias from repeated testing, children who had received Bayley-III assessment, either as part of their routine NHS assessment or owing to enrolment in other research studies, were excluded. Children from non-English-speaking families were also excluded because the Bayley-III neurodevelopmental assessment was developed to be administered in English and parents would not be able to complete the Q-CHAT and Bayley-III Social-Emotional questionnaires independently.

Recruitment

Local collaborators (i.e. clinical consultants) identified eligible participants and approached parents, who were sent the study information sheet and a letter of invitation to participate (see Appendix 5). If they were interested in participating or wished to discuss the study, they were asked to provide their contact details on a pre-printed response form and send it to the researcher in a pre-paid envelope. Alternatively, parents were given study information at the time of their child’s NHS follow-up appointment.

Researcher training

The researcher received training on Bayley-III assessment techniques through attendance at a 2-day training workshop, followed by practice sessions that were supervised by Bayley-III expert trainers. Techniques were accredited through a pilot assessment that was independently scored. A score of 100% agreement on all items on the assessment scales was achieved. To ensure reliability and consistency during the study, validation sessions were attended by an observer, who scored assessments administered by the researcher on non-study participants who were born at < 30 weeks’ gestation and were 20–28 months old (corrected age). The interobserver agreement between scores was evaluated and the researcher received feedback. The researcher also received training in the standardised neurological examination based on the HINE209 from an expert trainer.

The research assessment

At the time of assessing the participant, the researcher was blinded to the results from the NHS assessment. The assessments took place in an outpatient clinic room at the same site as the routine NHS assessments. Each participant was accompanied by one or both parent(s). For the Bayley-III assessment, the participant was seated either at a children’s table or on his/her parent’s lap at the office desk. The test items were administered sitting across the table facing the participant. In the case of twins or triplets, one child was assessed at a time.

Timing of research assessment

The intention was to complete the research assessment within 1 month before or after the participant’s NHS follow-up assessment. To minimise potential information bias caused by changes in development during the interval between the NHS and the research assessments, the intention was to administer the research assessment before the routine assessment in approximately half the cohort and after in the other half.

Assessment of cognition, language and neuromotor development

Participants’ cognitive, language and motor development were assessed using the Bayley-III. 201 Each test item was scored as 1 (pass) or 0 (fail). If the participant refused to respond to any test item, it was scored as ‘failed’. For each scale, the sum of the scores for all items tested between the basal and ceiling levels constitute the participant’s raw score. Two types of norm-referenced scores were obtained: scaled scores, which are standardised to a mean of 10 and SD of 3; and composite scores, which have a mean of 100 and SD of 15. For the cognitive scale, both the scaled score and the composite score were derived from the raw score. The language composite score was derived from the sum of the receptive communication and expressive communication scaled scores. Similarly, the motor composite score was obtained from the sum of the fine motor and gross motor scaled scores. The algorithm developed by Moore et al. 204 was used to convert the Bayley-III cognitive and language scores into a predicted BSID-II MDI, for the purpose of comparing the classification of neurodevelopmental outcomes into categories of severity based on the two scores. The algorithm is:

On the Bayley-III Social-Emotional questionnaire, parents were asked to rate how often their child demonstrated certain behaviours. Scores for each item were allocated according to behaviour frequency as follows: all of the time (5 points), most of the time (4 points), half of the time (3 points), some of the time (2 points), none of the time (1 point) and can’t tell (0 point). A score of 0 (equivalent to ‘can’t tell’) was given to questions with incomplete responses; if more than one response was given, the response with the highest score was used.

Assessment for neurological deficits and cerebral palsy

The tools utilised were the standardised neurological examination, based on the HINE,209 and the ‘extremely low gestational age newborn’ algorithm, as a structured guide to diagnose and classify cerebral palsy into topography-based categories of quadriparesis (at least three-limb involvement), diparesis (involvement of one or both lower limbs) and hemiparesis (involvement of one side of the body). Functional severity of cerebral palsy was classified into five levels based on the GMFCS;226,228,257 social communication abilities were judged using the parent-completed Q-CHAT220 and Bayley-III Social-Emotional201 questionnaires; parents were sent the questionnaires prior to the appointment; the Q-CHAT consisting of 25 items used a 5-point Likert scale (0–4 points) and scores were allocated according to the methods described by the research team that developed it. 220 Questionnaires with more than six incomplete responses were excluded. The scores from all items were summed to obtain a total Q-CHAT score within a possible range of 0–100.

Record of observed behaviour during the research assessment

The Behavioural Observation Inventory included in the standard Bayley-III was used to record behaviour observed during the assessment. Thirteen types of behaviour were noted: positive affect, enthusiasm, exploration, ease of engagement, co-operativeness, appropriate activity level, adaptability to change, alertness, distractibility, appropriate motor tone, tactile defensiveness, fear or anxiety, and negative affect. Numerical scores were assigned for each behaviour: a score of 2 was given if the behaviour was ‘observed most of the time’, a score of 1 was given if it was ‘observed some of the time’ and a score of 0 was given if it was ‘never or rarely observed’. The presence of ‘distractibility’, ‘tactile defensiveness’, ‘fear/anxiety’ and ‘negative affect’ were reverse-scored. Using the same form, the parent(s) or caregiver accompanying the participant was asked to rate how much the child’s behaviour during the assessment was representative of his/her usual conduct. A score of 2 points was given for ‘very typical (child is like this most of the time)’, a score of 1 was given for ‘somewhat typical’ and a score of 0 was given for ‘not at all typical’. Hence, two behavioural rating scores (each with maximum score of 26) were obtained; an examiner rated the behavioural score for the frequency of positive behaviour and a parent rated the score for the typicality of behaviour.

Classification of impairment from the research assessment

Participants were classified into categories of neurodevelopmental status using two methods:

1. SD score groups ‘higher than –1 SD’, ‘–1 to –2 SDs’ and ‘lower than –2 SDs’ based on their Bayley-III scores.

2. ‘No’, ‘mild–moderate’ and ‘severe’ impairment groups according to the modified NPEU/Oxford criteria.

Classification of impairment

The Bayley-III composite score was used to assign the SD score group in the cognitive domain. In the language and motor domains, composite scores were derived from combining scaled scores from the receptive and expressive communication subtests and the fine motor and gross motor subtests, respectively. Therefore, if a child had a specific impairment in only one subtest, it is possible for compensation from the other subtest to occur, resulting in a composite score within the normal range. Hence, the scaled score was used to identify specific impairment in the subdomains of receptive communication, expressive communication, fine motor skills and gross motor skills. In the combined language and motor domains, impairment was taken as the worst category of outcome assigned in the respective subdomains and based on the Bayley-III composite scores. The overall outcome of each participant was based on the worst category of impairment from the cognitive, language and motor domains. Participants who received Bayley-III scores of lower than –1 SD were considered to have at least a mild form of impairment and scores of lower than –2 SDs were considered to represent at least moderate to severe impairment.

Impairments were also classified according to the modified NPEU/Oxford criteria.

Outcome data from NHS follow-up assessments

Participants were assessed by their local clinicians as part of their routine NHS post-discharge follow-up. Assessors were blinded to the results of the research assessment. Results were entered into the electronic ‘2-year outcome’ form on the BadgerNet EPR as required for the National Neonatal Audit Programme. The specific questions for the development (cognitive), communication and motor domains were as shown in Box 1.

A positive response to any of the questions implied the presence of impairment. Questions D3, RC2, EC3, FM3, GM3 and GM5 denote the criteria for severe impairment. Additional information on whether or not the child was diagnosed with cerebral palsy, whether or not a standardised neurodevelopment test was used during the NHS assessment and whether or not the child was difficult to assess were also entered. The electronic form could be completed by the examining health professional or by administrators, such as secretaries or data entry clerks, based on the information given to them by the examiner.

With parental consent, the participants’ unique identifier on the NNRD (the Badger ID) was obtained from the local collaborator at each study site. Neonatal and 2-year outcome data were then obtained from the NNRD with assistance from the NDAU data managers.

Classification of disability based on National Neonatal Research Database data

Participants were classified into categories of ‘no’, ‘mild–moderate’ and ‘severe’ impairment within each outcome domain (i.e. cognitive, receptive communication, expressive communication, fine motor and gross motor) using the electronic 2-year outcome data and according to the algorithm outlined in Figure 16. A missing response did not count as a ‘no’; therefore, complete data entry is required to assign participants as having no impairment. An overall level of impairment was defined based on the worst outcome from the five domains.

FIGURE 16.

Algorithm for the classification of impairment using data from NHS assessments.

In addition, for the purpose of assessing selection bias, the following data were extracted from the NNRD for all infants born between 1 January 2008 and 31 December 2010 at gestational ages of < 30 weeks and discharged from the participating study sites (the ‘baseline population’): gestation at birth, birthweight, sex, ethnicity, singleton or multiple pregnancy, mode of delivery, days of mechanical ventilation, oxygen therapy at 36 weeks’ corrected gestational age, maternal age and the IMD based on maternal residence at the time of birth. The IMD is a summary measure of relative area deprivation, calculated through a weighted combination of scores from 38 different indicators covering factors, such as income, employment, education, health, living environment and crime, for each area in England, using national census data. The IMD was obtained based on the postcode of the mother at the time of birth of her child and according to the English Indices of Deprivation 2010. 258

Statistical tests

Data were coded for analysis using Microsoft Excel^® 2007 (Microsoft Corporation, Redmond, WA, USA). Data were double-entered, examined and outliers were verified. All analyses were performed using Stata statistical package, version 11.0.

Quantitative variables are presented as means and SDs for normally distributed data, or medians and IQRs when the distribution was skewed. Qualitative variables are presented as numbers of subjects and percentages. Differences between categorical variables were analysed using Pearson’s chi-squared test. For continuous variables, Student’s t-test was used for parametric comparison and the Mann–Whitney U-test was used for non-parametric comparison. The p-values derived from statistical tests are presented and the conventional 5% level is used to define statistical significance. Several key statistical measures used in the analyses are described below.

The validity of an assessment, in the context of this research, refers to the ability of the assessment to differentiate accurately between children with and without neurodevelopmental impairment, as defined. It is described using sensitivity and specificity, which are derived through a 2 × 2 table (Table 20).

Sensitivity is the proportion of children with impairment who were accurately identified as having impairment from the assessment under evaluation. It is calculated as:

Specificity is the proportion of children without impairment who were accurately identified by the assessment under evaluation and is calculated with the formula:

Sensitivity and specificity calculations are expressed as either proportions or percentages with corresponding 95% CIs. Values of < 0.7 (or 70%) were interpreted as low, values of 0.7 to 0.85 (70% to 85%) were interpreted as moderate and values of > 0.85 (> 85%) were interpreted as high. 259

Cohen’s kappa statistic was used to compare the agreement in classifying neurodevelopmental outcomes into the three categories of ‘no’, ‘mild–moderate’ and ‘severe’ impairment (ordinal data). The κ coefficient is a measure of the proportion of agreement above that is due to chance alone and is calculated by:

A κ value of 1 indicates perfect agreement and a value of 0 reflects agreement that is no better than by chance. Unweighted and weighted forms of κ coefficient were obtained. The purpose of the weighting was to derive a coefficient that provided a closer reflection of the clinical implications of disagreement between the ordinal categories. It is clinically more important to distinguish patients with impairments from those without impairment than to differentiate between the severity of ‘mild–moderate’ and ‘severe’ impairments. Hence, in the calculations for the weighted κ coefficient, discrepancy between ‘mild–moderate’ and ‘severe’ impairments was considered to be partial agreement.

The weighting matrix used is shown in Table 21.

The κ values were interpreted according to the standards proposed by Landis and Koch: 0–0.4, slight to fair agreement; 0.4–0.6, moderate agreement; and 0.6–1, substantial to perfect agreement. 260

Sample size

A precision analysis for the estimated sensitivity of the NHS assessment in identifying children with Bayley-III scores of < –2 SDs in study 1 was used to calculate the target sample size for recruitment. The desired sensitivity of a developmental test is conventionally between 70% and 80%. The precisions (widths of CI) of the observed sensitivity and specificity of a test vary depending on sample size and the observed estimates. The aim was to achieve a precision of 95% CI half-width within 10% for the estimated sensitivity of identifying children with Bayley-III scores of < –2 SDs by the NHS assessment.

Based on the London Perinatal Networks 2008 Annual Report,261 it was estimated that approximately 500 children are born at < 30 weeks’ gestation and survive to discharge from the participating hospitals per year. Assuming that 10% of these children have Bayley-III scores of < –2 SDs, with an unstratified random sample, 650 participants would be required to achieve a CI half-width within 10% for an estimated sensitivity of 80%. We attempted to recruit a stratified sample to include higher proportions of children with medium and high risk for impairment to improve the precision of the study while maintaining a practical sample size262 (Table 22).

Of the infants born at < 30 weeks’ gestation who survived to discharge in London in 2008, 20% were born at ≤ 25 weeks’ gestation (higher-risk group), 30% were born at 26 to 27 weeks’ gestation (medium-risk group) and 50% were born at 28 to 29 weeks’ gestation (lower-risk group). Assuming that 25% of higher-risk, 15% of medium-risk and 0% of lower-risk children achieve Bayley-III scores of < –2 SDs, and the sensitivity of identifying different severity of impairment is the same for all risk groups, Table 22 shows various sample size options and the resulting CI half-width for different sensitivity estimates. We aimed to recruit 500 children (i.e. 200 from the higher-risk group, 200 from the medium-risk group and 100 from the lower-risk group) over the 2-year recruitment period.

Representativeness of the study population

Neonatal and sociodemographic characteristics of study participants and non-participants were compared using data extracted from the NNRD.

Comparing classification of impairments

To estimate how comparable the three levels of impairment (i.e. none, mild–moderate and severe) based on the modified NPEU/Oxford criteria are to the three Bayley-III SD score groups of ‘> –1 SD’, ‘–1 to –2 SDs’ and ‘< –2 SDs’, using only the data obtained from the research assessment, the two sets of criteria were cross-tabulated and the unweighted and weighted Cohen’s κ coefficients were calculated. This was also performed for each neurodevelopmental domain.

Any participant with Bayley-III scores of < –1 SD was considered to have at least mild impairment. Taking the research assessment to be the reference ‘gold standard’, the sensitivity and specificity of the NHS data in identifying children with any impairment were calculated and the sensitivity and specificity of the severe impairment category in the NHS data for identifying children with Bayley-III scores of < –2 SDs were calculated.

To account for correlation clustering by study sites, robust standard errors were used to calculate the 95% CI for the estimated sensitivities and specificities. To examine the effect of correlated outcomes within multiple birth sets, analyses were repeated on all singleton births and one randomly selected child from each multiple birth set.

Weighted and unweighted κ coefficients were used to measure the concordance between the research and the NHS assessments, again matching the ‘no impairment’ category to Bayley-III scores of higher than –1 SD, ‘mild–moderate’ to Bayley-III scores of between –1 and –2 SDs and ‘severe’ to Bayley-III scores of lower than –2 SDs.

Defining question sets for identifying severe impairment

The NPEU/Oxford expert group suggested that a criterion of –3 SD scores be used to represent ‘severe cognitive (developmental) disability’ at the age of 2 years. 221 However, this cut-off point was not feasible because of the floor effect of the Bayley-III cognitive composite scores, which ranged between 55 and 145. A post hoc analysis was therefore performed to evaluate if applying broader criteria at the severe end of the impairment spectrum would improve the validity of NHS data in identifying children with Bayley-III scores of lower than –2 SDs. For this, impairment categories were re-defined as ‘none’, ‘mild’ and ‘moderate–severe’. Referring back to the ‘2-year’ questions on the EPR, participants who received a positive response to the following questions were recategorised into the ‘moderate–severe’ category (Table 23).

Participants who received a positive response to any of the other questions were classified as having mild impairment. The sensitivity and specificity of the ‘moderate–severe’ category in predicting children with Bayley-III scores of lower than –2 SDs was then calculated and the concordance of NHS data classified into these new categories with the Bayley-III SD score groups, matching ‘no impairment’ to Bayley-III scores of higher than –1 SD, ‘mild impairment’ to Bayley-III scores of between –1 to –2 SDs and ‘moderate–severe’ impairment to Bayley-III scores of lower than –2 SDs.

Variables associated with the validity of NHS neurodevelopmental data

The effect of the following factors on the validity of the NHS data were examined: gestation at birth, sex, supplemental oxygen requirement at 36 weeks corrected gestational age, IMD quintile at the time of assessment, English as the only language spoken at home, corrected age at NHS assessment, use of a standardised neurodevelopmental test or screening test during NHS follow-up, grade of NHS assessor, time interval between NHS and research appointments, behaviour during the research assessments as measured by the examiner-rated behavioural score, and whether or not the NHS assessor thought that the child was difficult to test during the NHS assessment. Cross-tabulations and the calculation of the sensitivities and specificities of NHS assessment, stratified by the factor under study, were performed for each domain of neurodevelopment.

Assessment of social communication and autistic traits in early childhood

For the purpose of assessing the applicability of the Q-CHAT for children who were born preterm, children with cerebral palsy and children with severe neurosensory impairments (defined as a hearing deficit not correctable with hearing aids or a visual deficit not correctable with glasses) were excluded from this analysis. Differences in characteristics between respondents and non-respondents, and between respondents and the ‘baseline population’, were compared to evaluate selection bias.

The overall and sex-specific Q-CHAT scores from the study population were compared with published scores from the general population [general population overall mean 26.7 (SD 7.8), mean for boys 27.5 (SD 7.8), mean for girls 25.8 (SD 7.7)]220 using the Student’s t-test. Differences in the distributions of item-specific scores between the study cohort and the general population in each category of autistic-like behaviour were examined by chi-squared tests. To overcome the chi-squared test restriction for low numbers, the proportions in adjacent score categories were combined to ensure that all expected values were larger than 5. 263

The correlation between the Q-CHAT scores and the Bayley-III cognitive, language and motor composite scores was explored using linear regression to determine if any observed differences in Q-CHAT scores between the study population and the general population were explained by delayed neurodevelopment in the preterm population. Post hoc analysis of the correlation between subcategorical Q-CHAT scores (total score from items within each category of autistic-like behaviour) and Bayley-III cognitive, language and motor composite scores was carried out with Bonferroni correction for multiple testing.

The following neonatal and sociodemographic factors were analysed for possible association with Q-CHAT scores: gestation at birth, birthweight z-score, sex, single versus multiple pregnancy, white versus non-white ethnicity, maternal age at birth, mode of delivery, length of mechanical ventilation, supplemental oxygen requirement at 36 weeks’ corrected gestational age and IMD quintile at the time of completion of the Q-CHAT. The current IMD quintile for participant was chosen rather than the birth quintile. Comparing the IMD quintiles at birth with those at the time of assessment, 177 (83.9%) participants continued to live within the same IMD quintile, 13 (9.2%) moved to a more deprived IMD quintile and 15 (10.6%) moved to a less deprived quintile. Linear regression models were created to determine the association between predictive variables and Q-CHAT scores. To account for correlated outcomes within multiple birth sets, cluster bootstrap analysis was used to estimate standard errors and the resultant 95% CI. Variables identified to be significant at a 5% level in univariable models were included in forward stepwise multivariable regression analyses to determine the independent effect of each factor on Q-CHAT scores. Post hoc analysis was conducted to explore possible interactions between ethnicity, Bayley-III language scores and IMD.

Using an arbitrary cut-off score of 2 SDs above the general population mean for Q-CHAT scores and 2 SDs below the standardised mean for Bayley-III Social-Emotional scores, participants were classified as ‘at risk for ASD’. A scatterplot was used to examine the relationship in score distribution between the Q-CHAT and Bayley-III Social-Emotional questionnaires and the agreement between the questionnaires in identifying children ‘at risk’ was measured using Cohen’s κ statistic.

Systematic literature review and meta-analysis

A systematic electronic literature search was conducted on MEDLINE for information on the early developmental outcomes and corresponding school-age cognitive outcomes of preterm children. The methods adopted in this review were based on recommendations outlined in the Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. 264,265 Results are reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement. 265

Any cohort or matched-control studies published since 1 January 1990 on study populations of infants born ≤ 32 weeks’ gestation and/or who had a birthweight of < 1500 g (VLBW), in which at least two serial assessments, consisting of a neurodevelopmental assessment conducted between 1 and 3 years of age and a cognitive assessment at ≥ 5 years of age, were conducted and reported using validated standardised psychometric assessments [e.g. BSID, Griffiths Mental Development Scales (GMDS), Wechsler Preschool and Primary Scale of Intelligence] were considered for inclusion in the review. Assessments conducted before 1 year of age were not included because impairment, particularly if mild, may not be evident at this stage. Studies with populations that did not meet the gestation or birthweight criteria or reported outcomes using non-standardised assessments including measures of academic attainment were excluded. Studies that reported only outcomes in language or executive function (e.g. memory) were excluded as they would not reflect the overall cognitive function of the study populations. Case reports, narrative reviews, editorials, letters and comments on published articles were excluded.

The electronic search was conducted on MEDLINE through the PubMed interface on 13 April 2012, covering English-language literature published between 1 January 1990 and 31 March 2012. Search terms were selected a priori through a preliminary review of the literature. The following search terms were used both as keywords and as subject headings: (combinations of ‘preterm’ or ‘premature’ with ‘infant’ or ‘neonate’ or ‘children’) or (‘low birthweight’ or ‘extremely low birthweight’) and (‘cogniti*’ or ‘neurodevelopment*’ or ‘mental retardation’ or ‘disability’ or ‘intelligence’ or ‘IQ’). The ‘explode’ feature was used with subject headings to include articles categorised under more specific subheadings. The detailed search strategy was as follows:

((‘preterm children’[tiab] OR ‘premature children’[tiab]) OR (‘premature infant’[tiab] OR (‘preterm infant’[tiab]) OR (‘preterm neonate’[tiab] OR ‘premature neonate’[tiab]) OR (‘Infant, Premature’[MeSH]) OR (‘Infant, Very Low Birthweight’[MeSH]) OR (‘very low birthweight’[tiab] OR ‘very low birthweight’[tiab]) OR (‘extremely low birthweight’[tiab]) OR (‘extremely low birthweight’[tiab])) AND ((cogniti*[tiab]) OR (neurodevelopment*[tiab]) OR (mental retardation) OR (‘Developmental Disabilities’[Mesh] OR disability[tiab]) OR (intelligence[tiab] OR IQ[tiab])).

The electronic search was supplemented by a manual search of the reference lists of studies that met the inclusion criteria.

The titles and abstracts of studies retrieved from the literature search were screened to identify studies that reported developmental and/or cognitive outcomes among preterm children who were born before 32 weeks’ gestation and/or were VLBW. These were grouped into three categories: (1) studies that reported both early developmental outcomes between ages 1 and 3 years as well as school-age cognitive outcomes at ≥ 5 years, (2) studies that reported only early developmental outcomes and (3) studies that only reported school-age cognitive outcomes. The author lists for articles in groups (2) and (3) were matched to identify assessments and publications on the same population at different time points. Studies that satisfied the initial screening process were retrieved for full-text evaluation for final inclusion in the review.

The quality of included studies was assessed using a checklist adapted from the Quality of Diagnostic Accuracy Studies version 2 (QUADAS-2) appraisal tool. 266 The aim was to provide a qualitative judgement for the risk of bias and the applicability of each study to the review question. The QUADAS-2 tool uses ‘signalling questions’ to assess bias in four domains: patient selection, index test, reference standard, and flow of participants through the study and timing of the index test. The applicability of the study to the review question in the first three domains was also assessed. In the context of this review, the index tests referred to the early developmental assessments and the reference standards were the school-age cognitive assessments. An essential feature of QUADAS-2 was the tailoring of the signalling questions to enable review-specific appraisal. Table 24 lists the signalling questions and the quality standards set for this review. By appraising against the set standards, each study was given a rating of ‘low’, ‘high’ or ‘unclear’ for risk of bias and concerns regarding applicability in each domain. No summary ‘quality score’ was generated as such scores lack statistical justification and are not comparable across different scoring systems. 267 It was decided not to exclude any study on the basis of its quality, to achieve a review on the topic that was as comprehensive as possible.

From each included study, the following data were extracted into a table (unpublished data were sought from study authors through e-mail requests): study characteristics (i.e. location, city, country); sampling method (i.e. single centre, multicentre, population based); inclusion and exclusion criteria; anticipation and/or follow-up rates (as percentage of eligible survivors); final sample size included in meta-analysis (i.e. number of participants who completed both early and school-age assessments); early developmental and school-age cognitive assessment tool used; and study population characteristics [i.e. year(s) of birth of participants, mean or median gestational age, mean or median birthweight, ages at assessment, mean test scores at assessment, data on the predictive validity of early developmental assessments].

For this review, mild–moderate deficit was defined as developmental or cognitive test scores of between 1 and 2 SDs below the means of the standardised or control groups. Severe deficit was defined as test scores of lower than 2 SDs below the means of the standardised or control groups. In studies for which a control group of children who were born at full term were recruited and assessed simultaneously, the mean and SD of the control group were used as the references for defining the presence of deficits. Data on the number of ‘true-positive’, ‘false-positive’, ‘false-negative’ and ‘true-negative’ cognitive deficits identified by early assessments were collated from each study. If serial assessments were performed at different time points, data obtained from participants at the oldest age were included in the meta-analysis. The estimated sensitivity and specificity with corresponding 95% CI for mild–moderate and severe deficits were calculated.

Meta-analysis

The goals of the meta-analysis were to evaluate the variation in the estimates of the diagnostic accuracy (sensitivity and specificity) of early developmental assessments between studies and to combine results from all studies to yield a more precise estimate than is possible from individual studies. Coupled forest plots were generated to depict the ranges of sensitivity and specificity derived from the studies. Homogeneity of the sensitivities and specificities from the studies were tested using chi-squared tests. It has been noted that, in meta-analyses of diagnostic tests, significant between-study heterogeneity often exists. One source of heterogeneity is attributable to variations in diagnostic threshold and the related ‘trade off’ between sensitivity and specificity. 264,268 This may occur even when the same diagnostic criterion was applied across the studies (as was in this review) because of, for example, inherent differences in the spectrum of impairments in the patient populations or interobserver interpretation of test performances. To examine this, a scatterplot of the true-positive rate (TPR) (or sensitivity) against the false-positive rate (or 1 – specificity) for each study was created and the Spearman correlation coefficient was computed. ‘Threshold effect’ was demonstrated when the points assume the shape of a ROC curve and the sensitivity and specificity were significantly correlated. In this circumstance, separate pooling of sensitivities and specificities that ignore the correlation between the two measures would lead to an underestimation of the diagnostic accuracy. 269 It is possible to combine estimates using the Moses–Littenberg method to generate a summary ROC curve. 268,270 However, this does not allow for between-study variation. Instead, the Rutter and Gatsonis approach was used to fit a hierarchical summary ROC (HSROC) curve of the data. 271 The HSROC model accounts for both sampling variation within study at a lower level and between-study heterogeneity at a higher level using random effects. It models the log-odds of a positive test result in each study and each impairment group as a function of the positivity threshold in each study and the true impairment status, with model parameters describing the accuracy and asymmetry of the ROC curves. The output includes a summary operating point (pooled values for sensitivity and specificity) with 95% confidence region and a 95% prediction region for a forecast of the true sensitivity and specificity in a future study. As this is a hierarchical model, the summary operating point represents an average of study effects rather than a common effect. Individual study effects may differ considerably because of heterogeneity, and this variation is represented by the 95% prediction region.

The possible association of the diagnostic validity with study-level variables that could account for the observed heterogeneity among studies was investigated using metaregression methods for continuous variables and subgroup analysis for categorical variables. The variables were gestational age, birthweight, age at early assessment, age at late assessment, time interval between assessments, year of birth of participants, prevalences of total and severe impairment, the developmental assessment tool used, and the inclusion/exclusion of neurosensory impaired participants. For categorical variables, couple forest plots stratified by the subgroups were generated to allow for visual assessment of the differences in diagnostic validity between subgroups.

For continuous variables, scatterplots of sensitivity and specificity against each study-level covariates were generated by taking the mean value for continuous variables within each study except for year of birth, when the earliest date was used, as the mean/median value was not available. Bivariate models272 were used to formally test whether or not sensitivity and specificity were associated with study-level covariates. Bivariate models are equivalent to HSROC models when no covariate is included. 273 When including covariates, the bivariate model measures the association with sensitivity and specificity (on the logit scale), whereas the HSROC model measures the association with the accuracy and threshold parameters; therefore, the former was chosen for ease of interpretation. For each study-level covariate, associations with sensitivity and specificity were tested separately; likelihood ratio test was then used to test both associations jointly. Results are reported as estimated ORs with associated 95% CI and p-values.

For the studies that reported data from multiple assessments at different time points, scatterplots were created of sensitivity and specificity against mean age at assessment to explore the stability of sensitivity and specificity estimates over time. For reviews of interventional trials, the funnel plot, a graphical display of the estimates of study effects plotted against their sample size or precision (standard error), is the recommended method for examining publication bias. Statistical tests, such as Egger’s regression test and Begg’s rank correlation, are used to test for funnel plot asymmetry, which would indicate the presence of publication bias and other sample size-related effects. The appropriate method for investigating publication bias for studies of diagnostic test accuracy is unclear. Funnel plots of the estimates of log-diagnostic odds ratio (DOR) against corresponding precision were proposed. 274 The DOR is a single statistic measure of diagnostic performance that is defined as:

Therefore, the larger the DOR, the more accurate the test is. In the Cochrane Handbook for Systematic Reviews of Interventions: Version 5.1.0,275 the application of tests for funnel plot asymmetry designed for use in randomised trials, including the Egger and Begg tests, is specifically discouraged as these are associated with inflated type I error rates. Instead, a regressions test for the association between the log-DOR and the ‘effective sample size (ESS)’, developed by Deeks et al. ,276 was suggested. The ESS is a function of the number of non-diseased (n₁) and diseased (n₂) participants, in which:

Following the proposed methods outlined in the paper by Deeks et al. ,276 the possibility of publication and other sample size-related effects was investigated by developing funnel plots of log-DOR against 1/ESS^1/2 and tested for plot asymmetry using linear regression of the two variables, weighted by ESS.

For this review, forest plots were generated using RevMan, version 5.2 (The Cochrane Collaboration, The Nordic Cochrane Centre, Copenhagen, Denmark). All other analyses were performed using Stata statistical package, version 11.0, and SAS.

Two-year neurodevelopmental outcomes

Two hundred and eight children were recruited from 13 hospitals (Addenbrooke’s, Cambridge; Queen’s, Romford; Chelsea and Westminster, London; Ealing Hospital, London; Hillingdon Hospital, London; Homerton University, London; Newham, London; North Middlesex University, London; Northwick Park, London; Royal London, London; St Thomas’, London; West Middlesex, London; Whipps Cross Hospital, London).

Figure 17 shows the flow of children through recruitment to the completion of research and NHS assessments. Two hundred and four children completed all the subtests of the Bayley-III assessment. One child with ataxic cerebral palsy could not be assessed for the cognitive and language scales, two children did not co-operate for the receptive communication assessment and one child did not co-operate for the gross motor assessment. Of the children who completed the research assessments, three did not attend their routine NHS follow-up visit. Data from the NHS assessment were not entered onto the EPR by the examining clinician in 15 cases and the overall category of impairment could not be assigned for nine children because of missing EPR data. The 190 children for whom both research and NHS data were available in at least one outcome domain formed the study cohort. A complete set of data in all outcome domains was available for 177 children. Although the original plan was to stratify recruitment based on the gestational ages of the eligible children, it became clear during the recruitment phase that the final study cohort would be smaller than the initial projection. Therefore, all children whose parents agreed to participate were recruited and assessed.

FIGURE 17.

Flow chart of children through research and NHS assessments to form the study population.

The characteristics of the study population were compared with the ‘baseline population’ (all infants born between 1 January 2008 and 31 December 2010, at gestational ages below 30 weeks and discharged from the participating hospitals) (Table 25). Participants received a shorter duration of mechanical ventilation and were less likely to be receiving oxygen therapy at 36 weeks postmenstrual age than the baseline population (p < 0.001). The study population was comparable to the baseline population in terms of gestational age, birthweight, sex, proportions of singletons, mode of delivery and maternal age, and consisted of larger proportions of children of white ethnicity and those born to mothers living in the least deprived IMD quintile. Nevertheless, a wide range of ethnic groups was represented, reflecting the diversity of the population living in London. Consequentially, 92 (48.4%) children were raised in a bilingual or multilingual environment.

The mean (SD) corrected age of the children at assessment was 24.8 (2.2) months. The research assessment took place at a median (IQR) interval of 8 (0–27) days after the children received their NHS assessment, with a range of between 89 days before and 82 days after the NHS assessment.

Based on information given by the parents, 30 (15.8%) children had a visual defect including reduced visual acuity and/or squints, although only 11 (5.8%) required glasses. A total of 16 (8.4%) children had a hearing impairment, of whom three (1.6%) wore hearing aids.

The children performed significantly worse than the normative population, in which Bayley-III scores were standardised in all domains other than fine motor skills (Table 26).

Based on the worst score achieved in the cognitive, language and motor Bayley-III domains, 114 (61.3%) children were classified as having scores of higher than –1 SD from the standardised mean, 42 (22.6%) children had scores of between –1 and –2 SDs and 30 (16.1%) children had scores of lower than –2 SDs from the standardised mean.

In the cognitive domain, 156 (82.1%) children obtained Bayley-III scores of higher than –1 SD, 26 (13.7%) children had scores of between –1 and –2 SDs and seven (3.7%) children had scores of lower than –2 SDs from the standardised mean.

Nineteen (10.0%) children had specific expressive communication impairment (scaled score of < 7), with no impairment in receptive communication. Five of these children would have been classified as having no impairment based on the Bayley-III language composite score alone (language composite score of higher than –1 SD, i.e. ≥ 85) because of the compensation from the receptive communication subtest. Based on the worst SD score category from the receptive and expressive communication subtests and the language composite score, 120 (63.2%) children achieved scores of higher than –1 SD, 42 (22.1%) had scores of between –1 and –2 SDs and 25 (13.2%) had scores of lower than –2 SDs from the standardised mean.

Motor function was generally intact among the children, with only 11 children (5.8%) receiving scores of between –1 and –2 SDs and 11 (5.8%) children having scores of lower than –2 SDs from the standardised mean. The mean (SD) predicted BSID-II MDI for the study population was 77.9 (20.5) and was significantly lower than the mean Bayley-III cognitive and language composite scores (p < 0.001 for both). Figure 18 shows the proportions of children with scores of higher than –1 SD (≥ 85), scores of between –1 and –2 SDs (70–84), scores of between –2 and –3 SDs (55–69) and scores of lower than –3 SDs (< 55) from the standardised mean for the Bayley-III cognitive and language composite scores individually, when the lower of the two Bayley-III scores was used and for the predicted BSID-II MDI score. A post hoc analysis using McNemar’s test was performed to compare the proportions diagnosed with impairment using these cut-off scores. The proportions of children classified with impairment using a cut-off point of < 85 on the Bayley-III cognitive score (17.4%), language score (32.6%) and the lower of the cognitive and language scores (35.8%) were statistically dissimilar to the proportions with predicted BSID-II MDI < 70 (25.3%) (p < 0.001). However, the proportions of children with predicted BSID-II MDI of < 55 (11.6%) were similar to the proportions who scored < 70 (classified with severe impairment) on the Bayley-III language composite score (13.1%; p = 0.26) and the lower of the cognitive and language scores (13.7%; p = 0.16).

FIGURE 18.

Neurodevelopmental status by Bayley-III scores and predicted BSID-II MDI.

Classification using the modified NPEU/Oxford criteria showed that, for the communication domain, the assignment of outcome was heavily influenced by the presence of specific expressive communication impairment: 107 (56.3%) children were classified as having no language impairment, 55 (28.9%) children had mild–moderate language impairment and 27 (14.2%) children had severe language impairment. Thirteen (6.8%) children had isolated gross motor impairment with no fine motor difficulties, and only one (0.5%) child had specific fine motor impairment. The combined motor outcome was normal in 172 (90.5%) children. Nine (4.7%) children were classed as having mild–moderate motor impairment and nine (4.7%) children had severe motor impairment.

Evaluating the concordance in the classification of neurodevelopmental status by Bayley-III scores and NPEU/Oxford criteria, of the 187 children tested for their communication skills, 144 (77.0%) were classified in the same category. Of the other children, none differed by more than one category. The weighted kappa coefficient (κ) was 0.59 (95% CI 0.49 to 0.69), which indicated moderate agreement between the two criteria for the communication outcome. For the motor domain, 180 out of 189 (95.2%) children were classified in the same category. Classification differed by one category for eight children. One child who was assessed as having severe motor impairment by the Bayley-III was classified as having ‘no impairment’ on the NPEU/Oxford criteria. The weighted κ for concordance between the two methods in the motor domain was 0.76 (95% CI 0.62 to 0.93), which represented substantial agreement.

The per cent agreement across all assessed items was 97.2% (69/71 items in agreement) in the first session (midway) and 98.6% (69/70 items in agreement) in the second session end of study assessments).

Neurodevelopmental outcomes from NHS electronic patient record data

Children attended their NHS follow-up assessment at a mean (SD) corrected age of 24.4 (2.3) months. Data were entered on the EPR ‘2-year outcome’ screen by clinical consultants in 111 (58.4%) cases [36 (19.0%) by consultant neonatologists, 42 (22.1%) by hospital paediatrics consultants and 33 (17.4%) by community paediatrics consultants], trainee doctors in 15 (7.9%) cases, staff-grade doctors in 58 (30.5%) cases and administrative staff in six (3.2%) cases. Sixty-seven (35.3%) children received standardised neurodevelopmental assessment or screening tests during their NHS appointment [19 (10.0%) using the Schedule of Growing Scales, 44 (23.2%) using the GMDS and 4 (2.1%) using the Alberta Infant Motor Scale]. Table 27 shows the responses to each question on the electronic form and the classification of impairment for the developmental domains. The classification of overall neurodevelopmental outcome was possible in 181 children, of whom 124 (68.5%) had no impairments, 38 (21.0%) had mild–moderate impairments and 19 (10.5%) had severe impairments.

The proportions of children classified into each category of impairment by the research assessment (using Bayley-III scores and NPEU/Oxford criteria) and by the NHS assessment are displayed in Appendix 2, Figures 36–39.

Cross-tabulations to compare the agreement between the research and the NHS assessments for the classification of ‘any impairment’ and ‘severe impairment’ are displayed in Tables 28 and 29. The estimated sensitivities and specificities for NHS assessments in each developmental domain are presented taking the research assessment as the ‘gold standard’. The CIs for sensitivities and specificities were calculated using robust standard errors to account for clustering of data by study sites. Sensitivity analyses revealed that potential correlated outcomes from siblings did not affect the results. Therefore, the results presented included data from all participating children.

The validity of the NHS assessments in identifying children with no impairments was high, with estimated specificities ranging between 83.9% and 100.0% for ‘any impairment’, and between 96.6% and 100.0% for ‘severe impairment’. However, the validity of the NHS and the research assessment in identifying and categorising children with impairments was variable. The sensitivities for identifying gross motor impairment were high, particularly when the impairment was severe. In the cognitive domain, the sensitivity for the identification of any impairment was 69.7% (95% CI 55.1% to 84.3%) but dropped to only 28.6% (95% CI 5.0% to 52.2%) for the identification of severe impairment. Of the seven children diagnosed with severe cognitive impairment through the research assessment, two were also classified as having impairment in the ‘severe’ category in the NHS data set, four were classified as having impairment ‘mild–moderate’ and one was classified as having ‘no’ impairment; hence, the disagreement in the classification occurred mainly between the ‘mild–moderate’ and ‘severe’ categories. Agreement between the NHS and research assessments was worst in the language domain, especially in receptive communication, where the sensitivity in identifying the presence of any impairment was only 23.1% (95% CI 6.7% to 39.5%). In the combined language domain, the 21 ‘false negatives’ for severe impairment, based on Bayley-III classification, were evenly distributed among the impairment categories in the NHS data (‘severe’, n = 9; ‘mild–moderate’, n = 6; ‘no’ impairments, n = 6). The sensitivities were estimated with low precision (wide CIs), particularly in the motor domains and with severe impairments when the prevalence of impairment was low.

Although the sensitivities in the receptive communication and fine motor domains appeared considerably higher when impairment was assigned using the NPEU/Oxford criteria than using Bayley-III scores for the research assessment, this was driven by the small numbers in the ‘false-negative’ cells and the estimated sensitivities were associated with wide and overlapping CIs, which suggested that the differences in sensitivities may not be statistically significant.

The sensitivities and specificities of NHS assessment in identifying cognitive deficit were 69.7% (95% CI 55.1% to 84.3%) and 83.9% (95% CI 75.6% to 92.1%) for the presence of any impairment and 28.6% (95% CI 5.0% to 52.2%) and 97.8% (95% CI 95.1% to 100.0%) for severe impairments (see Tables 28 and 29). The analyses were repeated using the predicted BSID-II MDI scores as the ‘gold standard’. Using the cut-off point of MDI < 85 (–1 SD) to define mild–moderate impairment and < 70 (–2 SDs) to define severe impairment, there was a reduction in the sensitivities and an increment in the corresponding specificities [sensitivity 39.3% (95% CI 30.2% to 49.0%) and specificity 94.6% (95% CI 86.7% to 98.5%) for any cognitive impairment; sensitivity 12.5% (95% CI 4.7% to 25.2%) and specificity 100% (95% CI 97.4% to 100%) for severe cognitive impairment]. However, if thresholds of MDI < 70 (–2 SDs) for mild–moderate and < 55 (–3 SDs) for severe impairments were used, the results were similar to the reported findings using the Bayley-III [sensitivity 64.6% (95% CI 49.5% to 77.8%), specificity 87.7% (95% CI 81.0% to 92.7%) for any impairment; sensitivity 18.2% (95% CI 5.1% to 40.3%), specificity 98.8% (95% CI 95.7% to 99.9%) for severe impairment].

The concordance of the research and the NHS assessments as measured by κ was consistent with the findings from the estimated sensitivities and specificities. The agreement between NHS and research assessment was substantial in the motor domain with weighted κ > 0.6. In the cognitive and communication domains, agreement was moderate at best.

Post hoc analysis of the validity of NHS assessments using a different question set to identify ‘moderate–severe’ impairment

The purpose of this post hoc analysis was to assess if, by applying a broader criterion to define ‘moderate–severe’ impairment, the validity of the NHS data in identifying children with Bayley-III scores of lower than –2 SDs could be improved. Children were reclassified as having moderate–severe impairment if they met the broader criterion in the NHS data. The results are displayed in Appendix 1, Table 58. The use of a broader category of moderate–severe impairment improved the sensitivity of the NHS data, although this was at a cost of a small reduction in specificity. The biggest increase in sensitivity was observed in the cognitive and expressive communication domains.

Variables affecting the validity of the NHS assessments

As the diagnostic validity of the NHS assessment did not differ between the use of Bayley-III scores and NPEU/Oxford criteria for classifying impairment, subgroup analyses were performed using only the results from Bayley-III assessments. Lower prevalence of impairment with higher gestational age at birth across all domains was observed, with apparent reduction in sensitivity but increased specificity of NHS assessments in identifying overall impairment with increasing gestational age. Sensitivity in identifying cognitive impairment was higher if a standardised neurodevelopmental test was used during NHS assessment. Accuracy in identifying impairment also appeared higher across all domains with increasing postnatal age at assessment. However, as the CIs for the estimated sensitivities and specificities overlapped widely, the observed effect of these factors on the diagnostic validity of NHS assessment can be conservatively considered statistically insignificant. 277 Similarly, there was no clear effect of the exposure to English language, the grade of the NHS assessor, IMD and the time interval between NHS and research assessments on the validity of NHS assessment.

Behaviour during assessments and the effect on study findings

The prevalence of impairment was significantly higher among children who were difficult to assess during the NHS assessment (86.7% vs. 31.7% for impairment in any domain; p < 0.001) or who had received lower examiner-rated behaviour scores (less positive behaviour) (73.8% for impairment in any domain among children with behaviour score of ≤ 22 vs. 28.5% among children with behaviour score of > 22; p < 0.001). However, challenging behaviour demonstrated during assessments did not appear to affect the test validity of the NHS assessment against the research assessment. The prevalence of impairment, sensitivity and specificity of NHS assessment did not differ by parent-rated behaviour scores.

Hammersmith Infant Neurological Examination and diagnosis of cerebral palsy

Forty-seven (24.7%) children had a suboptimal global score (< 73/78) on the HINE. In general, in the preterm population, although scores below 73 are suboptimal, those with scores of > 64 will walk independently by 2 years, those with scores of < 64 but > 52 will sit independently by 2 years and those with scores of < 52 will not be able to do either. The proportions of participants who achieved suboptimal score in each subsection were as follows: 30 (15.8%) for cranial nerve function, 39 (20.5%) for posture, 16 (8.4%) for movement, 36 (18.9%) for tone and 8 (4.2%) for reflexes.

Nine (4.7%) children were found to have cerebral palsy during the research assessment. The HINE scores for these children (median 53, IQR 38.5–59.5) were significantly lower than those without cerebral palsy (median 78, IQR 74–78; p < 0.001) and consistent with published data. 210 Two children had spastic quadriplegia, five had spastic diplegia, one had three-limb involvement and one had dyskinetic cerebral palsy. The gross motor function varied from GMFCS level 1 (walks without limitations) for one child with spastic diplegia to GMFCS level 5 (transported in manual wheelchair) for the child with dyskinetic cerebral palsy and one of the children with spastic quadriplegia. Most children with spastic diplegic cerebral palsy functioned at GMFCS level 2 (walks with limitations).

Two children with spastic diplegia were not identified to have cerebral palsy in the NHS data. The topographic classifications entered in the NHS data for all other children identified to have cerebral palsy were in agreement with the research assessment.

Early childhood social communication difficulties

The Quantitative Checklist for Autism in Toddlers (Q-CHAT) questionnaire was sent to the parents of all 208 children who attended the research assessment. Ten children were assessed to have major functional impairments (nine with cerebral palsy and one with severe hearing impairment) and were ineligible for this study. The parents of three children who declined to participate in the research assessment agreed to complete the Q-CHAT and Bayley-III Social-Emotional questionnaires. A total of 150 questionnaires, including eight from children who were ineligible, were returned. One questionnaire with seven missing responses was treated as a non-respondent and excluded, leaving 141 participants (70.1% of eligible participants) for the analyses.

Non-respondents were more likely to be parents of girls (66.7%; p = 0.02). Nonetheless, both boys and girls were equally represented in the respondent group. Respondents showed over-representation of children of white ethnicity who were born to mothers living in less deprived IMD quintiles, with significantly shorter duration of mechanical ventilation and who were less likely to have required supplemental oxygen therapy at 36 weeks corrected age (see Appendix 1, Table 59).

The mean corrected age of the respondents was 24.7 (SD 2.6, range 18.5–35.6) months at the time of completion of the questionnaires. The mean Bayley-III composite score of the 138 respondents who completed the assessment was 94.6 (SD 13.0) for the cognitive scale, 87.7 (SD 13.0) for the language scale and 98.0 (SD 10.1) for the motor scale.

The Q-CHAT scores of the study population (mean 33.7, SD 8.3, range 15–55) were normally distributed and significantly higher (less favourable) than the published general population scores (difference in means 7.0, 95% CI 5.6 to 8.3; p < 0.001) (Figure 19). In contrast with the higher scores described in boys in the general population, no sex differences in Q-CHAT scores were observed in the preterm population (p = 0.85).

FIGURE 19.

Histogram of Q-CHAT scores of the preterm study population with superimposed distribution of published Q-CHAT scores of unselected toddlers (general population).

The distribution of scores between the preterm study cohort and the general population differed significantly in 17 items. In all of these items, there were greater proportions of preterm children receiving higher scores, indicating greater social communication difficulties and autistic behaviour characteristics. The differences were most prominent in the categories of restricted, repetitive, stereotyped behaviour (seven out of nine items differ significantly), communication (three out of four items) and sensory abnormalities (all three items). Only four out of the nine items exploring social relatedness were scored differently in the preterm population.

On multivariable testing, cognitive and motor function did not appear to affect Q-CHAT scores (p = 0.18 for cognitive scores and p = 0.67 for motor scores). Bayley-III language composite scores independently predicted Q-CHAT scores in a linear fashion (correlation coefficient –0.51; p = 0.001) and accounted for 24.5% of the variance in Q-CHAT scores. The relationship between language and Q-CHAT scores was attributable to expressive communication ability (regression coefficient Bayley-III expressive communication subscale scores and Q-CHAT scores: –1.35, 95% CI –1.96 to –0.74, correlation coefficient –0.43; p < 0.001). There was no association between receptive communication ability and Q-CHAT scores (p = 0.22).

Non-white ethnicity and living in deprived areas were associated with higher Q-CHAT scores in univariable analyses. Although non-white children were more likely to live in areas of higher deprivation (test for trend p < 0.001), there was no interaction between ethnicity and IMD in the association with Q-CHAT scores (p = 0.72). As lower Bayley-III language scores were observed among non-white children (mean difference 7.31, 95% CI 3.07 to 11.5; p < 0.001) and children living in more deprived areas (mean decrease of 1.89, 95% CI 0.24 to 3.55; p = 0.03) points per IMD quintile increase in deprivation, language ability was considered to be a potential confounder in the relationship between ethnicity, IMD and Q-CHAT scores. There was no interaction between Bayley-III language scores and IMD quintiles (p = 0.88) or ethnicity (p = 0.51). The final multivariable regression model included all variables found to be statistically significant during univariable analysis (Bayley-III language composite score, ethnicity and IMD) and is displayed in Appendix 1, Table 60.

The Bayley-III Social-Emotional questionnaire was completed in 140 out of the 141 eligible respondents to the Q-CHAT questionnaires. The Bayley-III Social-Emotional score distribution of the preterm population (mean 97.8, SD 17.2, range 55–145) did not differ significantly from the standardised norm of mean 100 and SD of 15 (p = 0.12). Twenty-three (16.5%) children had Q-CHAT scores of higher than 2 SDs above the general population mean (i.e. > 42.3). Only five (3.6%) children scored lower than 2 SDs below the standardised mean (i.e. > 70) for the Bayley-III Social-Emotional scale. There was poor concordance between the questionnaires, with only three children classed to be ‘at risk’ for ASD by both questionnaires and a resulting Cohen’s κ coefficient of 0.17 (95% CI 0 to 0.36).

Systematic literature review and meta-analysis

The PRISMA flow diagram is shown in Figure 20. The electronic literature search yielded 3600 unique citations (one of which was a duplicate). Application of search limits excluded 343 non-English articles and 413 articles published before 1990. Two additional studies were identified through manual search and author correspondence. Sixty-eight studies were selected for full-text evaluation from the title/abstract screen and 44 met the eligibility criteria. By matching 375 articles that reported the conduct of early developmental assessments with 323 articles that reported school-age assessments, 10 additional studies (in 23 articles) were identified. Data required for the review and meta-analysis were extractable directly from six articles. The authors of 18 of the remaining 48 studies contributed unpublished data for this review. The list of included studies is in Appendix 1, Table 61. For simplicity of referencing, studies that are represented by more than one article are denoted by the first author and year of publication of the earliest article in all tables and figures.

FIGURE 20.

The PRISMA flow diagram depicting the literature search process.

Predictive validity of early developmental assessment

The results of the cross-tabulations and the estimated sensitivities and specificities of early assessments for identifying any cognitive deficit for each study, in the form of coupled forest plots ordered by the sample size of the study, are shown in Figure 21 with the same information for the diagnosis of severe cognitive impairment. In studies for which participants were examined at different time points, only the results from the assessment performed at the oldest age are presented. This gives a final sample size of 3060 children for the meta-analysis. There was significant heterogeneity in the reported sensitivities and specificities among studies (p < 0.001 for both). The estimated sensitivities of diagnosing any impairment ranged from 17.0% to 90.5% and the corresponding estimated specificities ranged from 46.8% to 98.4%. For the diagnosis of severe impairment, the range of sensitivities was 0.0% to 100.0% and the range of specificities was 70.8% to 100.0%. The sensitivity of detecting severe impairment could not be estimated in the studies by Cohen286 and Fedrizzi et al. 279 as no participant had severe impairment. There appears to be a wider range and poorer precision (wider CIs) in the estimated sensitivity than in the specificity across studies. This may reflect the presence of heterogeneity or more likely as a result of estimates of sensitivity being based on smaller samples than estimates of specificity. The estimated sensitivity of 0.0% for severe impairment was based on a denominator of 1283,284,290 and 10293 diagnosed cases at school-age assessments. In general, the larger the sample size, the more precise (the smaller the 95% CI) the sensitivity estimates. The precision of specificity estimates appears to be high with the CI half-widths in 10 studies being < 10.0%. 160,161,173,225,280–282,285,288,291

FIGURE 21.

Results of cross-tabulations and coupled forest plots of the estimated sensitivities and specificities of early developmental assessments in identifying the presence of (a) coupled forest plots for the identification of any cognitive impairment; and (b) coupled forest plots for the identification of severe cognitive impairment. Sensitivities and specificities are expressed as proportions.

Meta-analytic pooled estimates of sensitivity and specificity

There was significant correlation between estimated sensitivities and specificities (see Appendix 2, Figure 41; Spearman’s rho –0.76; < 0.001). Therefore, the weighted averages of sensitivities and specificities were not computed separately. The pooled measures were estimated from the Rutter and Gatsonis HSROC curves that are presented in Appendix 2, Figure 42, for the presence of any impairment and severe impairments. The summary points and 95% CI regions are mapped out in the figures as well as the 95% prediction regions, which provide a forecast of the true sensitivity and specificity in a future study. The summary points corresponded to a pooled sensitivity of 55.0% (95% CI 45.7% to 63.9%) and pooled specificity of 84.1% (95% CI 77.5% to 89.1%) for the identification of any impairment. For the diagnosis of severe impairment, the pooled sensitivity was 39.2% (95% CI 26.8% to 53.3%) and pooled specificity was 95.1% (95% CI 92.3% to 97.0%).

Validity of early assessment assessed at different time points

In three studies,161,283,293 participants were assessed at two different time points for the early developmental assessments. In the five studies by Cohen,286 Reuss et al. ,288 Marlow et al. ,225 Smith et al. ,282 and Wolke and Meyer,173 participants received school-age cognitive assessments more than once. In Figure 22, the sensitivity and specificity for the identification of any impairment are plotted over the age at developmental assessment for the three studies that examined early assessment at two different time points. Figure 23 shows similar plots for the results obtained at serial school-age assessments in the five studies. It would appear, from these graphical displays, that the specificity of early assessment in excluding cognitive deficit remains relatively stable over time whereas no real correlation between sensitivity and age at assessment was apparent.

FIGURE 22.

Line graphs demonstrating the change in (a) sensitivity and (b) in specificity when early developmental assessments were repeated at different ages in three studies.

FIGURE 23.

Line graphs demonstrating the change in (a) sensitivity and (b) in specificity when school-age cognitive assessments were repeated at different ages in four studies.

Metaregression: association of study-level variables with diagnostic validity

The ORs and 95% CIs, together with the corresponding p-values, for the association of study-level variables with sensitivity and specificity of identifying cognitive deficit by early developmental assessment are presented in Table 30. There was reduction in specificity with increased observed prevalence of impairment in the study population. For each 1% increase in cognitive impairment prevalence, the odds of identifying an additional ‘true-negative’ case among those with no cognitive impairment reduced by 3% (p = 0.01). The associations between mean gestational age and mean birthweight and specificity of identifying cognitive impairment reached borderline statistical significance (specificity increased with mean gestational age and mean birthweight of the study population). Post hoc analysis revealed no association between the prevalence of impairment reported in each study and the mean gestational age (p = 0.55) and mean birthweight (p = 0.95) of the study population; therefore, excluding the speculation that the observed association between specificity and mean gestational age and birthweight was mediated by the prevalence of impairment. The age at the assessments, the time interval between early and school-age assessments and the year of participant birth were not associated with sensitivity or specificity and, therefore, did not explain the heterogeneity present between studies.

Funnel plot for sample size-related effects and publication bias

The funnel plot of the log-DOR against the inverse of the square root of the effect sample size is presented in Figure 24. Significance testing (ESS weighted regression test) confirmed that asymmetry was not present in the funnel plot (p = 0.22), indicating the absence of sample size-related effects in the meta-analysis.

FIGURE 24.

Funnel plot of the log-DOR against the inverse of the square root of the ESS, with pseudo-95% confidence limits.

Agreement between NHS and research-standard data

Among children who were born before 30 weeks’ gestation, the agreement in classifying neurodevelopmental status at the age of 2 years between data recorded during routine NHS assessments and those obtained through a research assessment was strong in the absence of neurodevelopment impairment. However, NHS assessments lack satisfactory sensitivity for identifying children with impairment, particularly in the cognitive and language domains. Using the Bayley-III scores as the ‘gold-standard’ tool, approximately 30% of children with at least mild cognitive impairments and nearly 50% with at least mild language impairments were falsely classified as having no impairment through NHS assessment. The discordance between NHS and research assessment remained, irrespective of the criteria used to categorise outcomes. This implies that the structural and content differences between the classification tools are unlikely to account for the discordances identified.

The strengths of the study include a single, carefully trained research assessor blinded to the results of the NHS assessment; the involvement of 13 hospitals serving patients from a wide range of ethnic, social, economic and cultural backgrounds; analyses that took into account the possibility of clustering by study sites and multiple births; and overestimation of abilities by the Bayley-III assessment. A limitation to the study was that the targeted sample size was not achieved. As the recruitment of participants occurred at a steady rate over the planned 2-year period, factors that may be relevant are the high population mobility in London, leading to loss to follow-up, and the large number of consultant and trainee doctors involved in outpatient follow-up clinics leading to missed opportunities to invite participation, because health professionals were unaware of the study. The sample size target was calculated with the desire to estimate the sensitivity of NHS assessment in correctly classifying children with severe impairment to a high precision, achieving a narrow 95% CI with half-widths within 10%. In addition to sample size, the precision was also dependent on the actual value of the sensitivity estimate: the lower the sensitivity, the wider the CI. As the estimated sensitivity for diagnosing severe overall impairment was low (52.0%), based on the 14.1% prevalence of severe impairment among the study participants, a sample of at least 680 children providing independent (unclustered) observations would be necessary to have achieved the intended precision. The increment in precision with increasing sample size followed the law of diminishing returns. It was therefore difficult to justify the continued provision of additional time and resources required to achieve the desired precision for the sensitivity estimate.

The study population differed from the population of very preterm children discharged from participating hospitals in that it consisted of proportionally more white children, who were less likely to have been mechanically ventilated, diagnosed with chronic lung disease (bronchopulmonary dysplasia) and/or were living in less deprived areas. It is possible, therefore, that the study population was at lower risk for neurodevelopmental impairment. The selection bias was introduced by attrition of children from routine NHS follow-up and the non-random recruitment method. In the literature, the proportions of VLBW children who were lost to follow-up or reviewed with difficulty in regional follow-up programmes were reported to be around 11–27% at 2 years296–298 and 25% at 5 years. 299 Characteristics associated with dropping out from follow-up included non-white ethnicity, young maternal age and low socioeconomic and maternal educational status. 297–300 Ideally, a random sample of participants selected from a known sampling frame (e.g. list of all children with scheduled follow-up appointments) would provide the most representative study cohort.

The presence of selection bias may have affected the accuracy of the estimated prevalence of impairment in the population. Traditionally, sensitivity and specificity are considered to be independent of disease prevalence. 301 Consequently, the adverse effect of selection bias on the validity of this study can be regarded as minimal. However, a number of studies have shown that variation in prevalence can result in either clinical or artefactual variation in test accuracy. 302,303 As it is probably easier to diagnose impairment in severe than in mild–moderate cases, a study population with a lower spectrum of impairment might have more false-negative or false-positive results.

Inter-rater variability in outcome assignment is likely to have been one of the main reasons for the disagreement between NHS assessments and research assessments. Clinical judgement is inevitably influenced by the assessor’s knowledge, experience, beliefs and preconceptions. Studies on behavioural psychology have shown that people tend to rely on judgemental heuristics (e.g. intuition), which are, by nature, unreliable, to simplify the complex task of assessing probabilities and predicting values to provide reasoning on the outcome of an event, such as the diagnosis of disability in a child. 304 The use of standardised assessment tools improves inter-rater agreement by establishing objective measures. Without using a standardised assessment, the judgement and interpretation of clinical findings may be highly variable. It is therefore unsurprising that, in a study comparing the diagnosis of cognitive impairment made using an intelligence test with judgements by paediatricians, the agreement was only fair (κ 0.39). Even if standardised assessments were used, the agreement between the different tools in classifying impairment was uncertain. Chaudhary et al. 305 reported that, at 22 months, children scored 5 points higher on the BSID-II MDI than on the Griffiths Scales developmental quotient. Furthermore, the interpretation and translation of the standardised assessment scores into the NPEU/Oxford classification instruction can still be inconsistent and subject to biases and errors.

Another difference between the conduct of the NHS assessments and research assessments was the reliance of the latter on parents’ reports on their child’s ability, particularly for language and cognitive skills. Parents are a valuable source of information in a time-restricted appointment, especially if the child does not engage in the assessment. However, studies investigating the level of agreement of neurodevelopmental status between parent and paediatrician evaluation have reported variable results. 244–246,248

Intrasubject (participant) variability in performance between assessments could also contribute to the discordance between NHS and research assessments. There are multiple factors, such as mood and ease of engagement of the participant, time of the day (meals/snacks or nap times) and environment, that can influence the children’s performance. Preterm children have been shown to be at risk of inattention/hyperactivity306,307 and social-emotional delays,308 which could manifest as inability to complete a task. The testing time is also generally longer for research assessments and it was not unusual for children to become tired during testing. These issues may not have been taken into account by the assessors and, specifically, the objective scoring of a standardised assessment would not have made allowance for underperformance because of these factors.

There are several possible reasons for the higher sensitivities for diagnosing motor impairment than for cognitive or language impairments at routine NHS assessments. Important motor developmental milestones (e.g. sitting and walking) are reached at a relatively young age and parents and health professionals place great emphasis on checking that children achieve these milestones. Cerebral palsy is the most commonly quoted morbidity of preterm birth; therefore, motor assessments are regularly performed at follow-up appointments. Cognitive and language skills can be difficult to ascertain in a single setting, particularly without the use of standardised assessment tools, and can be affected by the issues of judgement and reporting bias discussed above. Furthermore, in the modified NPEU/Oxford classification, the categorising of cognitive impairment by ‘number of months behind corrected age’ introduced another level of variability. In addition, in the electronic ‘2-year outcome’ form, the term ‘development’ was used as the heading for the cognitive domain. As a result, there was misinterpretation among the NHS assessors that the questions in that category applied to ‘overall development in all domains’ rather than being specific to cognitive function, potentially leading to misclassification.

The impact of inter-rater and intrasubject variability would be exacerbated by the classification of neurodevelopment skills, a continuous trait, into categories of ability or impairment. Levels of abilities or skills near the ‘cut-off’ point between categories are more difficult to discriminate and are at risk of being misclassified into higher or lower impairment categories.

Subgroup analyses were used to investigate whether or not the validity of the NHS assessment was affected by neonatal and sociodemographic factors, as well as factors related to the conduct of the assessments. However, given that the numbers of children with impairment (‘true positives’) within each subgroup were small, it was likely to that subgroup analyses were underpowered. Therefore, the possibility remains that the negative findings were a reflection of type II errors (‘false negatives’).

The Bayley-III was selected as the research assessment as it is the most commonly used assessment in neonatal outcome studies. However, the validity of the Bayley-III, particularly in identifying school-age outcomes, is unknown. Several studies have raised concerns that, when compared with the BSID-II, the Bayley-III was underestimating neurodevelopmental impairment. 202–205 It is, however, reassuring that validation studies of the Bayley-III showed that the scores are consistent with other revised ability tests such as the Wechsler Preschool and Primary Scale of Intelligence – Third Edition309 and the Preschool Language Scale – Fourth Edition. 201,310 In this study, more children were classified as being impaired using the predicted BSID-II MDI scores than using the Bayley-III scores if the same threshold were applied. Therefore, as expected, the sensitivities of the NHS assessment dropped when the predicted BSID-II MDI scores were used as the ‘gold standard’ instead of the Bayley-III scores.

Another issue that needs to be considered is the impact of administering the English-based Bayley-III assessment on cognitive and language scores in children whose primary language is not English. Although families who require interpretation for English were excluded, 52% of the study population was living in a bilingual environment. Studies that examined the effect of bilingualism on language acquisition have provided conflicting evidence,311–313 but testing bias cannot be ruled out. However, for a child who is functioning in the ‘severe impairment’ category, communication skills are assessed by the observation of gestures and the production of consonant and/or vowel sounds, which are not language specific, and, hence, the assessment of children with severe language impairment is likely to be valid. Testing bias can also occur in NHS assessments where there is likely to be greater reliance on parental reporting.

Social communication skills of children who were born very preterm

At 24 months corrected age, children who were born before 30 weeks’ gestation were rated by their parents on the Q-CHAT as having greater social communication difficulties and autistic traits than of the general population. The higher frequency in autistic traits was observed mainly in the areas of restricted, repetitive, stereotyped behaviour, communication and sensory abnormalities.

Previous studies have reported significantly higher odds of positive autism screening on the M-CHAT in children with motor, visual, hearing and cognitive impairments. 213,214 The Q-CHAT contains similar questions, leading to children with such disabilities receiving higher Q-CHAT scores. Therefore, it is likely that the distribution of Q-CHAT scores in a very preterm population would be even higher if children with cerebral palsy and severe neurosensory disabilities were included.

There was no sex difference in our study population. This may be due to insufficient statistical power, given that a sample size of 24,000 children would be required for the 0.3-point sex difference in Q-CHAT scores that we detected to be significantly different. Nevertheless, it has been suggested that the autistic phenotype seen in preterm children resembles more closely syndromic or medically explained autism, the sex ratio of which is closer to 1 : 1, than those with idiopathic autism,213 supporting the hypothesis that autism in preterm children, rather than being a primary deficit, represents part of a ‘preterm phenotype’ with different aetiology.

Preterm children experience difficulties across all aspects of autistic behaviour but particularly in the categories of restricted, repetitive, stereotyped behaviour, communication and sensory abnormalities. The presence of reduced language abilities among children who were born preterm is well described. 176,177 Dysfunction in sensory modulation in preterm children, characterised by either hyposensitivity or hypersensitivity to sensory input, is a problem anecdotally recognised by parents and clinicians. It is hypothesised that exposure to the stressful environment of the neonatal intensive care unit at a critical period of brain development in the third trimester interferes with the normal maturation of the sensory system. 314,315 Sensory modulation dysfunction is thought to be negatively associated with emotional development and can affect social interactive capabilities. 316

There is some evidence that restricted and repetitive behaviours are associated with cognitive status. 317,318 EPICure study investigators also concluded that cognitive deficits in their extremely preterm cohort accounted for the excess of repetitive and stereotyped behaviour when compared with the full-term controls. 319 Although there was no correlation between cognitive scores and subcategorical Q-CHAT scores in the restricted and repetitive behaviour domain, as the mean cognitive score of the preterm population was lower than would be expected in the general population. The potential association between cognition and restricted and repetitive behaviour could, in part, explain the higher Q-CHAT scores obtained by the participants in this category.

There were fewer differences between preterm children and the general population in response to items exploring social relatedness. Q-CHAT items exploring social relatedness may provide a higher degree of specificity for differentiating early autistic features from concurrent developmental delay in children without severe physical and neurosensory impairment compared with items in the other categories. Although parents reported a lower frequency of pretend play among the preterm children, development in joint attention (elucidated by questions on protodeclarative pointing and following a gaze) was similar in the general population. Focusing on elucidating social relatedness for autism screening in the preterm population may reduce the ‘false-positive’ screening rate associated with currently available screening tools.

The significant association between language ability at the age of 2 years and Q-CHAT scores was unsurprising, as four items on the Q-CHAT specifically examined language development. Furthermore, language ability was closely related to cognitive function, which, in turn, influenced performance on other Q-CHAT items. Separate cognitive and language scores were obtained from the Bayley-III assessment in this study. Language scores confounded and accounted for the association observed between cognitive scores and Q-CHAT scores.

This study also highlights the inter-relationship between ethnicity, area deprivation, language skills and Q-CHAT scores. Our findings suggest the possibility of an environmental impact of socioeconomic disadvantage on early social communication development.

The Q-CHAT, M-CHAT and the Bayley-III Social-Emotional are some of the developmental surveillance tools designed to identify toddlers at risk for developing ASD, with the aim of implementing timely intervention strategies to achieve better outcomes for these children. The M-CHAT has a sensitivity of 87% and specificity of 98% for ASD when applied in a mixed sample of children aged between 16 and 30 months. 211 The Bayley-III Social-Emotional questionnaire, using a scaled score of 6, reportedly had a sensitivity of 87.0% and specificity of 90.0% for the identification of ASD. 207 However, the predictive validity of these screening tools when applied to the preterm population has not been investigated. Furthermore, there is little understanding of the differences in properties of the available screening tools. Oosterling et al. 320 compared four instruments: the Early Screening of Autistic Traits Questionnaire;321–323 the Social Communication Questionnaire;323 the Communication and Symbolic Behaviour Scales-Developmental Profile, Infant-Toddler Checklist;216 and key items of the Checklist for Autism in Toddlers. 324 They found that no particular tool showed superior discriminating power for distinguishing children with ASD from those without.

Meta-analysis

The specificities of early neurodevelopmental assessment in predicting later school-age cognitive outcomes were generally high, especially for severe cognitive impairment, but sensitivities were inconsistent. Early neurodevelopmental assessment has low sensitivity and high specificity for identifying school-age cognitive deficit. This means that, when a neurodevelopmental impairment was diagnosed at ages 1–3 years, the likelihood of having cognitive deficit at school age was high (low false-positive rate, or 1 – specificity). However, it would not be possible to exclude later cognitive deficit even when an early assessment demonstrated normal neurodevelopmental outcomes (high false negative; or 1 – sensitivity). The results suggest that almost half the children who were thought to have normal neurodevelopmental function at ages 1–3 years will experience cognitive difficulties at school age. Even for cases of severe cognitive deficit, the accuracy in early detection was low (meta-analytic sensitivity of 39.2%). This finding is not unexpected. Cognitive function in infancy is a poor predictor of later IQ in the general population. 325 This may reflect changes in cognitive function during childhood, unveiling of deficits in complex task performance that are non-essential in early childhood, or the increasing effect of social and environmental influences on cognition over time. Other explanations may be the impact of behaviour and attention during testing at different ages, and differences in the contents and psychometric properties of early neurodevelopmental and later cognitive assessment tools.

The internal validity of this study is influenced by the quality of the data from the included studies as well as by the methods adopted. Data quality as appraised by the QUADAS-2 tool was good, with most studies considered to be at a low risk of bias. Nevertheless, the presence of missing data from participants who were lost to follow-up over time is a common problem affecting these longitudinal studies. Incomplete outcome ascertainment can distort the result in either direction. Another source of missing data arose from the exclusion of children with severe neurosensory and motor impairment who were unable to complete the assessments. If we assume that these children had stable diagnoses of severe neurodevelopmental and cognitive deficits throughout childhood, then the impact of excluding them from the study population would be an underestimation of the sensitivity of early neurodevelopmental assessments.

An additional bias that could affect accuracy, and which was not identified through the QUADAS-2 appraisal, is the experience of the assessors. Although all included studies employed trained assessors using standard assessment tools, interobserver differences are inevitable. Neurodevelopmental and cognitive abilities exist as a continuum but, for the purpose of the study, participants were dichotomised using a ‘cut-off’ score into groups ‘with impairment’ and ‘without impairment’. Interobserver variations around the ‘cut-off’ score would result in misclassification of outcomes. The effect of differential misclassification on the study results is difficult to predict but in general it can be expected to have a bigger impact on sensitivity, which is calculated using a small number of ‘positives’ in this condition of relative low prevalence, than on specificity, which is based on a large number of ‘negatives’.

Participants were included in the review if they fulfilled either the gestational age or the birthweight inclusion criterion. The birthweight criterion was used in order to capture all relevant studies, as it has been common for neonatal studies to base eligibility on birthweight rather than gestational age. However, the methodological bias in using a birthweight criterion is the inclusion of more mature but growth-restricted children. Notably, in the study by Bassan et al. 278 all the participants were small for gestational age (birthweight < 10th percentile for gestational age). Intrauterine growth restriction is a risk factor for poor neurodevelopmental outcome. 326 The QUADAS-2 appraisal highlighted the lack of applicability of older study populations and outdated assessment tools in more than half of the included studies and, hence, raises the question on the wider generalisability of the study findings. This is, of course, a reflection of the nature of all longitudinal studies but it is a significant limitation, particularly in the context of a rapidly advancing neonatal specialty. The past couple of decades have seen an overall reduction in the proportions of survivors of very preterm birth with adverse neurodevelopmental outcomes at the age of 2 years;231,327,328 therefore, we can expect the characteristics of the current preterm population to be different to those from past eras. Only 14 of the 24 included studies recruited participants born after 1990 and none was born in the previous 10 years (i.e. after 2004).

More importantly, the assessment tools used in the included studies, although validated and contemporary at the time of each study, have mostly been superseded by newer editions. Therefore, caution should be exercised when extrapolating results based on earlier versions of assessment tools to current practice. The timing and setting of the assessments also played a part in determining the external validity of the study findings. The early neurodevelopmental assessments were performed between 12 and 36 months and the timings matched common clinical practice. School-age assessments were mostly conducted between the ages of 5 and 8 years, when children were at the primary stages of schooling. Only three studies reported cognitive assessment during adolescence, one of which had only 20 participants. Therefore, the validity of early assessment in diagnosing cognitive deficit extending into adulthood could not be estimated from this study, although one could speculate that the sensitivity might be even poorer. As the sensitivity estimates from individual studies were based on a small number of participants with cognitive impairment, the corresponding 95% CIs were very wide. The use of a meta-analytic approach increases the sample size and improves the precision of the pooled estimate.

The review was restricted to English-language literature. There is concern that the English-language journals publish a skewed sample of studies that report positive and more noteworthy results. 329 Similarly, it is common for articles with negative or inconclusive findings to remain unpublished. The exclusion of grey literature, including abstracts and dissertations, could have led to the omission of essential and more recent information.

Heterogeneity between studies was investigated using metaregression. This method has a few drawbacks. The statistical power to detect associations between the study estimates and the explanatory variables is related to the magnitude of the relationship between them, and is typically considered low in metaregression. 330 This was compounded by the narrow range of values available for each of the explanatory variables under evaluation. For example, the mean gestational age of the included studies ranged only between 25.9 and 33.1 weeks. Hence, a type II error could not be excluded. More importantly, metaregression is subject to ecological fallacy (or aggregation bias) (i.e. the mistaken assumption that a statistical between-study relationship based on aggregated data reflects a within-study relationship). Therefore, in order to reliably identify factors that influence the validity of early developmental assessments, it would be necessary to obtain individual patient-level data.

In conclusion, early neurodevelopmental assessment has high specificity but low sensitivity in identifying later school-age cognitive deficit.

Clinical relevance of results

Routine NHS assessments have low sensitivity for identifying mild to moderate neurodevelopmental impairment. This has significant clinical implications. At an individual level, children with impairment may be missed. At a population level, current documentation of 2-year outcomes during routine NHS assessments, using the standardised EPR in its present format, will underestimate the proportion of children with impairment, compared with a research-standard Bayley-III assessment. Many neonatal networks and units rely on routine follow-up for impairment rates of their graduates. The results of this study question the validity of these practices.

The findings of higher Q-CHAT scores in the preterm population suggest that suboptimal development of social communication skills exists from early childhood. The 7-point right-shift in mean Q-CHAT score of the preterm population corresponds to nearly 1 SD difference. As ASD exists on a continuum, with autism representing the extreme end of the spectrum, the results also support the likelihood that a large proportion of preterm children experience clinically significant social communication difficulties below the diagnostic threshold for ASD from a young age, when early intervention may be possible. The findings draw attention to the need for better understanding and potentially early assessment of social communication skills in the preterm population.

The results from the systematic review and meta-analysis confirm that a significant proportion of children who were born very preterm and who are assessed as having normal neurodevelopment in early childhood go on to experience cognitive difficulties later in school. The implications of this finding on current clinical practice are considerable because neurodevelopmental assessment at 2 or 3 years of age is often used as the end point for post-discharge follow-up of very preterm infants. Outcome data used in discussions with parents during the antenatal and neonatal periods are commonly based on neurodevelopmental outcomes determined in early childhood. Given these findings, it is essential to discuss potential difficulties at school that children may face, even in the absence of obvious impairment or disability at the 2-year assessment.

Reassuringly, the false-positive rate for early diagnosis of impairment was low, indicating that children with more severe impairments, who would receive greater benefit from early intervention, will be correctly identified.

Implications for health care

There are advantages in the current practice of embedding neurodevelopmental follow-up of very preterm children with neonatal services. These include the continued involvement of health professionals known to the families and local flexibility in organisation. However, this also risks regional variation in follow-up criteria, reliability assessments and quality of data recording. Standardising the neurodevelopmental tool and ensuring that staff are trained in its use during follow-up assessment would be an obvious way of minimising some of this variability. Such a tool should have strong psychometric properties, be user friendly and, ideally, be adaptable for use in non-English-speaking patients. Since this study, the NICE guideline331 for developmental follow-up of children and young people born preterm has been published. The guideline recommends that the PARCA-R parent-completed questionnaire is used to identify children at risk of developmental delay. Misclassification occurs during categorisation of outcomes; hence, the strategy of presenting outcome data in categories should also be further considered. Categorical outcomes are easy to interpret and to communicate, and mirror clinical practice (e.g. referral of children below a certain threshold for further assessment or intervention). However, for an individual child, the labelling of ‘outcome category’ is unhelpful. Besides, as shown, categories of outcomes do not remain stable over time. It is, arguably, more valuable to present the distribution of standardised scores.

Under the UK Healthy Child Programme, all children receive health visitor-led developmental screening. There has been some interest in extending the roles of health visitors to capture developmental outcome data of children who were born preterm, assessed using developmental screening tools or through questions similar to those listed on the electronic ‘2-year outcome forms’ (NPEU/Oxford criteria). 332 The findings of this study indicate a need for caution in this approach. Even with the use of developmental screening tools, the false-negative rates (sensitivities) are unacceptably high. Other factors, such as shortage of health visitors, requirement for further training and lack of universal uptake of the screening programme, might further limit success.

Based on the findings of this study, a centralised approach to the assessment and recording of 2-year outcome data for children who were born very preterm is worthy of consideration. Typically, very preterm children are offered post-discharge appointments every 3 to 6 months; these visits might be conducted at hospitals where allied health professional support (e.g. dietetics, physiotherapy) can be sought if necessary. There may be advantages for the 2-year neurodevelopmental assessment to be organised at neonatal network level, as this could ensure that each child receives assessment by an appropriately trained team of health professionals using standardised tools, and could benefit from centralised, co-ordinated administrative support to trace and contact families.

In addition to being of high quality, the data recorded during clinical care should be complete to enable meaningful analysis. Currently, the utility of the routinely recorded electronic clinical data as a source of population-based outcome information is limited by poor data completeness. According to the National Neonatal Audit Programme report, 2-year outcome data were available from only 44% of all infants born before 30 weeks’ gestation in England and Wales between July 2010 and June 2011. Strategies to reduce missing data need to be aimed at clinician engagement.

Since 2007, the American Academy of Pediatrics (AAP) has recommended ASD-specific screening at 18 months for all children to facilitate early diagnosis and to prevent delay in the initiation of early intervention. 333 The UK National Screening Committee does not currently recommend universal screening, on the basis that none of the available screening tools has sufficient reliability in identifying children at risk for ASD when applied to the general population. 334 Regardless of an ASD diagnosis, toddlers who were born preterm experience problems in current functioning that may interfere with adaptive exploration and social engagement. It is important that clinicians recognise these difficulties and the impact that they have on families. Parents require information on the social communication difficulties that preterm children experience, particularly as some of these behaviours may be amenable to specific interventions, such as speech and language, occupation and sensory integration therapies, as well as educational programmes targeted at enhancing communication, social skills instruction and reducing interfering maladaptive behaviours. 335

Improve the electronic ‘2-year outcome’ form

In the absence of the standardised use of a single assessment tool to allow comparison of outcomes between centres, improvements to routine data recording could be sought. On the electronic ‘2-year outcome’ form, the documentation of outcomes in the cognitive and language domains is more subjective than in the motor domains. It is possible that, by modifying the form to increase the objectivity of the items recorded, the validity of the data would be improved. For example, for the cognitive domain, it may be possible to identify a standardised set of cognitive test items, perhaps from the Bayley-III assessment or other tools that can be easily administered in a clinical setting. Language function can be ascertained by determining if a child can identify or say words from a list of commonly expressed words.

The NICE guidelines331 for developmental follow-up of children and young people born preterm recommend the use of the validated parent questionnaire PARCA-R to identify children at risk of global developmental delay, learning disability or language problems, and for the PARCA-R scores to be documented in the NNRD. The predictive validity of the PARCA-R at the age of 2 years in identifying impairments at a later age and special educational needs to be evaluated. The guidelines also recommend using different approaches, such as e-mails or text messages, to provide enhanced developmental support. The utility of these approaches in assessment and outcome data acquisition should also be explored.

Comprehensive behavioural assessment and identification of risk factors for ASD in the preterm population

As stated in Types of neurodevelopmental outcome measures, preterm children are at a higher risk of a range of behavioural problems, including ADHD and internalising behaviour, that were not examined in this study. The age at emergence of these behavioural difficulties is unclear and should be examined in future studies. Future studies would also benefit from an examination of a more comprehensive set of neonatal and environmental variables in order to identify potential moderators and mediators of risk for ASD and other behavioural difficulties. It would then be possible to develop risk scores or risk prediction models that could aid in early diagnosis and initiation of interventional therapies. Future studies will also need to focus on the challenges faced in the early assessment of behavioural features of children with major functional disabilities and of children in non-English-speaking groups.

Linkage with school-age outcome data

Currently, there is no process or provision in the UK for continuing formal follow-up assessment beyond early childhood. Long-term programmes require significant manpower and financial investment and the likelihood of high attrition rates will further jeopardise success. Therefore, it is worthwhile considering other sources of school-age outcome data, for example primary care or community child health records, or educational data. UK national structures provide a unique opportunity for data linkage. Future research could investigate the utility of these data sources through linkage of neonatal data with later outcomes.

Overview

The study population for this empirical investigation comprised infant participants in the Probiotic in Preterm babies Study (PiPS). For each study infant, health-care resource utilisation was measured using three primary data sources: (1) the PiPS trial CRFs, (2) the NNRD; and (3) a data source that combined information from both PiPS trial CRFs and the NNRD.

Resource inputs captured by each data source were primarily valued using national tariffs and expressed in Great British pounds (GBP) (2012/13 prices). In our empirical investigation we sought to estimate (1) the level of agreement for hospital resource utilisation and costs between the alternative data sources and (2) the level of precision of incremental cost-effectiveness for the probiotic evaluated in PiPS using alternative data sources.

PiPS trial: design

PiPS was a multicentre, double-blind, placebo-controlled, randomised trial of probiotic administration in infants born between 23⁺⁰ and 30⁺⁶ weeks gestational age. Infants were recruited within 48 hours of birth from 24 hospitals within 60 miles of London over a 37-month period from July 2010 onwards. They were randomised to either the probiotic (given in a daily oral dose of 8.3–8.8 log₁₀ colony-forming units) or the placebo (provided as an identical powder in identical sachets, until 36 weeks postmenstrual age or discharge from hospital, if sooner). There were three primary outcomes: any episode of neonatal NEC Bell stage II or III;112 any positive blood culture of an organism not recognised as a skin commensal on a sample drawn > 72 hours after birth and < 46 weeks postmenstrual age or discharge if sooner (hereafter sepsis for brevity); and death before discharge from hospital. Secondary outcomes included a composite of the three primary outcomes. The trial was sized (n = 1300) to detect a 40% relative risk reduction from 15% to 9.1% for each of the primary outcomes at a two-sided significance level of 5% and with 90% power. PiPS was approved by a national research ethics committee and co-ordinated by the NPEU, University of Oxford. Further details about PiPS, sampling procedures, methodology, outcome measures and responses rates are reported in full elsewhere. 342

PiPS trial: measurement of resource use and costs

A comprehensive profile of resource inputs was integrated at the outset into the PiPS trial CRFs. There were four main trial CRFs: (1) form 1: entry, (2) form 2: daily data collection, (3) form 3: transfer/discharge and (4) form 4: abdominal pathology. The bulk of the relevant resource inputs were captured by the second and third of these trial CRFs. The forms captured a comprehensive profile of resource use by each infant, encompassing length of stay by intensity of care, surgeries, investigations, procedures, transfers and post-mortem examinations until final hospital discharge or death (whichever was earliest). Resource inputs were primarily valued based on data collated from secondary national tariff sets343,344 (Table 31). All costs were expressed in GBP and reflected values for the financial year 2012/13.

The total length of stay (total inpatient hospital days) was computed as the total number of hospital days until final discharge to home or death. Information was available on time spent in the neonatal unit by level of care (normal, transitional, special, high dependency or intensive). The cost of routine neonatal care was calculated for each infant by multiplying the length of stay by intensity (intensive, high dependency, special care, transitional) by the per diem cost of the respective level of care using data from the NHS Reference Costs trusts schedule 2012/13. 343 Non-routine investigations excluded from these per diem costs were valued using a combination of primary and secondary costs. The costs of surgeries were calculated by assignment of surgical procedures to relevant Healthcare Resource Group (HRG) codes and application of unit costs from national tariffs. Transfers were recorded whenever an infant was transported between specialist hospitals for neonatal critical care, and were valued using costs from the NHS Reference Costs trusts schedule 2010/11,336 and inflated using a health care specific pay and prices index to 2012/13 prices. Post-mortem costs were based on data from secondary sources. 345 Where costs of additional non-routine investigations excluded from per diem values for neonatal care were not available from national tariffs, clinicians were asked to identify the staff and material inputs required for these investigations. Staff time was valued using the Unit Costs of Health and Social Care 2012344 tariffs.

Linkage and data extraction from the National Neonatal Research Database

In order to compare resource use, cost and cost-effectiveness estimates based on data solely extracted from the PiPS trial CRFs and data solely extracted from the NNRD, a NNRD extract was created for infants participating in the PiPS trial. The NNRD has been created through the collaborative efforts of neonatal services across the country to be a national resource. The NNRD contains a defined set of data items (the Neonatal Data Set) that have been extracted from the Badger.net neonatal EPR of all admissions to NHS neonatal units. Badger.net is managed by Clevermed Ltd, an authorised NHS hosting company. The Neonatal Data Set is an approved NHS Information Standard (SCCI1575). Contributing neonatal units are known as the UK Neonatal Collaborative.

The trial co-ordinating centre for the PiPS trial, namely the NPEU at the University of Oxford, provided the NDAU at Imperial College with the final PiPS data set. This included data on 1310 of a total of 1315 infants recruited into the PiPS trial (five infants were excluded because of withdrawals from the study). Clevermed Ltd, the NHS hosting company, which separately receives data from individual neonatal units, was able to match Badger IDs to NHS numbers for 1280 (98%) of the 1310 infants (Figure 25). These 1280 infants had 2360 episodes of care which were linked to episodes in the NNRD using the Badger ID and hospital name. Episodes were renumbered on both databases as necessary to provide linkage. A total of 81 episodes that were effectively the second episode of care on the NNRD were renumbered as the first episode to match the PiPS episodes, because the infants were recruited into the PiPS trial after transfer from the neonatal unit at the hospital of birth. Similarly, 103 episodes of care that existed on the PiPS database but not on the NNRD were excluded from comparison; these largely occurred in non-Badger hospitals or wards. This resulted in the exclusion of all episodes of care from 22 infants, leaving 2257 episodes of care from 1258 infants eligible for our comparative analyses (see Figure 25).

FIGURE 25.

Linkage between PiPS trial data and the NNRD.

A comprehensive profile of resource use between randomisation into the PiPS trial until final hospital discharge or death (whichever was earliest) was compiled from the NNRD for the 1258 infants eligible for our comparative analyses. Our direct comparisons of resource use estimates between the PiPS and NNRD data sources required a further process of data manipulation to reconcile definitional and labelling differences for individual variables between the data sources, and coding differences by episodic and infant level. For example, information on surgery for PDA, medical treatment of PDA using ibuprofen or indometacin, and ROP treatment whether by laser or cryotherapy, can appear in multiple locations in the NNRD (e.g. discharge diagnoses, daily data, ad hoc forms). An overall summary of the interventions received during linkable episodes of care was generated on a ‘by infant’ rather than episodic level.

Statistical methods

A comprehensive statistical analysis plan was followed. All statistical analyses were performed using Stata or R (version 2.01).

The clinical and sociodemographic characteristics of the PiPS participants who were (n = 1258) and were not (n = 52) included in our comparative analyses of resource use, costs and cost-effectiveness were compared using the chi-squared test. Differences in resource use and costs, by category, were estimated (1) within trial by data source, thereby allowing us to draw comparisons between the probiotic and placebo arms, and (2) by pooling data between the trial arms, thereby allowing us to draw comparisons between the alternative data sources. In addition to data solely extracted from the PiPS trial CRFs and data solely extracted from the NNRD, we created a third data source for these comparative analyses. The third data source, hereafter termed the ‘combined’ data source for brevity, was constructed by selecting the preferred data source for each resource variable in terms of volume and granularity of information provided. It broadly followed the processes described in Chapter 4. The selection process for each resource variable was undertaken by the clinical investigators (KC, CB). For comparisons within trial by data source, differences in resource use and costs, by resource category, were tested using the independent-sample t-test for continuous variables, the chi-squared test for categorical variables and the Mann–Whitney U-test for medians. For comparisons between the alternative data sources, the levels of agreement in resource use and cost estimates, by category, for alternative combinations of data sources (PiPS vs. NNRD, PiPS vs. combined, NNRD vs. combined) were estimated using the Lin concordance correlation coefficient. 346 This statistic measures the agreement between two continuous variables obtained by two methods; the value of the statistic lies between 1 (perfect agreement) and –1 (perfect inverse agreement). A threshold 0.40 value for the statistic was adopted to indicate acceptable clinical or practical significance. 347 In addition, we estimated mean differences and 95% CIs to identify potential systematic biases, and the 95% limits of agreement, indicating random variation between individual measurements. 348

We additionally performed an economic evaluation of the probiotic. For comparisons within trial by data source, the economic evaluation took the form of an incremental cost-effectiveness analysis in which we estimated the incremental costs (ΔC) and incremental effects (ΔE) attributable to the probiotic in very preterm infants, with reference to the placebo. The results were primarily expressed each in terms of an incremental cost-effectiveness ratio (ICER) (ΔC/ΔE). Estimates of incremental cost-effectiveness were made for each of the three primary clinical outcomes (any episode of NEC Bell stage II or III, any case of sepsis, death before discharge from hospital), and for the composite secondary outcome. The economic evaluation was conducted from a health system perspective. 339 The time horizon for the economic evaluation was the period between trial randomisation and final hospital discharge or death, whichever was earlier. Non-parametric bootstrapping, involving 1000 bias-corrected replications of each of the ICERs, was used to calculate uncertainty around all cost-effectiveness estimates. This was represented on four quadrant cost-effectiveness planes. 349 Decision uncertainty was addressed by estimating net benefit statistics and constructing cost-effectiveness acceptability curves (CEACs) across cost-effectiveness threshold values (λ) of between £0 and £70,000 for the health outcomes of interest. The probability that the probiotic is less costly or more effective than the placebo was based on the proportion of bootstrap replicates that had negative incremental costs or positive incremental health benefits, respectively. A series of prespecified subgroup analyses repeated all analyses by selected subgroups (sex, birthweight, gestational age, colonisation status, randomisation age) for the primary and secondary cost-effectiveness outcomes.

For comparisons of cost-effectiveness outcomes between the alternative data sources, we estimated the overall probability of miscoverage of incremental net monetary benefit based on resource use data solely extracted from the NNRD and resource use data solely extracted from the PiPS trial CRFs. In order to estimate miscoverage for incremental net monetary benefit, the bootstrap replications of each of the ICERs were rearranged on a linear scale using the formula:

The miscoverage statistic was estimated as the percentage of bootstrap samples of incremental net monetary benefit that fell outside the CI for the reference data source. 350 For the purpose of these analyses, the combined data source acted as the referent, although we additionally assumed that the PiPS data source acted as a referent for analyses of data solely extracted from the NNRD. This was replicated for the primary and secondary outcomes of interest and for all prespecified subgroup analyses.

Finally, we performed sensitivity analyses that compared the key outputs of the economic evaluation using either resource use data and clinical outcomes extracted solely from the PiPS data set or resource use data and clinical outcomes extracted solely from the NNRD. These analyses were restricted to the two clinical outcomes used in the PiPS trial that were available in both data sources, namely (1) death before discharge from hospital and (2) sepsis. They acted as exemplars of the likely differences in economic outcomes that will be observed if we rely on the NNRD as a complete source of information (resource inputs, clinical outcomes) for an economic evaluation. The cost-effectiveness outcomes considered by these sensitivity analyses included estimates of incremental cost-effectiveness, probabilities of cost-effectiveness for the probiotic at alternative cost-effectiveness thresholds, and miscoverage of incremental net monetary benefit against the PiPS trial referent.

Study population

A total of 1315 infants were recruited from 24 hospitals within 60 miles of London over 37 months, from July 2010 onwards. Data for 1310 infants were available for analysis in the PIPS trial. Of these 1310 infants, 52 infants were excluded from our empirical investigations of either because failure to match their Badger IDs to their NHS numbers or because they received part of their neonatal care in non-Badger hospitals or wards. A total of 1258 infants were therefore eligible for our comparative analyses. There were no significant differences between the baseline clinical and sociodemographic characteristics of the 1258 infants included in the analyses and the 52 infants excluded from the analyses, regardless of the use of NNRD or PiPS data (Table 32).

The key clinical and sociodemographic characteristics of the 1258 infants included in our empirical investigations are presented by trial arm in Table 32. There were no significant differences in these key characteristics between the trial arms. Furthermore, there was no evidence of clinical benefit associated with administration of the probiotic for any of the primary outcomes or the composite secondary outcome (Table 33).

Resource use and cost estimates: comparisons within trial by data source

Resource use measures and their values between trial randomisation and final hospital discharge (or death) are summarised by trial arm in Tables 34 and 35 for the PiPS and NNRD data, respectively. Based on the PiPS data (see Table 33), the mean (SE) overall duration of hospitalisation was 75.49 (1.95) days for infants in the control arm, whereas the infants in the probiotic arm had a mean overall duration of hospitalisation of 76.60 (2.02) days. There were no statistically significant differences in the values for any resource input by trial arm. The results for the NNRD data (see Table 35) followed a similar pattern with no statistically significant differences in resource values by trial arm. However, the mean numbers of cranial ultrasound scans were higher in the PiPS data than in the NNRD data. Table 36 presents the resource use measures and their values by trial arm for the combined data set.