Development and validation of prediction models for fetal growth restriction and birthweight: an individual participant data meta-analysis

Article history

The research reported in this issue of the journal was funded by the HTA programme as project number 17/148/07. The contractual start date was in May 2019. The draft report began editorial review in January 2022 and was accepted for publication in September 2022. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Copyright statement

Copyright © 2024 Allotey et al. This work was produced by Allotey et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This is an Open Access publication distributed under the terms of the Creative Commons Attribution CC BY 4.0 licence, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. See: https://creativecommons.org/licenses/by/4.0/. For attribution the title, original author(s), the publication source – NIHR Journals Library, and the DOI of the publication must be cited.

2024 Allotey et al.

Primary

1. To establish whether existing prediction models for FGR and birthweight are suitable for the target population or if new models are needed through external validation, and where possible, recalibration of existing prediction models.

2. Using IPD meta-analysis, to develop and validate [using internal-external cross-validation (IECV)] new multivariable prediction models for (i) FGR (SGA with serious perinatal complications) (IPPIC-FGR Model 1); and (ii) birthweight at various potential gestational ages at delivery (IPPIC-birthweight Model 2) based on:

   1. – clinical characteristics only

   2. – clinical and biochemical markers

   3. – clinical and ultrasound markers

   4. – clinical, biochemical and ultrasound markers.

Secondary

1. To compare the predictive performance of models according to (i) population (selected – high/low risk; unselected); (ii) trimester of testing (first <14 weeks; second ~20 weeks; third ~28 weeks); (iii) choice of predictors (clinical only; clinical and ultrasound; clinical and biochemical; clinical, ultrasound and biochemical; and (iv) onset of FGR (early <32 weeks; late ≥32 weeks).

2. To assess if the performance of the prediction models is generalisable for various definitions of FGR such as (i) ultrasound parameters determined by Delphi consensus;37 and (ii) birthweight <10th centile adjusted for gestational age with associated neonatal morbidity,38 and assess the association between various birthweight centiles (<10th, <5th, <3rd centiles) calculated using (i) customised and (ii) population-based standards, and perinatal mortality and morbidity.

3. To examine the clinical utility of the prediction models using decision curve analysis (DCA).

4. To assess the costs and outcomes and the potential impact of resource use of the prediction models.

The International Prediction of Pregnancy Complications Network

The IPPIC Network database is a living data repository of IPD from pregnant women. Methods on how cohorts within the IPPIC Network database were identified and harmonised have been described in detail in our earlier publications. 36,44,45 Briefly, cohorts within the database were identified through a systematic search to identify primary studies reporting risk factors for pregnancy complications including pre-eclampsia (PE), stillbirth and FGR. 46 Authors of relevant studies were invited to join the network and share their primary IPD in any format, along with data dictionaries or descriptions. The data were deposited in a custom-built database, formatted, cleaned and harmonised, and the quality of each cohort and its IPD was assessed using the following domains of the Prediction study Risk of Bias Assessment (PROBAST) Tool: participants (adequate description of data sources, details on recruitment), predictors (appropriately defined, assessed blinded to outcome, assessed in the same way for all participants) and outcome (appropriately defined and determined in a similar way for all participants, predictors excluded from the outcome definition, outcome determined without knowledge of predictor information and appropriate interval between assessment of predictor and outcome determination). 47

The IPPIC-IPD included data from large birth or population-based cohorts, registry data, unpublished data from hospital records or study cohort data. Study population varied from low to high risk of development of complications. The predictor variables harmonised within the IPPIC Network repository are those that are easy to obtain in a clinical setting, as agreed by the collaborative group. 44 The network currently includes more than 150 collaborators from 26 countries, contributing IPD from over 3 million pregnancies, reporting maternal characteristics, obstetric history, clinical assessment and tests, as well as various maternal and offspring outcomes. Cohorts that addressed the structured question in Table 1 were considered for inclusion in the IPD meta-analysis.

Primary outcomes

The primary outcomes were (1) FGR (birthweight <10th centile adjusted for gestational age, with stillbirth, neonatal death or delivery before 32 weeks); and (2) birthweight for deliveries at various gestational ages to reflect the extent of the restricted growth.

Updating literature search

Prioritisation of predictors

We carried out a prospective two-round e-survey of IPPIC Network collaborators, to prioritise the most clinically relevant predictors of FGR, to be considered in the development of the prediction models. Predictors were identified from existing systematic reviews. 22 The first round of the survey included explanation of the study and consent process, followed by a list of predictors identified from the systematic reviews. Collaborators were asked to rank the importance of each predictor variables identified on a scale from 1 (not important) to 5 (very important). The predictors were classified as ‘consensus in’ if ≥70% of responders gave a score of 4 or 5 and <15% score 1 or 2, or ‘consensus out’ if ≤50% of responders gave a score of 4 or 5 and ≤30% gave a score of 3. Anything else was classified as ‘no consensus’.

In the second round of the prioritisation survey, collaborators were invited to a video Zoom conference on 28 July 2020 and asked to reassess predictors ranked as ‘consensus in’ or ‘no consensus’ from the first round of voting. An open discussion took place on each outcome and collaborators were encouraged to consider how important the measurement of each predictor was as a predictor of FGR. The Zoom polling function was used to vote on a scale from 1 (not important) to 5 (very important) for each predictor and analysed using the same method in the round one survey. Any variable still classed as ‘no consensus’ was discussed at the meeting and a final classification agreed upon.

Sample size considerations

The effective sample size for the development and validation of prediction models is driven mainly by the total number of events (for logistic regression of a binary outcome) or the total number of subjects (for linear regression of a continuous outcome). To reduce the potential for overfitting and optimism during model development, the number of subjects/events must be large relative to the number of candidate predictor parameters to be considered for inclusion in the model.

For the external validation of published prediction models, sample size calculations aim for precise estimates of the predictive performance,53–55 and suggest at least 100 events and 100 non-events for binary outcomes, which we hoped to meet – though again our sample size was fixed, based on the IPD available that recorded the required predictors available for each model. 54

The IPPIC-FGR model to be developed has the binary outcome of FGR (birthweight <10th centile adjusted for gestational age, with serious perinatal complications). For this, Riley et al. proposed sample size calculations to ensure small optimism in predictor effect estimates, a small difference in the apparent and adjusted estimates of Nagelkerke’s R², and precise estimation of the overall risk in the population. 56 For example, based on an estimated FGR prevalence of 0.73%, with a maximum possible Cox–Snell RCS2 of 0.08, and an assumed lower bound for the apparent Nagelkerke’s RN2 of 0.32 based on previously published models,57 a minimum sample size of 34,906 women with 255 FGR events is required to meet the criteria when considering up to 50 predictor parameters. As our sample size was fixed (as it is dependent on the available IPD), for the models developed we restricted the number of candidate predictor parameters below 50 so that our sample size would easily meet the criteria of Riley et al.

The IPPIC-birthweight model has the continuous outcome of birthweight. Riley et al. further recommend that the sample size used to develop such a model should be sufficient to ensure small optimism in predictor effect estimates, a small difference between the apparent and adjusted R², precise estimation of the mean predicted birthweight (the model intercept), and precise estimation of the model’s residual standard deviation (SD). 58 For example, assuming a lower bound for the anticipated adjusted R² of 0.5 in the new model, and an intercept value of −0.935 with standard error 0.043 (on the log₁₀ scale) based on previous literature,59 a minimum sample of 618 women is required to consider up to 50 predictor parameters. Again, our sample size was fixed according to the IPD available, and we restricted the number of candidate predictor parameters to below 50, in order to meet the criteria by Riley et al.

Data synthesis

We used SPSS Version 27 (IBM SPSS Statistics for Windows) to analyse the Delphi survey findings that prioritised the predictors of FGR. All other analyses were carried out using Stata MP Version 16.

Development and validation of new or updated prediction models

To develop and validate new prediction models for (1) FGR (birthweight <10th centile adjusted for gestational age, with stillbirth or neonatal death or delivery before 32 weeks) and (2) birthweight at various gestational ages, we considered cohorts contained within the data repository at final database lock in August 2020.

Candidate predictors for model development were informed by predictors included in existing prediction models and by clinical experts in the collaborative group as detailed in Prioritisation of predictors. Our aim was to produce predictions conditional on assumed gestational ages at delivery, and therefore gestational age at delivery was included as a predictor in our models. Although the actual gestational age at delivery would not be available at the moment of prediction, producing the models in this way allows a range of assumed gestational ages at delivery to be entered for each woman, and a graph of predictions against gestational age to be made for them, to give a more complete picture over time. Example plots of such predictions are given later in the report.

To select datasets to use for development of a new prediction model, it was necessary to compromise between the number of datasets included and the potential predictors that could be considered for inclusion in the models (as not all predictors were available in all datasets). The aim was to do this in such a way to maximise both. We undertook the following process:

1. Summarised the number of datasets, total sample size and number of events available for each candidate predictor considered for inclusion.

2. Ranked the prioritised predictors based on the number of cohorts reporting the predictor in the IPD.

3. Started with the most commonly reported predictor and added prioritised predictors in a sequential manner to obtain the set of predictors which maximised the number of cohorts in the IPD, number of participants and number of events.

4. Stopped when adding any further predictors resulted in a sizeable loss of cohorts, participants or events and ensuring there was sufficient data to meet the sample size criteria set out in Sample size considerations.

Missing data

The number and proportion of missing values for each potential predictor and outcome were summarised by cohorts. Predictors were considered to be systematically missing for a cohort if they were not recorded for any or were recorded for very few individuals (<10%) in that cohort. Predictor values were not imputed for any cohort in which they were systematically missing.

Multiple imputation was implemented in each cohort separately to acknowledge the clustering of individuals within, and to retain heterogeneity between, cohorts. 35 We generated 100 imputed datasets (to exceed the largest percentage of incomplete observations in any of the individual cohorts), using chained equations, for each IPD cohort with any partially missing candidate predictors or outcome variables. Continuous variables were imputed using linear regression, binary variables were imputed using logistic regression and categorical variables were imputed using predictive mean matching. Complete predictors were also included in the imputation models as auxiliary variables. The imputation model included all candidate predictors and both outcome variables (birthweight and FGR).

Due to the difficulties in handling non-linearity in model development, and accounting for different non-linear functions in the imputation, a pragmatic decision was made to perform a preliminary complete case analysis to look for potential non-linear relationships between continuous candidate predictors and each outcome variable using multivariable fractional polynomial (MFP) models. Visual comparison of FP1 and FP2 functions was used to decide on the complexity of the functions to be included. If there was little difference between the shape of FP1 and FP2 functions, the simpler FP1 function was selected. Where a non-linear function was selected for a variable in the complete case analysis, rather than assuming that the FP1 function selected was correct, we included each of the possible (FP) functions in the imputation model, to enable this non-linearity to be considered during model development.

After imputation, the distributions of values for imputed variables were checked by plotting the mean ± SD for continuous variables against the imputation number (including the original unimputed data, imputation 0, for reference). For categorical variables, the proportions in each category were compared across imputations and to the original unimputed data.

Model development and variable selection

Prediction models were developed using random intercept regression models with backward elimination for variable selection. The random intercept was used to account for clustering of women within the individual cohorts.

Variable selection and consideration of the functional form for continuous variables took place within each cycle of the IECV (detailed below). An MFP approach was used, in which fractional polynomial functions were tested for each continuous variable (identified in the previous complete case analysis to potentially have a non-linear association with the outcome) to determine the ‘best’ functional form of that variable in the multivariable model (i.e. in the presence of all other variables).

At each stage of the variable selection process, the same model (i.e. including the same candidate predictors) was fitted to all imputations, and pooled Wald tests (using Rubin’s rules) were used for backward elimination, with p > 0.157 (proxy for AIC) for exclusion. 73

Heuristic shrinkage was calculated following the method proposed by Van Houwelingen and le Cessie74 for the final model in each imputation and pooled across imputations using Rubin’s rules to obtain the average shrinkage factor. This average shrinkage factor was then applied to each beta coefficient in the models, and subsequently average intercept values were re-estimated (holding fixed the shrunken beta coefficients) to ensure predictions in each dataset were correct on average.

Internal-external cross-validation

An IECV approach was used for validation, as IPD were available from multiple cohorts. 31,32 Using this approach, a model is developed using all but one cohort which is reserved for ‘external’ validation. The model is then internally validated using the same data, using methods such as bootstrapping to calculate the optimism in the model performance and the shrinkage factor. Bootstrapping was not practical computationally given the need to incorporate both non-linear trend examinations, variable selection and multiple imputation. Therefore, an approximate heuristic shrinkage factor was calculated (not accounting for the variable selection process) following the method proposed by van Houwelingen and le Cessie74 and applied to the regression coefficients as described above. 65

Following shrinkage, the model’s average intercepts were re-estimated to ensure predictions were correct on average. This then provided the ‘shrunken’ model equation. This ‘shrunken’ model was then applied to the omitted study to calculate the predicted birthweight at the observed gestational age at delivery, and then the predictive performance measures were calculated using CITL, the calibration slope, the c-statistic and Nagelkerke’s R² (as described in Assessment of model performance). This completes one cycle of IECV, and the process was repeated multiple times, each time reserving a different study for ‘external’ validation (see Figure 1). Calibration plots were also produced for each cycle of IECV, plotting average observed and expected values across imputations (where imputation-specific calibration plots were consistent with one another).

FIGURE 1.

Flow diagram showing processes involved in development and validation of the prediction model.

Following IECV, there were multiple values for each predictive performance measure (one from each cohort). These estimates were summarised using random-effects meta-analysis to give a pooled estimate of overall performance on IECV. Apparent performance of the model was also calculated for each cohort individually (and across the full dataset) using the average intercept term, to better approximate how the model would be applied in new individuals. Cohort-specific apparent predictive performance was also summarised across cohorts using random-effects meta-analysis to give a pooled estimate of overall apparent model performance.

For these random-effects meta-analyses, the calibration slope and CITL were pooled on their original scale, while the c-statistic was pooled on the logit scale68 with the standard errors of logit-C calculated using the delta method. 66 CIs were derived using the Hartung–Knapp–Sidik–Jonkman variance correction. 69

Decision curve analysis

For the binary outcome model, decision curves were produced (as described in Decision curve analysis) within each study cohort individually, as well as within the full dataset used for development. Expected numbers of true/false positives (T/FP) and true/false negatives (T/FN) per 1000 women based on using the model are also reported for a selection of potentially clinically relevant threshold probabilities, along with estimates of sensitivity and specificity for the model at each threshold, with the region between thresholds of 1% to 20% of most interest.

Characteristics of IPPIC cohorts

Overall, 94 cohorts were available in the IPPIC data repository (including 16 added cohorts), contributing data from 4,539,640 pregnancies. 18,75–164 About half the studies in the repository were prospective cohort studies (57%, 54/94), 16% (15/94) were randomised trials and 14% (13/94) were prospective registry datasets or birth cohorts. One dataset was an IPD of 31 RCTs. Most of the datasets included pregnant women from Europe (61%, 57/94), 16% (15/94) from North America, 6% (6/94) from South America, 6% (6/94) from Asia and Australia and 1 from Africa. Five datasets provided included participants from multiple countries such as Argentina, Colombia, Kenya, India, Peru, Thailand, Vietnam, Lebanon, Mexico, Mongolia, Uganda, Nigeria and New Zealand. About a quarter of datasets received were on women with high-risk pregnancies only (26%, 24/94), 13% (12/94) on low-risk pregnancies and more than half (61%, 57/94) included women with any risk pregnancies. Detailed study characteristics of all IPPIC datasets are provided in Appendix 1.

Prioritisation of candidate predictors of fetal growth restriction: Delphi survey findings

Forty collaborators participated in the first round of the e-survey. Most of the participants were from Europe (65%, 26/40), five each from the American and Oceania continents, three from Asia and one from Africa. Twenty-three participants took part in the second round of the prioritisation survey which took place via a Zoom video conference. Thirteen participants were from Europe (57%, 13/23), five from America (22%, 5/23), three from Oceania (3/23) and one each from Asia and African continents.

We identified 33 predictor variables from existing systematic reviews (18 clinical characteristics, 7 biochemical markers and 8 ultrasound markers). Additionally, between the first and second round of the survey, our external validation of existing prediction models for FGR identified a promising model with reasonable performance (see Chapter 5). 59 It was decided to take forward all predictors in the model as candidate predictors in our model development. These predictors were therefore included as candidate factors regardless of the ranking obtained from the first round of voting, and they were not considered by collaborators during the second round of voting.

The predictors included from the Poon 2011 prediction model, as well as predictors voted in/out following the two-round survey are provided in Table 2. A comparison of possible sample sizes based on combinations of candidate predictors in addition to the predictor variables from the Poon 2011 model59 was conducted and yielded the below (see Table 3). At each stage, the additional variable that maximised the number of cohorts, participants and FGR events was carried forward with the variables already selected. The process was then repeated considering the other candidate variables in the next iteration. In Table 3, bold text shows which variable was carried through to the next iteration, while red italics shows a variable was removed at that point, as only one study measured that combination of variables.

The final list of candidate predictors included those from the Poon 2011 model,59 along with previous PE, previous stillbirth and having had a previous SGA baby. This combination of predictors resulted in a restriction of analysis to 4 cohorts with 237,228 pregnancies and 1729 events (which met the sample size requirements discussed in Sample size considerations) for model development.

Identification of existing prediction models

We identified 119 prediction models (55 articles) for fetal growth and birthweight (see Figure 2). No model reported FGR as pre-specified by us. Of the eleven models that predicted birthweight on a continuous scale, eight (73%) included predictors not reported in the IPPIC cohorts IPD,59,158,165–167 and two (18%) included combinations of variables not available in the IPPIC IPD cohorts and could not be externally validated. 1,168 One model (Poon 2011) was eligible for external validation using the IPPIC cohorts. 59

FIGURE 2.

Flow chart of identification of eligible FGR prediction models for external validation.

Characteristics of the validated model

The Poon 2011 model predicted birthweight (with a log₁₀ transformation) on a continuous scale and included only clinical characteristics as predictors. The model equation is given below in Table 4. The model included gestational age at delivery, mother’s weight, height, age, parity, smoking status, ethnicity (white, black, Asian, Hispanic, mixed or other), pre-existing chronic hypertension, diabetes and assisted conception. Gestational age at delivery had the largest impact on predicted birthweight, with an increase in expected birthweight for each week increase in gestational age.

Characteristics of the IPPIC validation cohorts

At database lock for external validation of existing models on 31 January 2020, the IPD of 87 cohorts had been harmonised and were available in the IPPIC data repository. IPD from 10% (9/87) of the cohorts [Allen, ALSPAC (Avon Longitudinal Study of Parents and Children), Baschat, Generation R, Odibo, Rumbold, JSOG (Japan Society of Obstetrics and Gynecology), STORKG, POP]18,75,77,80,87,106,107,120,128 contained all predictor variables and outcomes allowing for external validation of the Poon 2011 prognostic model. Two of the nine cohorts included only nulliparous women. 18,128 The proportion of nulliparous women ranged from 46% to 62% across the remaining cohorts. Five of the included studies were prospective cohorts (Allen, Baschat, Odibo, STORKG, POP),18,80,87,107,120 three were from prospective registry datasets (ALSPAC, Generation R, JSOG),75,77,106 and one was a cohort from a randomised trial (Rumbold). 128 All cohorts consisted of unselected pregnant women, except the Rumbold cohort which included only low-risk pregnant women. The median gestational age at delivery was similar across all the cohorts. Most cohorts that recorded ethnicity predominantly consisted of white women, apart from Allen, Baschat and JSOG (47% Asian, 47% black and 100% Eastern Asian included as ‘other ethnicity’, respectively).

Summary characteristics for the cohorts used in the external validation of the Poon 2011 model are shown in Table 5. The greatest proportion of observations with at least one missing value for the variables of interest was observed in ALSPAC (89% incomplete); where mother’s height and weight, or birthweight of baby (outcome) were most commonly missing. As the required number of imputations, m, was set to at least the proportion of incomplete observations,60 this informed a minimum requirement of 89 imputed data sets for each study. We chose to impute 100 times for each study, for completeness and to fulfil this requirement. Detailed study characteristics of included IPPIC cohorts used for external validation are provided in Appendix 1. Risk of bias assessment of the cohorts by the PROBAST tool considered all cohorts to be at low risk of bias in the domains of participant selection and outcome reporting. All cohorts except the JSOG cohort were considered to be at low risk of bias for the domain of predictor reporting, which had an unclear risk of bias because not enough information was available to make the assessment (see Appendix 2).

Performance of existing model in predicting birthweight: external validation and meta-analysis

Average calibration across imputations

Calibration plots (with calibration curves) of the Poon 2011 model were generated in each IPD cohort separately, for each imputation, to assess the similarity of observed and predicted birthweights across the full range of predicted values. A comparison of the observed birthweight distribution and the predicted birthweight distribution in each cohort is given in Appendix 3, Figures 21 and 22. As calibration plots were very similar on visual inspection across imputations, it was concluded that predictions were similar enough across imputations for pooling to be appropriate. We present in Figures 3 and 4 calibration plots for the Poon 2011 model in each cohort, comparing the observed birthweight to the average predicted birthweight across imputations. These are presented on the more clinically interpretable birthweight (g) scale (see Figure 3), and on the original model scale of log₁₀ (birthweight) (see Figure 4), which allows better focus on the birthweights at the lower end of the predicted scale, where pregnancies at higher risk of FGR are more likely to be seen.

FIGURE 3.

Average calibration plots across imputations for individual cohorts on external validation of the Poon 2011 model, with the observed birthweight (g) plotted against predicted birthweight. The dashed line shows perfect calibration (where observed birthweight equals expected birthweight), while the blue line gives the smoothed calibration slope across all pregnancies.

FIGURE 4.

Average calibration plots across imputations for individual cohorts on external validation of the Poon 2011 model, with the observed 1og₁₀ birthweight plotted against predicted 1og₁₀ birthweight. The dashed line shows perfect calibration (where observed value equals expected value), while the blue line gives the smoothed calibration slope across all pregnancies.

On both outcome scales, the light blue LOWESS smoothed calibration curves can be seen to lie close to the diagonal (where expected equals observed outcome value) for all cohorts, suggesting impressive calibration performance on average across individuals from all populations included. We clearly see, though, from the individual points (green) in Figure 3, that for predictions at the higher end of the scale (where the bulk of the observations lie), there is a large variation in observed birthweights compared to a relatively narrow range of predictions for all datasets. For example, in the POP18 cohort predictions in the range of 2500 g to 4000 g correspond to observed birthweights in the range 2000 g to 5000 g. While the model predicts well on average within datasets, there is still some miscalibration in the higher range for some observations.

Calibration plots and curves on the original model scale (log₁₀ birthweight) also show this wider spread of observed values at the upper end of the scale, but this is less pronounced due to the log scale. When focusing on the lower predicted birthweights, those at higher concern regarding FGR, we see more clearly on this scale that calibration in generally good in this clinically important range. Given the clinical requirement of identifying low-birthweight babies at risk of FGR for early intervention, good calibration on average and small variation in predicted birthweights in the lower ranges make the model promising with this use in mind.

Pooled calibration across external validation cohorts

The Poon 2011 model showed reasonable calibration overall in each of the validation cohorts. Individual calibration slopes ranged from 0.91 (95% CI 0.83 to 0.98) in the Allen cohort, to 1.05 (95% CI 1.01 to 1.08) in the POP cohort, suggesting only a small and potentially unimportant miscalibration on average in terms of the slope (as seen visually within the calibration plots and smoothed calibration curves).

The pooled calibration slope across all cohorts of 0.97 (95% CI 0.94 to 1.01, τ2=0.0018) (see Figure 5, panel A) implies that the model is well calibrated across cohorts (given the summary calibration slope very close to the ideal value of 1, and its CI also crosses 1). There was also some heterogeneity evident across cohorts; for example, with a 95% prediction interval for the calibration slope in a new study of 0.87 to 1.08, when considering predictions on the birthweight scale. However, this range is still very narrow, and generally miscalibration is predicted to be quite small as measured by the slope.

FIGURE 5.

Forest plot for the calibration slope of the Poon 2011 model across external validation datasets for predictions made on the birthweight (g) scale (panel A) and the log₁₀ birthweight (log₁₀ grams) scale.

On the original log₁₀ (grams) scale of the model (see Figure 5, panel B), a summary calibration slope of 0.93 (95% CI 0.90 to 0.96, τ2=0.0012) also suggests slight overfitting, with a small amount of heterogeneity across cohorts. The 95% prediction interval suggests that the calibration slope in a new study would be between 0.84 and 1.02. In practice, predictions of interest are on the grams scale and so we suggest it is better to focus on the previous results.

In the individual cohorts, the smallest CITL value of −26.4 g (−27.5 to −25.3) suggests systematic over-prediction of birthweight on average in the JSOG cohort of 26.4 g, while the largest value suggests an under-prediction of 220.3 g (206.5 to 234.0) on average in the ALSPAC cohort.

The pooled CITL of 90.4 g (37.9 g to 142.9 g, τ2=4578 g2), when summarised across all gestational ages (Figure 6) showed systematic under-prediction of birthweight by around 90.4 g. This is reflected by the calibration curves being slightly above the 45° line of perfect calibration in most cohorts. The Poon 2011 model showed moderate between-study heterogeneity in CITL performance, with τ2=67.7 g, and a 95% prediction interval suggesting that we would expect a CITL for a new study to be between −78.4 g and 259.2 g.

FIGURE 6.

Forest plot for the CITL across cohorts (g).

Assessing CITL separately by gestational age at delivery (see Figure 7) showed that this average under-prediction was consistent across gestational age groups, with the pooled CITL ranging from 94.2 g (95% CI 23.6 g to 164.8 g) in those born 32- and 36-weeks’ gestation, to 108.5 g (95% CI −18.5 g to 235.4 g) in those born before 28 weeks. Uncertainty was much higher (with wider CIs for pooled CITL) for estimates at earlier gestational ages for delivery, due to the lower number of observed births before 32 weeks in all cohorts.

FIGURE 7.

Forest plot for CITL across cohorts, grouped by gestational age at delivery. GA, gestational age.

There was moderate to high heterogeneity seen between cohorts in the meta-analysis in all gestational age groups, with relatively wide 95% prediction intervals for all groups. For example, the prediction interval for CITL in those with gestational age <28 weeks at delivery suggests the new study may under-predict birthweight by up to 402.5 g or over-predict birthweight by up to 185.6 g in a new (but similar) cohort. Given the small average birthweights for babies born at this gestational age, such differences between predicted and observed birthweights are extremely large.

Summary of calibration of the Poon 2011 model

A summary of the meta-analysis results for the calibration slope and the CITL across different gestational age groups is given in Table 6. Both calibration measures are important to be considered in combination to assess the calibration performance of a prediction model, and thus a scatter plot including both measures on the individual dataset level is given in Figure 8. While no study shows perfect calibration by either measure, the cluster of points in Figure 8 demonstrates how the Poon 2011 model consistently under-predicts birthweight across cohorts (with the exception of JSOG), regardless of whether the associated calibration slope implied under- or over-fitting. The JSOG dataset can be seen to be an outlier, with one of the lowest calibration slope estimates, and was the only cohort to suggest an over-prediction of birthweight on average when using the Poon 2011 model.

TABLE 6 - Pooled calibration measures by gestational age at delivery

Gestational age at delivery	Number of datasets in meta-analysis	Performance measure	Pooled estimate	CI	Prediction interval	τ2
Any	9	Calibration slope	0.974	0.938 to 1.011	0.868 to 1.081	0.0018
		CITL	90.39 g	37.9 to 142.9	−78.4 to 259.2	4578
<28 weeks	8	Calibration slope	1.163	0.893 to 1.432	0.53 to 1.79	0.0531
		CITL	126.61 g	49.9 to 203.3	−9.5 to 262.7	2041
28–31 weeks	9	Calibration slope	0.894	0.850 to 0.937	0.85 to 0.94	0.0000
		CITL	107.54 g	76.8 to 138.3	60.3 to 154.8	222
32–36 weeks	9	Calibration slope	1.043	0.887 to 1.199	0.62 to 1.47	0.0276
		CITL	80.33 g	7.4 to 153.3	−137.9 to 298.6	7519
≥37 weeks	9	Calibration slope	0.907	0.838 to 0.976	0.70 to 1.11	0.0067
		CITL	90.26 g	10.4 to 170.1	−167.5 to 348.0	10,685

FIGURE 8.

Scatterplot comparing CITL and the calibration slope of the Poon 2011 model, as estimated in each cohort. The dotted lines indicate perfect calibration by each measure.

On average across external validation cohorts, the calibration slope of the Poon 2011 model was impressive when including all gestational age groups in the analysis, suggesting minimal overfitting of the model on average (pooled calibration slope: 0.97) across all age groups. Most overfitting was seen for those with gestational age 28–31 weeks, where a pooled calibration slope of 0.89 suggests that predictions were too extreme.

Calibration-in-the-large was also promising on average, with an average under-prediction of birthweight by 90.4 g (where under-prediction is clinically preferable in the determination of FGR risk). This average underprediction was consistent across gestational ages, which would have more of a relative impact on the usefulness of predictions for smaller babies born at earlier gestational ages.

Calibration curves for the Poon 2011 model reflect the similarity of observed and predicted birthweights suggested from the promising calibration slope and CITL values. The LOWESS smoothed calibration curves can be seen to lie close to the diagonal (where expected equals observed outcome value) for all cohorts, suggesting impressive calibration performance on average across individuals from all populations included.

Summary

In summary, from 119 prediction models for fetal growth and birthweight identified in our literature search, no prediction models were found to predict the probability of our predefined definition of FGR. One birthweight model could be externally validated. The Poon 2011 model predicts log₁₀ (birthweight) using 10 variables based on maternal characteristics only.

External validation of the Poon 2011 model was possible in 9 cohorts from the IPPIC repository, containing data on 441,415 pregnancies. Calibration of the Poon 2011 model was promising, with the pooled calibration slope only slightly lower than one on average across cohorts. However, there was some heterogeneity in the calibration performance across cohorts, with the calibration slope in individual cohorts lying slightly above or below the ideal value of one (implying predictions are slightly too extreme in some cohorts, and not quite extreme enough in others).

The model predictions could also be systematically too low or too high depending on the cohorts used to validate the model, although the Poon 2011 model was most seen to slightly under-predict birthweight. Under-prediction was by around 100 g on average across datasets, regardless of gestational age at delivery. The relative effect of this under-prediction would be greater in babies born at younger gestational ages, where expected birthweight is lower.

However, calibration was very good in general. Hence, due to the reasonably good performance of the Poon 2011 model on average across cohorts, we concluded that it would be illogical to begin building a new prediction model from scratch. Therefore, in the next chapter, we update the Poon 2011 model for predicting birthweight by using their included predictor variables as a basis for an updated model predicting the probability of FGR in pregnant women. By considering additional variables, agreed by clinical consensus, we further develop a model for predicting birthweight to ascertain whether the inclusion of new variables might improve the consistency of calibration across populations.

Characteristics of IPPIC cohorts included in the IPD meta-analysis

At database lock for the development of the FGR and birthweight models on 31 August 2020, 94 cohorts were available in the IPPIC data repository. After prioritisation of predictors from existing literature and clinical consensus (see Prioritisation of candidate predictors of fetal growth restriction: Delphi survey findings), IPD from four cohorts were selected as giving the best combination of predictor variables while maximising the numbers of cohorts, participants, and events for model development (see Prioritisation of candidate predictors of fetal growth restriction: Delphi survey findings). Three of the included cohorts were from prospective observational studies [Allen, STORKG, NICHD CSL (National Institute of Child Health and Human Development Consortium on Safe Labour)]80,107,164 and included unselected pregnant women. The Rumbold cohort was from a randomised trial and included low-risk women. 128

One cohort included only nulliparous women,128 while the remaining three had proportions of nulliparous women ranging from 40% to 56%. Across cohorts, the most common ethnicity was white (50%), followed by black (22%). Hispanic mothers were also well represented (17%) due to the high proportion of this ethnicity in the NICHD CSL cohort. The median gestational age of delivery was similar across all the cohorts (39–40 weeks), as well as the mean birthweight. The mean birthweight for all cohorts lay within a range of around 200 g, from 3199.8 g in NICHD CSL, up to 3418.3 g in STORKG. The composite FGR outcome was rare in all cohorts: notably only two pregnancies (0.2%) in the Allen cohorts and no women in the STORKG cohort met our criteria for FGR with complications. Across all four cohorts, 1729 (0.7%) pregnancies reported the outcome of FGR with complications, of these 1389 (80.3%) delivered before 32 weeks, 505 (29.2%) were stillbirths and 420 (26.7%) resulted in a neonatal death.

Detailed study characteristics of IPPIC cohorts used in model development are provided in Appendix 1, risk of bias assessment of the cohorts using the PROBAST tool is provided in Appendix 2 and plots of predictor distributions across the model development cohorts are provided in Appendix 4, Figures 23–28.

Missingness and multiple imputation

The birthweight outcome was rarely missing across cohorts, with the maximum proportion missing seen in the STORKG cohort at just 4.6%. 107 The composite FGR outcome was based upon the gestational age at delivery and birthweight (both of which were mostly complete in all cohorts), and complications of preterm birth (defined by gestational age at delivery, mostly complete), stillbirth (complete in all cohorts), or neonatal death. Neonatal death was well recorded in two of the cohorts (Rumbold, NICHD CSL),128,164 but was entirely missing in the remaining two (Allen, STORKG). 80,107 Given the rarity of neonatal death in the underlying populations (0.4%) and the small size of these datasets, we chose to assume neonatal death was not observed for all pregnancies included in these two datasets. Due to the rarity of neonatal death in combination with birthweight <10th centile, we would not expect this assumption to greatly influence the model estimates.

Summary characteristics for the cohorts used in development of the FGR and birthweight models, including proportions missing for each predictor, are shown in Table 7. The greatest proportion of observations with at least one item of missing information was observed in NICHD CSL (95% incomplete), where conception mode or previous stillbirth were most commonly missing. As the required number of imputations, m, was set to at least the proportion of incomplete observations,60 this informed a minimum requirement of 95 imputed data sets for each study. We again chose to impute 100 times for each study to fulfil this requirement. Details of imputation checks are included in Appendix 5, Figures 29–33.

Identification of non-linear associations in complete case data

As discussed in Missing data, we performed a complete-case analysis to identify potential non-linear associations between continuous predictors and the outcomes. Best-fitting fractional polynomial transformations were assessed in the presence of all other model predictors.

Upon visual inspection of the selected non-linear functions selected for mother’s height and mother’s weight, there was no improvement in functional fit with FP2 compared to FP1. Therefore, FP1 transformations of height3 and weight−1 were taken forward to imputation.

While the best fit for gestational age at delivery (GA) was linear at FP1, FP2 analysis suggested a transformation, in the presence of other predictors, with powers of GA−2 and GA⁻² ln GA. The selected FP2 function included a flattening of the curve at the extremes, as can be seen in Figure 9, avoiding the negative expected birthweights at lower gestational ages that would arise if assuming a linear association. Instead, the selected FP2 function suggests that birthweights decrease with increasing gestational ages up to about 23 weeks, which is also illogical. However, gestational ages affected by this would be outside the range of the expected model use, and so predictions were unlikely to be affected in practice. While a FP1 transformation (linear fit in this case) might be preferable for the sake of parsimony, we elected to take forward the possibility of non-linear transformation into the final analysis, to align with the FP2 transformation of gestational age at delivery in the Poon model, which we aimed to update.

FIGURE 9.

Best-fitting fractional polynomial transformations for continuous predictors in complete case data: mother’s height (cm), mother’s weight (kg) and gestational age at delivery (weeks). Note: Given the shape of selected FP2 transformation, predictions are not clinically relevant when based on assumed gestational ages below 23 weeks (dotted line).

Linear functions were selected as the best fit for mother’s age compared to both FP1 and FP2 functions, therefore, only the linear term was taken forward. Although the ‘best-fitting’ functional form was selected for each continuous predictor, it is evident from visual inspection that individual values for each predictor can vary a great deal from the line showing the selected function.

Predicting fetal growth restriction IPPIC-FGR prediction model

We developed the IPPIC-FGR model to predict FGR (a binary outcome) using data from all four IPPIC cohorts used to develop the IPPIC-birthweight model. A summary of the predictors retained in the FGR model after variable selection is given in Table 8. Spontaneous conception, a history of diabetes and mother’s weight were not retained in the model to predict FGR. The weight variable was excluded, with neither the linear nor the transformed terms being below the significance threshold for retention in the model, when considered along with the other model variables.

Gestational age at delivery was retained in the model, allowing predicted FGR risk to be generated conditional on any assumed (clinically relevant) value for gestational age at delivery, or indeed across a range of assumed values, as desired. Conditional on the other model variables, increased gestational age at delivery was associated with a reduced risk of FGR, as were increased mother’s height and being of ‘other’ ethnicity. Being nulliparous, smoking during pregnancy, an increase in mother’s age, a history of hypertension, previous PE, previous stillbirth or having had a previous SGA baby all increased FGR risk. All ethnicities other than white or ‘other’ were also associated with an increased risk of an FGR pregnancy.

An estimate of heuristic shrinkage was calculated in each imputation, and when averaged across imputations was 0.9985, implying very little overfitting in the development data due to the large effective sample size. Given this, it was concluded that application of shrinkage was unnecessary, and so no shrinkage (or associated re-estimation of the intercept term) was applied to the model.

Apparent overall performance and by cohorts

The apparent performance of the model was calculated by applying the FGR model directly back into each dataset using the average intercept term, using the observed gestational age at delivery for each participant (which would be unknown at the time of prediction in practice). On visual inspection, separation of the LP between events and non-events (discrimination) looked promising across datasets (see Figure 10). In particular, separation was good between the two LP distribution curves in the NICHD CSL dataset, where the bulk of the model development data originates. Note that no FGR events occurred in the STORKG dataset, hence the absence of the red FGR distribution, and there were only two FGR events in the Allen dataset. In both cases, the distribution of the LPs for pregnancies without FGR is similar to the corresponding distributions from the other two datasets.

FIGURE 10.

Distributions of LP values in the four model development datasets, separated by observed outcome status.

Study-specific model performance was not assessed for STORKG, as no FGR events were observed in the dataset: indeed, this study will have had very little weight towards the predictor effect estimates. The apparent predictive performance was pooled across datasets and is reported in Table 9 along with performance measures calculated in the full dataset, where predictions were calculated using study-specific intercepts. IECV was not used as planned for this binary model due to the low number of FGR outcomes in some of the smaller datasets.

TABLE 9 - Apparent predictive performance measures for the FGR model (applying predictions using the average intercept) for each dataset including all participants regardless of gestational age at delivery) and with pooled effect estimates across datasets

Notes

Prediction intervals not calculated due to small number of cohorts.

Note the full data calculations also include non-event pregnancies from the STORKG cohort.

	Pooled estimate	Allen	Rumbold	NICHD CSL	Full data
N (events)	236,405 (1729)	1045 (2)	1877 (18)	233,483 (1709)	237,228 (1729)
Calibration
Calibration slope
Point estimate	0.947	0.829	0.850	1.003	1.000
CI	0.665 to 1.230	0.361 to 1.298	0.669 to 1.031	0.979 to 1.027	0.976 to 1.024
τ2, 95% CI	0.007 (0.000 to 0.263)	–	–	–	–
CITL
Point estimate	0.323	−0.604	1.227	−0.006	−0.0001
CI	−1.881 to 2.527	−2.173 to 0.965	0.616 to 1.838	−0.062 to 0.050	−0.056 to 0.056
τ2, 95% CI	0.612 (0.055 to 14.468)	–	–	–	–
Observed/expected
Point estimate	0.323	0.634	2.190	0.996	1.000
CI	−1.881 to 2.527	0.633 to 0.636	2.181 to 2.199	0.996 to 0.996	1.000 to 1.000
τ2, 95% CI	0.393 (0.089 to 6.889)	–	–	–	–
Discrimination
c-statistic
Point estimate	0.962	0.990	0.874	0.962	0.961
CI	0.508 to 0.998	0.969 to 0.997	0.737 to 0.945	0.956 to 0.968	0.955 to 0.967
τ2 (95% CI)	1.373 (0.095 to 28.795)	–	–	–	–
Nagelkerke pseudo R2
Median (%)	40.1	30.4	40.1	50.2	50.1
Range	30.4–50.2	30.4–30.5	39.9–40.3	49.9–50.6	49.7–50.4
IQR	30.4–50.2	30.4–30.5	40.0–40.1	50.2–50.3	50.0–50.1

When including all participants (and thus any gestational age at delivery), the pooled c-statistic across datasets suggests excellent discrimination, at 0.962 (95% CI 0.508 to 0.998). The low number of events in the Allen dataset gives a misleadingly high estimate of the c-statistic, with narrow CIs (given the CI width is dependent on the c-statistic value). Apparent discrimination performance was impressive in both Rumbold and NICHD CSL datasets, with c-statistic estimates of 0.874 (95% CI 0.737 to 0.945) and 0.962 (95% CI 0.956 to 0.968), respectively.

The calibration slope was also promising, with a pooled apparent performance across datasets of 0.945 (95% CI 0.665 to 1.230) although again with wide CIs, given the small number of studies included in the meta-analysis.

Being the largest of the model development datasets, so contributing most to the number of events in the pooled data, we expect model performance to be at its best for NICHD CSL. Indeed, the model was calibrated best in this dataset by all performance measures, for example with an Observed to Expected ratio of 0.996 (95% CI 0.996 to 0.996) suggesting that the model is well calibrated for predicting FGR in this population, as anticipated.

Calibration plots in Figure 11 show the apparent calibration performance of the FGR model when applied to all participants in each dataset. Observed and expected FGR proportions are for risk groups by predicted FGR risk. On visual inspection of the smoothed calibration curve over all observations within a dataset, the model appears to overpredict FGR over the full range of predicted probabilities for both the Allen and Rumbold datasets, however, there were only 2 and 18 FGR outcomes, respectively in these datasets.

FIGURE 11.

Calibration plots of FGR prediction model, in all cohorts combined and in each of the model development datasets individually (apparent calibration performance), based on all participants (regardless of gestational age at delivery). The dashed line shows perfect calibration (where observed proportion equals expected proportion), while the blue line gives the smoothed calibration slope across all pregnancies. Number of pregnancies (events) is given for each. Average predicted risk is shown in 10 groups by predicted risk (green) and smoothed over all individuals (blue).

The model appears to be well calibrated in the NICHD CSL cohort for predicted probabilities of FGR below 0.5. This is in line with the O/E ratio and calibration slope estimates in this study. A similar calibration pattern is seen in the full data, as expected as the NICHD CSL study dominates the total dataset.

Model performance by assumed gestational age at delivery

Given gestational age at delivery is unknown at the time of prediction, clinically relevant timepoints must be chosen prior to prediction generation. This allows predictions of FGR to be produced conditional on various possible delivery times, and thus allows for calculation of distinct probabilities of FGR for a pregnancy at every possible gestational age at delivery.

Discrimination measures presented in Apparent overall performance and by cohorts are likely to be optimistic (e.g. c-statistic too high), as validation was conducted using predictions generated for a participant’s known delivery time. To give a more complete picture, we further examined the FGR model’s predictive performance for scenarios where all predictions in the data were generated conditional on the same assumed gestational age at delivery. Apparent calibration curves are presented in Figure 12 for the model’s calibration performance where all predictions were generated using the same assumed gestational age at delivery of (1) 34 weeks; (2) 36 weeks; (3) 38 weeks; and (4) 40 weeks, for everyone in the data, and then validated against (1) all women (regardless of their gestational age of delivery); and (2) the subset of individuals that actually had those gestational ages of delivery.

FIGURE 12.

Calibration plots of FGR prediction model in all cohorts combined, with predictions generated at the same assumed GA at delivery for every participant, but compared to observed risks at all Gas. Plots are given for assumed GA at delivery of 34 weeks (panel A), 36 weeks (panel B), 38 weeks (panel C), and 40 weeks (panel D), and evaluated against observed FGR status (regardless of gestational age of delivery). The dashed line shows perfect calibration (where observed proportion equals expected proportion), while the blue line gives the smoothed calibration slope across all pregnancies. Average predicted risk is shown in 10 groups by predicted risk (green) and smoothed over all individuals (blue).

Predictions at particular gestational ages and compared to observed FGR status regardless of gestational age at delivery

Predicted probabilities of FGR show greater spread when predictions are made conditional on a gestational age at delivery of 34 weeks, compared to later times. Calibration of the FGR model was best on average when assessed using an assumed gestational age at delivery of 34 weeks for all participants. When generating predictions conditional on the same fixed gestational age for all, the ordering of predicted probabilities did not vary when the assumed gestational age was changed, thus the discriminative ability of the model was consistent across all gestational age values when including validation in all women. When analysed in the largest of the datasets (NICHD CSL), a c-statistic of 0.742 (95% CI: 0.729 to 0.754) suggests good discrimination, even in the absence of a known gestational age at delivery, when evaluated across all women regardless of their gestational age at delivery. The pooled c-statistic across all datasets further suggests good discrimination, at 0.658 (95% CI: 0.262 to 0.913).

Predictions at particular gestational ages and compared to observed FGR status in the subset of participants who actually had that gestational age at delivery

Predictions conditional on an assumed gestational age at delivery were further assessed for calibration performance in the subgroup of participants who truly gave birth at (or close to) that assumed gestational age. The FGR model (generating predictions at a fixed gestational age) was best calibrated in pregnancies with a true gestational age <32 weeks, with calibration curves very close to the diagonal line of perfect calibration (see Figure 13, panels A and B). There was only very slight underprediction of FGR risk in pregnancies of gestational ages below 27 weeks, where the model was used to predict FGR risk at this time. Overprediction of FGR risk was evident in those with gestational ages >32 weeks, as seen in panels C and D of Figure 13, where the observed prevalence of FGR was much lower than predicted when using the model to predict FGR risk at 32 weeks (for those who truly gave birth between 32 and 36 weeks), or to predict FGR risk at 37 weeks (for those who gave birth at 37 weeks or later).

FIGURE 13.

Calibration plots of FGR prediction model in subgroups by gestational age at delivery, with predictions generated at the same assumed GA at delivery for every participant and evaluated against observed FGR status in subgroups defined by those with similar (but not identical) actual gestational ages. Plots are given for assumed GA at delivery of 27 weeks in those with a true GA ≤ 27 weeks (panel A), 28 weeks in those with a true GA between 28 and 31 weeks (panel B), 32 weeks in those with a true GA between 32 and 36 weeks (panel C) and 37 weeks in those with a true GA ≥ 37 weeks (panel D). The dashed line shows perfect calibration (where observed proportion equals expected proportion), while the blue line gives the smoothed calibration slope across all pregnancies in that GA group. Average predicted risk is shown in 10 groups by predicted risk (green) and smoothed over all individuals (blue).

Decision curve analysis

Net benefit

A comparison of using the model to inform treatment versus treat-all and treat-none strategies was done using DCA in all participants. Calculations were conducted separately in each of the cohorts with events used for model development (Allen, Rumbold and NICHD CSL) and in the combined data from all four cohorts. NB values were multiplied at each threshold by 1000, to give the extra number of women that would be correctly treated per 1000 women for whom the model is used, with none treated incorrectly, and are presented in Figure 14.

FIGURE 14.

Net benefit of using the binary outcome model to predict FGR (blue) in each cohort and in the combined model development data, in comparison to treat-all (green) and treat-none (orange) strategies, as evaluated in all women (regardless of their gestational age at delivery). Decision curves are shown for threshold probabilities between 0 and 0.5, with values greater than zero implying a NB from using the model to inform decisions and those less than zero implying a net harm.

A positive NB (with the curve lying above the zero line) was indicated when the model was used in the largest cohort, NICHD CSL, which was echoed in the analysis in the combined data, suggesting that a positive number of women per 1000 would benefit from being correctly identified as high risk based on the model’s use than would be harmed by incorrect identification. This is true in the range of threshold probabilities from 0 up to 0.5, with the decision curve lying entirely above zero over this range. There was therefore NB from using the FGR prediction model with ‘high risk’ defined by cut-offs anywhere in the predefined range of probabilities from 0.01 to 0.2 considered of key interest for clinical decision-making.

The FGR prediction model also showed a positive NB in this same clinically important range in the Rumbold cohort, with a higher expected NB than was seen in the NICHD CSL cohort in the 0 to 0.5 range. The range of threshold probabilities for defining ‘high risk’ with a positive NB was considerably narrower for Allen, possibly a reflection of how few events there were in this cohort, with the model becoming less favourable than a treat-none approach for threshold probabilities above 0.1, suggesting net harm when using the model in the Allen population for threshold probabilities above this range.

When considering NB separately by gestational age at delivery, decision curves show a NB in those with gestational ages before 28 weeks across the full range of threshold probabilities. A NB is further seen in those with a gestational age between 28 and 32 weeks (over and above the treat-all strategy) for all threshold probabilities >0.09 (see Figure 15). While no overall benefit is seen in those with gestational ages above 32 weeks, due to the extremely low prevalence of FGR in this group, NB analysis shows that using the FGR model results in no harm in these patients, while allowing substantial benefit in those who go on to deliver early. Given predictions must be generated conditional on some assumed gestational age at delivery (and true gestational age will be unknown until the time of delivery) it is important to confirm that no harm is expected in pregnancies where delivery is at later weeks.

FIGURE 15.

Net benefit of using the binary outcome model to predict FGR (blue) in the combined model development data (with predictions conditional on observed gestational age at delivery), in comparison to treat-all (green) and treat-none (orange) strategies, evaluated in subgroups by gestational age at delivery. Decision curves are shown for decision threshold probabilities between 0 and 0.5, with values greater than zero implying a NB from using the model to inform decisions and those less than zero implying a net harm.

Accuracy at specified probability thresholds

The expected number of TP, FP, TN and FN per 1000 women using the binary outcome model at different thresholds for predicted probability of FGR with complications are presented in Table 10. Threshold probabilities presented increase by 0.01 up to 0.2, in the range where predicted probabilities are likely to be of more interest clinically (as defined a priori by the IPPIC collaborative group), and then by increases of 0.1 afterwards.

TABLE 10 - Expected net benefit and number of TP, FP, TN and FN per 1000 women using the model at different predicted probability thresholds, based on FGR model’s apparent performance in full development data

Threshold probability	TP per 1000	FP per 1000	TN per 1000	FN per 1000	Sensitivity (%)	Specificity (%)	NB per 1000
0	7	993	0	0	100	0	7
0.01	6	47	946	1	88.3	95.3	6
0.02	6	30	963	1	85.7	97	6
0.03	6	23	969	1	81.9	97.7	5
0.04	6	20	973	2	78.6	98	5
0.05	5	17	975	2	75	98.3	5
0.06	5	16	977	2	72.7	98.4	4
0.07	5	14	978	2	70.1	98.5	4
0.08	5	13	979	2	67.8	98.7	4
0.09	5	12	980	2	66.5	98.8	4
0.10	5	12	981	3	65.2	98.8	3
0.11	5	11	982	3	63.9	98.9	3
0.12	5	10	982	3	62.6	99	3
0.13	4	10	983	3	61.6	99	3
0.14	4	9	983	3	60.6	99.1	3
0.15	4	9	984	3	59.5	99.1	3
0.16	4	9	984	3	58.7	99.1	3
0.17	4	8	985	3	58	99.2	3
0.18	4	8	985	3	57.1	99.2	2
0.19	4	7	985	3	56.4	99.2	2
0.20	4	7	986	3	55.4	99.3	2
0.30	3	4	988	4	46.4	99.6	2
0.40	3	2	991	5	34.8	99.8	1
0.50	1	1	992	6	19.1	99.9	1
0.60	1	0	992	7	9.5	100	0
0.70	0	0	993	7	5.8	100	0
0.80	0	0	993	7	3.2	100	0
0.90	0	0	993	7	0.8	100	0

For example, if the model was used with a threshold probability of 0.1 (10%), we would expect to identify 17 in every 1000 pregnancies as being at high risk of FGR, five of which would be expected to be truly FGR. Of the 983 rated as low risk of FGR, we would expect to miss three who would truly be FGR babies. The corresponding sensitivity of the model used with this threshold to identify those at risk of FGR would be 65.2%, with a specificity of 98.8%. This corresponds to a NB of three in every 1000 pregnancies where model risk predictions over 10% were used to determine a pregnancy as being high risk.

There is a positive NB expected per 1000 pregnancies for all threshold probabilities below 0.9. When rounding to whole numbers, we see a NB of at least one pregnancy per 1000 for threshold probabilities below 0.53 (with predicted FGR risk below 53% from the model).

Predicting birthweight

Model performance on internal-external cross-validation

To give a better representation of how the model might perform in new data, predictive performance measures were calculated on IECV. Given its large number of patients relative to the other cohorts, NICHD CSL164 was forced to remain throughout all cycles of the IECV approach. Therefore, we did not include a cycle where a model was built without NICHD CSL. Thus, although there were four cohorts available, there were only three cycles of the IECV approach reported in the validation below.

Model coefficients for each cycle of the IECV process are reported in Table 13. These coefficient estimates were reasonably consistent when estimated in subgroups of just three out of the four available cohorts, likely due to being highly influenced by the NICHD CSL cohort, which contributed the majority of the observations to every cycle of the IECV.

TABLE 13 - Predictive performance of the developed birthweight model with average intercept in each IECV cycle: the external validation performance in each dataset, for the cycle in which it was excluded from model development

Reported as median, range and IQR across imputations as R² cannot be summarised across imputations using Rubin’s rules.

	Pooled estimate	Allen	Rumbold	STORKG
N for model development	–	236,183	235,351	236,405
N for external validation	–	1045	1877	823
Calibration slope
Point estimate	1.002	0.895	1.065	1.038
CI	0.776 to 1.227	0.819 to 0.972	1.015 to 1.115	0.960 to 1.115
Prediction interval	−0.25 to 2.26	–	–	–
τ2 (95% CI)	0.007 (0.001 to 0.144)	–	–	–
CITL
Point estimate	9.720	−22.324	−33.419	86.406
CI	−154.317 to 173.756	−48.356 to 3.707	−53.363 to −13.474	57.308 to 115.504
Prediction interval	−943.23 to 962.67	–	–	–
τ2 (95% CI)	4200 (801 to 76,000)	–	–	–
Observed/expected
Point estimate	1.004	0.995	0.991	1.030
CI	0.938 to 1.070	0.949 to 1.041	0.935 to 1.047	0.974 to 1.086
Prediction interval	0.81 to 1.20	–	–	–
τ2 (95% CI)	0.000 (0.000 to 0.008)	–	–	–
R 2 a
Median (%)	45.7	32.6	47.4	45.7
Range	32.2–47.8	32.2–32.8	47.1–47.8	45.0–46.2
IQR	32.7–47.4	32.5–32.7	47.4–47.5	45.5–45.8

Calibration measures are reported separately by cohorts in Table 13, giving the performance of the model developed in all but one cohort when validated ‘externally’ in that reserved cohort. Pooled performance estimates give the average performance across IECV cycles. Predictions were generated for the true gestational age at delivery, using the study-specific intercept for the NICHD CSL study population, because the majority of the development data in each cycle came from this cohort and so average intercepts across cohorts would be heavily influenced by the mean birthweight from the NICHD CSL data.

The calibration slope estimates across IECV cycles suggest some slight overfitting to the development data in each cycle; specifically, the largest study dominates the model estimation, and as a result, the remaining studies were slightly miscalibrated. The largest miscalibration was seen in the model developed in the cohorts excluding Allen, with the calibration slope upon ‘external’ validation of 0.895 (95% CI 0.819 to 0.972) – however, this is still only very slight when shown visually (see Figure 16). The pooled calibration slope across IECV cycles was 1.002 (95% CI 0.776 to 1.227), with negligible miscalibration on average when models were applied in the cohorts held out from model development.

FIGURE 16.

Calibration plots for the birthweight model in each IECV cycle, on external validation in the dataset excluded from the model development stage of that cycle, and apparent calibration of the birthweight model with average intercept when applied in the full dataset. Observed birthweight is plotted against predicted birthweight, with the dashed line showing perfect calibration (where observed value equals expected value), while the blue line gives the smoothed calibration slope across all pregnancies.

The model developed in the cycle excluding the STORKG cohort was the best calibrated in terms of the calibration slope, at 1.038 (95% CI 0.960 to 1.115), although this model performed worst by CITL, a systematic underestimation of birthweight by 86.4 g on average (95% CI 57.3 to 115.5) in this cycle. The pooled CITL suggests an underestimation of birthweight by 9.7 g on average across IECV cycles (95% CI −154.3 to 173.8).

Predicted birthweights from the models developed in each IECV cycle (generated using the true gestational age at delivery and the study-specific intercept from the NICHD CSL study) were compared to the observed birthweights in the excluded study from that cycle. On visual inspection of the calibration plots, the smoothed calibration curves (shown in light blue) lie close to the diagonal line of perfect calibration for all cycles of IECV. Hence calibration is generally excellent, and the miscalibration noted above (in terms of calibration slope) appears to be minor.

While model calibration was good on average, miscalibration can be seen for individual observations at the higher end of the range of predicted birthweights, resulting in a wide spread of observed birthweights for a particular predicted birthweight in all cycles of the IECV. This spread is far narrower in the clinically important range of lower predicted birthweights, where pregnancies would be at higher risk of FGR.

Comparison of model performance to existing models

The Poon 2011 model performed well on average on external validation, although showed some heterogeneity in calibration across datasets, and slight miscalibration in the large. By including additional variables, on top of those from the Poon 2011 model, we hoped to reduce the heterogeneity in calibration performance across different populations. Our newly developed model is therefore referred to here as the ‘updated model’, as we updated the Poon 2011 model to include additional predictors.

Three of the model development cohorts were also used to externally validate the Poon 2011 model, and so predictive performance measures for the Poon 2011 and updated models were compared in these cohorts. We used the IECV performance for the newly developed model, that is, when that cohort was reserved for external validation for this comparison (see Table 13). Note that the final model built on all data was not represented in Table 13, as including the apparent performance of the newly developed model with the external validation performance of the Poon 2011 model would be an unfair comparison.

Calibration plots showing the predicted birthweight by each model (Poon 2011 and the updated model) compared to the observed birthweights in the Allen, Rumbold and STORKG datasets are given in Figure 17. On visual comparison, the calibration performances of both models are similar. Observed birthweights are similarly spread out for each predicted birthweight at the higher end of the range, with a narrower spread of observed values for the more clinically relevant predictions of lower birthweights. Both models appear to perform well on average, with the smoothed prediction curve (blue line) for each lying very close to the diagonal for all three cohorts.

Visual consistency in calibration plots is supported by the predictive performance statistics presented in Table 14, where the calibration slope, CITL and R² values of the two models are very similar for each of the cohorts. In particular, the R² values suggest that a similar amount of the variation in the observed birthweight for these three cohorts is explained by each model, despite the inclusion of new predictors (previous PE, previous stillbirth and previous FGR baby) in our updated model.

TABLE 14 - External validation performance of the updated birthweight model in each IECV cycle (performance in each dataset, for the cycle in which it was excluded for model development), and the Poon 2011 model in Allen, Rumbold and STORKG

Reported as median across imputations as R² cannot be summarised across imputations using Rubin’s rules.

	Allen (n = 1045)		Rumbold (n = 1877)		STORKG (n = 823)
	Updated model	Poon 2011	Updated model	Poon 2011	Updated model	Poon 2011
Calibration slope
Point estimate	0.895	0.906	1.065	0.963	1.038	1.009
95% CI	0.819 to 0.972	0.830 to 0.981	1.015 to 1.115	0.919 to 1.007	0.960 to 1.115	0.936 to 1.083
CITL
Point estimate (g)	−22.32	64.23	−33.42	125.06	86.41	118.92
95% CI	−48.36 to 3.71	38.39 to 90.07	−53.36 to −13.47	105.45 to 144.67	57.31 to 115.50	90.00 to 147.84
R 2 a
Median (%)	32.6	34.7	47.4	50.1	45.7	51.2
IQR	32.5–32.7	–	47.447.5	–	45.5–45.8	–

FIGURE 17.

Calibration plots for the updated birthweight model in each IECV cycle (performance in each dataset, for the cycle in which it was excluded for model development), and the Poon 2011 model in Allen, Rumbold and STORKG. Plots are from a single representative imputation. Observed birthweight is plotted against predicted birthweight, with the dashed line showing perfect calibration (where observed value equals expected value), while the blue line gives the smoothed calibration slope across all pregnancies.

In the Allen, Rumbold and STORKG cohorts, the Poon 2011 model had a calibration slope slightly closer to one than in the updated model, but conversely CITL was improved by the updated model. Systematic underestimation of weight was seen with the Poon 2011 model for both the Allen and Rumbold cohorts, while the updated model overestimated birthweight by 22.3 g and 33.4 g on average (compared to underestimation by 64.2 g and 125.1 g) for the Allen and Rumbold cohorts, respectively. In STORKG, the updated model underestimated birthweight by 86.4 g compared to underestimation by 118.9 g when using the Poon 2011 model. The calibration slopes of the updated model and the Poon 2011 model were 1.038 (95% CI 0.960 to 1.115) and 1.009 (95% CI 0.936 to 1.083), respectively.

Model equations and summary performance measures of the developed prediction model are shown in Table 15. The Poon 2011 model has some miscalibration-in-the-large, the magnitude (underestimation by 64.2 g to 125.1 g, dataset dependant) of which will be more pronounced in newborns born at earlier gestations. In the updated model, the miscalibration-in-the-large (which is closer to zero than that of the Poon 2011 model) was negligible. On the whole, both models perform similarly.

TABLE 15 - Model equations (with average intercept) and performance summary

Outcome	Model equation	Average statistic (95% CI) [95% prediction interval]
		c-statistic	Calibration slope	CITL
FGR (SGA with serious complications)	Logit(p) = −22.811 + 0.01 × (age) + 0.358 × (nulliparous) + 0.293 × (smoked) + 0.432 × (black) + 0.181 × (Asian) + 0.296 × (Hispanic) + 0.953 × (mixed) + 0.003 × (other) + 0.313 × (hypertension) + 0.887 × (previous PE) + 0.436 × (previous stillbirth) + 2.166 × (previous SGA baby) – 0.000000108 × (height3) + −56,010.23 × (GA−2) + 21652.92 × [GA−2 × ln(GA)]	0.962 (0.508 to 0.998)	0.947 (0.665 to 1.230)	0.323 (−1.881 to 2.527)
Birthweight	Birthweight = 9210.3 + 3.1 × (age) – 92 × (nulliparous) – 118 × (smoked) – 174.3 × (black) – 73.2 × (Asian) – 9.6 × (Hispanic) – 64.7 × (mixed) – 60.4 × (other) – 36.3 × (hypertension) + 150 × (diabetes) – 78.8 × (assisted conception) – 84.2 × (previous PE) – 13.6 × (previous stillbirth) – 481.7 × (previous SGA baby) + 2.7 × (weight) + 0.0000752 × (height3) + 24200000 × (GA−2) – 9274661 × [GA−2 × ln(GA)]	–	1.002 (0.776 to 1.227)	9.72 g (−154.3 to 173.8)
Birthweight (Poon 2011 model59)	log10 (birthweight) = −0.935219 + 0.186853(gestational age) – 0.002078 × gestational age2 + 0.003726 × weight −0.000030 × weight + 8.820640e−08 × weight3 + 0.000965 × height + 0.001466 × age) – 0.000026 × age2 + 0.016986 × parous – 0.024867 × age – 0.021769 × African – 0.017824 × South Asian – 0.005543 × East Asian – 0.009063 × mixed – 0.020995 × hypertension + 0.03143 × diabetes – 0.004015 × assisted conception	–	0.974 (0.938 to 1.011) [0.868 to 1.081]	90.39 g (37.9 to 142.9) [−78.4 to 259.2]

Summary and example predictions

In this chapter we used IPD from the IPPIC data repository to develop two models: the first to predict the probability of FGR, defined by a birthweight below the tenth centile by gestational age with serious complications (preterm birth <32 weeks, stillbirth or neonatal death); and the second to predict birthweight at various gestational ages. Both models extend the Poon 2011 prediction model,59 by incorporating predictors from the Poon 2011 model as a base, along with additional important predictors identified through a Delphi survey of the IPPIC Collaborative Network.

Both models can be used to generate predictions conditional on some assumed (clinically relevant) gestational age for delivery (or ideally a range of assumed values), as the true delivery time would be unknown at the moment of prediction. When used in combination, these models can give unique estimates of predicted birthweight and risk of FGR across the whole range of possible gestational ages at delivery. This is illustrated for two hypothetical babies in Figure 18, one which is clearly high risk and one that is low risk. Such plots of predictions allowing clinicians and patients to assess risks over the entire range of gestational ages at delivery, and give a complete picture to enable shared decision-making around the frequency of monitoring during a pregnancy.

FIGURE 18.

Predicted birthweight (red) and predicted FGR risk (blue) at different assumed gestational ages at delivery, using our models for two hypothetical babies (one high risk and one low risk). Shaded regions indicate birthweights below the 10th (lightest), 5th (middle) and 3rd (darkest) percentiles.

When these models were validated using study-specific intercepts across included IPD cohorts, the predictive performance of both models was best in the cohort contributing most to the development data (NICHD CSL). 164 For both models (with continuous and binary outcomes) the coefficients were informed mainly by this one study. Performance from IECV (for the birthweight model), and apparent performance in individual cohorts (for the FGR model) for the other three cohorts was promising, as was the performance of the Poon 2011 model for predicting birthweight when externally validated in these same cohorts.

Objective

The main objective of this model-based health economics analysis was to compare the costs and outcomes of the IPPIC prediction models for FGR, with existing strategies in the NICE 2008 Antenatal Care guideline for monitoring FGR. 169 We employed the same strategy used by NICE by using a decision analytical model framework, due to the lack of suitable evidence and data available for use in other economic model frameworks. The previous chapter developed two prediction models (one for FGR and another to predict birthweight), with the IPPIC prediction model for FGR, being suitable for evaluation using decision analytical model framework. Any strategy for predicting FGR needs to be balanced against the resources required to deliver this strategy, within the context of finite resources of the National Health Service (NHS) and to allow for redistribution of resources more efficiently across healthcare services. We set out to provide economic evidence to help decision-makers in different healthcare settings determine which strategy provides the greatest effectiveness (perinatal death avoided) at the most reduced cost in detecting FGR.

Method

For the health economics analysis, we developed a decision tree model based on NICE 2008 model, which was the most suitable model in this case, as the individuals in the model are independent of each other, and there are no recurring events. The time horizon for the economic model is less than a year: time from earliest entry into the model to delivery of fetus is <9 months; hence no discounting is required. The model was constructed using Microsoft Excel^® and compared to Strategy 1 and Strategy 3 of the NICE decision tree model as shown in Figures 19 and 20. The perspective adopted was that of the NHS and personal social services (PSS) as recommended by NICE;170 and private out-of-pocket costs to women or productivity losses have not been considered for the analysis.

FIGURE 19.

NICE decision tree for measuring and monitoring fetal growth. Strategy 1 and Strategy 3 of decision tree were compared to IPPIC prediction model strategy.

FIGURE 20.

The decision tree for measuring and monitoring FGR using the IPPIC prediction model.

Analysis

Results

Summary

Our economic analysis suggests that compared to strategies of no screening for FGR and measurement of FGR using SFH and ultrasound, based on the NICE 2008 model, there is potential for a strategy of using IPPIC-FGR model followed by ultrasound to prevent perinatal deaths. Sensitivity analyses conducted changing the model performance were in line with base-case results. The costs and outcomes analysis carried out using the NICE model is not reflective of the complex variation in current practice. The findings presented here will benefit from verification in well designed and conducted research studies with a full economic evaluation.

Summary of the findings

The newly developed and validated IPPIC-FGR model (FGR probability of FGR at various assumed gestational ages at delivery), and the updated and validated IPPIC-birthweight model accurately predict the probability of FGR (birthweight < 10th centile and preterm birth < 32 weeks, stillbirth or neonatal death) and birthweight, respectively, for various assumed gestational ages at birth. The IPPIC models have minimal miscalibration and excellent discrimination. Of the previously published models, the Poon 2011 birthweight model, which was used to update the IPPIC-birthweight model, has good calibration when validated in IPPIC cohorts. The IPPIC-FGR model shows net clinical benefit over a wide range of predicted probability thresholds. Its use is more cost-effective than alternate screening strategies for FGR.

Strengths and limitations

Our work complements the ongoing national efforts to reduce stillbirth and adverse perinatal outcomes, for which undiagnosed FGR is a major risk factor. Our IPD meta-analysis is the first to simultaneously develop and validate the performance of prediction models for FGR and birthweight. We used an unambiguous definition for FGR as our main outcome. By including both SGA, with severe complications such as stillbirths and neonatal deaths, we aimed to identify those babies at maximum risk of adverse outcomes, and not small but healthy babies. We accounted for treatment paradox of early delivery of a FGR baby preventing stillbirth or neonatal death,48 by including preterm delivery before 32 weeks as a component of the outcome. By keeping our predictions on the continuous scale in our IPPIC-birthweight model for various gestational ages at delivery, we were not limited by arbitrary cut-offs used to define FGR or SGA. Such an approach also allows clinicians to calculate predicted centiles using any fetal growth standard of choice (e.g. GROW, INTERGROWTH 21st, WHO). 50,51

Both IPPIC models can be used to generate predictions conditional on any assumed (clinically relevant) gestational age for delivery (or ideally a range of assumed values), as the true delivery time would be unknown at the moment of prediction. When used in combination, these models can give unique estimates of predicted birthweight and risk of FGR across the whole range of possible gestational ages at delivery, allowing patients to contribute to shared decision-making with clinicians around the frequency of monitoring during a pregnancy.

The IPPIC Network is collaborative in nature and was established with data provided by leading researchers with shared interest in the prediction and prevention of pregnancy complications. 44 By sharing their study data, these individuals have displayed buy-in to the research objective, which will help promote application of the developed prediction models in clinical practice. Use of this repository of cleaned, standardised and quality-assessed data from multiple cohorts increased the power of the study beyond what is achievable in a single primary study, minimised the potential for model overfitting and enabled the development and validation of robust prediction models.

We carried out a systematic approach to prediction of FGR, by first identifying existing prediction models, followed by external validation within individual IPPIC cohorts to assess transportability of identified prediction model to different populations and settings. Our model development work built on existing prediction models that showed promising performance, by informing candidate predictor selection. Clinical input was also used to prioritise predictors considered for model development. Multiple imputations were used to handle missing data for both predictors and outcome to avoid the loss of useful information. 60,180 We used rigorous statistical methods to develop the prediction models and assess their accuracy, including undertaking a formal internal and external validation within the IPD cohorts. Predictors included in the final models are those that are clinically relevant and routinely available in both low and high-resource settings. We based our economic modelling on the NICE economic model for monitoring fetal growth published as part of NICE Antenatal care guideline 2008. 21

There are some limitations to our study. Most of the published models identified to predict fetal growth or birthweight could not be externally validated due to differences in outcome definition reported by study authors, or because they included predictors measured late in pregnancy, and as such were more relevant for diagnosis than prediction of fetal growth. 26 We were also unable to validate eight prediction models that included predictors not available in any of the IPPIC cohorts. The use of data from existing studies for external validation of prediction models using IPD meta-analysis, is limited by variation across studies in whether and how relevant participant characteristics (as potential predictors) are recorded in these studies. However, IPD meta-analysis still provides the best opportunity to validate existing prediction models across multiple studies. Primary studies will require significant resources in order to accomplish what can be done using IPD meta-analysis from existing studies, especially with regards to generalisability, where multiple primary studies will be needed to validate prediction models. Some studies reported the development of various prediction models using data from the same cohort of women, with each subsequent publication assessing the addition of a new candidate predictor. This hinders the identification of published prediction models for external validation, and artificially increases the number of developed prediction models for fetal growth. Internal external validation of our model for FGR was limited by too few outcome events in some of the individual IPPIC cohorts. Our IPPIC-FGR prediction model was better calibrated in pregnancies with gestational age <32 weeks, however this is expected considering delivery at <32 weeks is part of our composite definition of FGR.

Our health economics analysis relied on data and structural assumptions for the decision tree model, which come with uncertainties. We however utilised high-quality data sources as much as possible such as from published meta-analysis and other economic models to inform our input and assessed the quality of all input parameters in a transparent way. This analysis was based on the NICE 2008 economic model and is not reflective of the complex variations in current practice. A detailed full-scale economic evaluation is needed, which evaluates the various strategies for risk assessment of FGR currently in use in management of pregnancies at risk of FGR. Ideally, with health economics models we can compare different interventions using outcomes based on robust clinical and health-related quality of life (HRQoL) data such as the QALY. However, due to the lack of reliable clinical and HRQoL data, the IPPIC IPD not being a primary study and consisting of studies none of which reported QALYs, our model does not follow the same structure. Instead, we take a similar approach to the NICE model,169 and consider the number of preventable perinatal deaths attributable to FGR for the strategy to be more effective for the measurement and monitoring of fetal growth.

Our models also require the user to enter the assumed gestational age at delivery. While the expected date of delivery is not known when making a prediction, entering various possible gestational ages for delivery allows the user to produce a plot of birthweight predictions across various time points. This was illustrated at the end of Chapter 6. In further research, a complementary model would be useful to predict the overall risk of FGR (averaged across all potential gestational ages), to go alongside our predictions which are conditional on gestational ages at delivery.

Although planned, we were unable to assess the performance of the models by population and trimester of use due to heterogeneity in reporting of population characteristics and paucity of data on onset of FGR. Our final model only included clinical predictors, so we did not compare performance based on choice of predictors (ultrasound and biochemical markers) for predicting FGR or birthweight. It is possible that addition of further predictive markers could have improved the performance of the IPPIC models.

Comparison to existing evidence

Until now, no individual test is satisfactorily predictive of FGR to warrant recommendation in routine clinical use. 22 There is considerable variation between guidelines on screening for FGR. The UK RCOG guideline provides a list of arbitrarily categorised ‘major’ and ‘minor’ risk factors based on clinical history, and recommends regular ultrasound for women with one ‘major’ or three or more ‘minor’ risk factors. 11 The ACOG recommends screening for unspecified medical and obstetric risk factors, but does not recommend use of uterine artery Doppler or biochemical markers, citing lack of evidence on improvement of outcomes. 12 The Society of Obstetricians and Gynaecologists of Canada calls for clinical risk factors-based screening, without specifications on what these are,13 while the Royal Australian and New Zealand College of Obstetricians and Gynaecologist suggests risk assessment through a combination of biomarkers, Doppler ultrasound and ‘major’ maternal clinical risk factors. 14 The choice of risk factors and their combination to predict FGR in any of the above guidelines is not based on formal predictive modelling. Their accuracy in predicting FGR is also not known.

Existing prediction models have predicted risk of SGA fetus as a surrogate measure for infants at risk of FGR,181 with variously defined cut-offs for FGR, limiting the power and usefulness of the prediction model, by not linking these birthweight cut-offs to serious perinatal complications such as stillbirth, neonatal death, extreme preterm birth or birth trauma. 38 These SGA prediction models have mostly never been externally validated, and those that have been independently validated report limited predictive performance. 16,159 As such none are recommended for routine clinical use. None of the models identified in our search predicted our predefined outcome for FGR (SGA with serious complications of stillbirth, neonatal death or preterm birth before 32 weeks’ gestation), and we were only able to independently validate one birthweight model59 which showed slight overfitting in the validation cohorts. Individual calibration of the model across the different IPPIC cohorts was good, with moderate heterogeneity in calibration performance between the cohorts. Underprediction of birthweight by the model was consistent across different gestational ages of delivery, and this would have more of a relative impact on predictions for babies born at earlier gestational ages where expected birthweight is lower. Heterogeneity was also high across the different gestational age groups with wide prediction intervals.

The IPPIC-FGR and IPPIC-birthweight models extended the Poon 2011 birthweight prediction model59 by building on predictors included in the original model. The IPPIC-birthweight model had good summary predictive performance, with only slight evidence of miscalibration in calibration performance, and minimal overprediction of birthweight. It underestimated the birthweight by less (12.9 g to 17.2 g) in the validation cohorts, compared to underestimation of birthweight of 64.2 g to 125.1 g by the Poon 2011 birthweight model. The IPPIC-FGR model had good summary predictive performance, with discrimination and calibration slope near 1. The model was also clinically useful across a wide range of predicted threshold probabilities, covering the identified range of 0.01 to 0.2 considered to be of interest for clinical decision-making.

There is scarce empirical evidence on the cost-effectiveness of screening for FGR. A recent study estimating the cost-effectiveness of universal routine screening by ultrasound for fetal growth reported that this was unlikely to be cost-effective. 182,183 Our health economics analysis of the IPPIC model to predict FGR, built upon the previously published economic model structure and care pathways for monitoring fetal growth by NICE 2008. 21 The NICE economic evaluation showed that although there was poor evidence on clinical effectiveness of monitoring of fetal growth by ultrasound or using SFH and ultrasound, these strategies were cost-effective compared to no screening. 21 Although our economic analysis of the IPPIC model showed that the model prevented more perinatal deaths than strategies of no screening for FGR or measurement of FGR in all fetuses using SFH and ultrasound scan, the screening strategies used in NICE 2008 are not reflective of current practice.

Relevance to clinical practice

Prediction of FGR allows for early identification of women at increased risk of FGR, who may benefit from closer monitoring in pregnancy or preventative interventions such as early administration of aspirin. Any effort to prevent adverse perinatal outcome will need to identify pregnancies that are at risk of delivering a growth-restricted baby with severe complications to assess the severity of smallness, determine the timing and frequency of surveillance and plan timing and mode of delivery. Current approach to screening differs by country and is mostly based on use of individual clinical risk factors to assess risk of FGR, which has been shown to have minimal predictive performance. 16 Our study combined clinical characteristic predictors in mathematical models to provide accurate FGR risk prediction, which have good performance when externally validated in different cohorts, looking across all observed gestational ages at delivery. Only clinical characteristic predictors are included in both IPPIC models, which make them applicable to both low and high-resource settings. The predictors included are easy to measure and routinely available in clinical practice. The prediction models will be particularly useful when used in combination to predict the risk of FGR and birthweight, as together they provide more scope to identify pregnancies that are at high risk of adverse outcome in addition to the birthweight at various gestational ages, which can inform decision for closer monitoring or intervention. Incorporating the IPPIC-FGR prediction models in practice will be straightforward as no additional measures are required to calculate the risk of FGR. However, we need to make sure that resources such as staff time and any training needs associated with using the prediction model have been costed appropriately. By working closely with clinical academics involved in the development of the RCOG Green Top national guideline on SGA fetus, and the RCOG fetal medicine clinical study group, we aim to facilitate their incorporation within national and international recommendations.

Relevance to research

FGR continues to be a research priority area. Our work is in direct response to the call of NICE guidelines and RCOG for predictive tests or strategies to identify women at risk of small baby, particularly for growth-restricted infant with complications,11,21 and the priorities of the Department of Health to reduce stillbirths and neonatal deaths. By developing models that predict the risk of developing a growth-restricted baby with serious complication, as well as the extent of its smallness, our models provide comprehensive information to help plan management. Also, a complementary model would be useful to predict the overall risk of FGR (averaged across all potential gestational ages), to go alongside our predictions which are conditional on particular gestational ages at delivery. Further research is needed on the implementability of the IPPIC models in routine clinical practice and to determine any barriers and facilitators to its use. This should include assessment of the acceptability of the prediction models as screening tools for pregnant women and their families, as well as healthcare providers. Research is also needed to identify the acceptable care pathways for various predicted risk thresholds, and this should involve relevant stakeholders, such as healthcare providers, pregnant women and their families. The impact of using the IPPIC-FGR prediction models in clinical practice may require evaluation through cluster-randomised trials to assess whether use of the models improves perinatal outcomes. Such a trial could evaluate use of the models to inform interventions (close monitoring or planned delivery) compared to routine care on perinatal mortality and morbidity. Although feasibility of such a trial is questioned due to the sample size required to show effect on perinatal mortality, such a study might look at proxies of perinatal mortality such as morbidity to achieve sufficient power. 183

Using IPD meta-analysis for the development and validation of the IPPIC prediction models has provided us with an increased sample size beyond any individual study, and more diverse populations for inclusion in our research. Using data from across the IPPIC data repository allowed us the opportunity for broader validation of models across different settings, populations and subgroups of interest, although this was limited due to the availability of predictor variables across the different cohorts. Primary studies on outcomes in pregnancy should collect information on fundamental predictors to minimise the impact of systematically missing data in their study when considered for use in IPD-based projects. 184 Despite there being more than 4.5 million pregnancies from 94 cohorts contained within the IPPIC data repository, the absence of fundamental predictors restricted the number of cohorts that could be used for model development and limited the number of existing models that could be externally validated.

A key problem in the prognostic field is that many prediction models are being developed, with far fewer externally validated. 185,186 While we attempted validation of all existing prediction models for FGR and birthweight, our ability to do so was limited by the lack of consistent predictor variables reported across the IPPIC cohorts, as well as the inclusion of model predictors that are rarely recorded in practice. While novel methods for multiple imputation can be used to account for systematically missing data (accounting for both the clustering of participants within studies and heterogeneity between studies), such methods can substantially complicate analyses and increase required computation time. The best approach to handle such data is often context specific, and in our study, there were issues of convergence of imputation models when we tried to impute for systematically missing predictors. We therefore decided it was better to impute within each study separately, which naturally retains the clustering of participants within studies and any heterogeneity between studies, though at the expense of not allowing systematically missing predictors to be imputed. The methods and software available for systematically missing data are constantly evolving, and while improved recording of core predictors at the primary study level would vastly reduce the need for such approaches, future IPD projects should also stay abreast of advances in methodology in this area, to ensure they maximise the use of the available data.

Our IPD meta-analyses allowed us to explore predictive performance more extensively than a single validation study. This is important, as calibration and discrimination performance of prediction models are known to vary across populations, which can clearly be seen in our external validation of the Poon 2011 model and the IECV of the IPPIC-birthweight model. This leaves a challenge for those wanting a single model for use everywhere, especially given models in this context appear to underpredict birthweight in some populations while overpredicting in others. It may be that locally recalibrated models may be a better way forward, where IPD gives us the ability to update and tailor these models to improve performance in specific settings and allow the same base model to be accurately fitted to multiple populations. 187

Researchers should adhere to recommended practice guidelines during economic evaluation of prediction models and follow approved methods for data collection, analysis and modelling. When including a prediction model in an economic model, key steps should be included, such as structure of the tree or model (to define possible pathways), estimate probabilities of the different pathways, assign values to costs and outcomes including any assumptions, analyse or roll back the tree, and explore any uncertainty in the model. Tools such as the CHEERS (Consolidated Health Economic Evaluation Reporting Standards)188 or Philips’s checklist189 should be followed to ensure that the health economic evaluation being carried out is consistently and transparently reported, and that good practices are followed when developing economic models to better inform health decisions.

Conclusion

The IPPIC-FGR and IPPIC-birthweight models accurately predict the risk of FGR and birthweight for various assumed gestational ages of delivery. The latter has better calibration performance than existing model. IPPIC-FGR model has clinical utility across wide probability thresholds and is more effective compared to alternate strategies of screening for FGR using SFH and ultrasound. Use of the IPPIC models in combination has the potential to identify women at high risk of FGR and assess the severity of smallness of fetuses across the range of potential gestational ages at delivery to plan appropriate management and minimise adverse perinatal outcomes.

Contributions of authors

John Allotey (https://orcid.org/0000-0003-4134-6246) (Lecturer, Women’s Health) developed the protocol, undertook the literature searches, study selection, drafted the manuscript, acquired Individual Participant Data, led the project, mapped the IPD variables and cleaned and quality checked the data.

Lucinda Archer (https://orcid.org/0000-0003-2504-2613) (Lecturer, Biostatistics) undertook the literature searches, study selection, drafted the manuscript, acquired Individual Participant Data, led the project and conducted the data analysis.

Dyuti Coomar (https://orcid.org/0000-0001-6229-0830) (Research Fellow, Biostatistics) conducted the economic evaluation.

Kym IE Snell (https://orcid.org/0000-0001-9373-6591) (Lecturer, Biostatistics) undertook the literature searches, study selection, drafted the manuscript, acquired Individual Participant Data, led the project and conducted the data analysis.

Melanie Smuk (https://orcid.org/0000-0002-1594-1458) (Senior Lecturer, Medical Statistics) undertook the literature searches, study selection, drafted the manuscript, acquired Individual Participant Data, led the project, mapped the IPD variables, cleaned and quality checked the data and harmonised the data.

Lucy Oakey (https://orcid.org/0000-0001-7664-5428) (Research Manager, Women’s Health) provided patient and public input and wrote the Plain language summary.

Sadia Haqnawaz (https://orcid.org/0000-0001-6777-8712) (PPI) provided patient and public input and wrote the Plain language summary.

Ana Pilar Betrán (https://orcid.org/0000-0002-5631-5883) (Medical Officer, Reproductive Health) developed the protocol.

Lucy C Chappell (https://orcid.org/0000-0001-6219-3379) (Professor, Obstetrics) developed the protocol.

Wessel Ganzevoort (https://orcid.org/0000-0002-7243-2115) (Specialist, Obstetrician and Gynaecologist) developed the protocol.

Sanne Gordijn (https://orcid.org/0000-0003-3915-8609) (Specialist, Obstetrician and Gynaecologist) developed the protocol.

Asma Khalil (https://orcid.org/0000-0003-2802-7670) (Professor, Obstetrics and Maternal-Fetal Medicine) developed the protocol.

Ben W Mol (https://orcid.org/0000-0001-8337-550X) (Professor, Obstetrics and Gynaecology) developed the protocol.

Rachel K Morris (https://orcid.org/0000-0003-1247-429X) (Professor, Obstetrics) developed the protocol.

Jenny Myers (https://orcid.org/0000-0003-0913-2096) (Professor, Maternal & Fetal Health) developed the protocol.

Aris T Papageorghiou (https://orcid.org/0000-0001-8143-2232) (Professor, Obstetrics and Maternal-Fetal Medicine) developed the protocol.

Basky Thilaganathan (https://orcid.org/0000-0002-5531-4301) (Professor, Fetal Medicine) developed the protocol.

Fabricio Da Silva Costa (https://orcid.org/0000-0002-0765-7780) (Adjunct Clinical A/Prof, Obstetrics and Gynaecology) provided input into the drafting of the initial manuscript.

Fabio Facchinetti (https://orcid.org/0000-0003-4694-9564) (Director, Obstetrics and Gynaecology) provided input into the drafting of the initial manuscript.

Arri Coomarasamy (https://orcid.org/0000-0002-3261-9807) (Professor, Obstetrics and Gynaecology) provided input into the drafting of the initial manuscript.

Akihide Ohkuchi (https://orcid.org/0000-0002-8861-1572) (Professor, Obstetrics and Gynecology) provided input into the drafting of the initial manuscript.

Anne Eskild (https://orcid.org/0000-0002-2756-1583) (Professor, Clinic Medicine) provided input into the drafting of the initial manuscript.

Javier Arenas Ramírez (https://orcid.org/0000-0003-2291-720X) (Researcher, Obstetrics and Gynaecology) provided input into the drafting of the initial manuscript.

Alberto Galindo (https://orcid.org/0000-0002-1334-4879) (Researcher, Obstetrics and Gynaecology) provided input into the drafting of the initial manuscript.

Ignacio Herraiz (https://orcid.org/0000-0001-6807-4944) (Researcher, Obstetrics and Gynaecology) provided input into the drafting of the initial manuscript.

Federico Prefumo (https://orcid.org/0000-0001-7793-714X) (Specialist, Gynaecology and Obstetrics) provided input into the drafting of the initial manuscript.

Shigeru Saito (https://orcid.org/0000-0002-8940-3708) (Professor, Obstetrics and Gynecology) provided input into the drafting of the initial manuscript.

Line Sletner (https://orcid.org/0000-0002-5085-7366) (Postdoc researcher, Metabolic Health) provided input into the drafting of the initial manuscript.

Jose Guilherm Cecatti (https://orcid.org/0000-0003-1285-8445) (Professor, Obstetrics and Gynecology) provided input into the drafting of the initial manuscript.

Rinat Gabbay-Benziv (https://orcid.org/0000-0001-6887-6314) (Professor, Obstetrics and Gynecology) provided input into the drafting of the initial manuscript.

Francois Goffinet (https://orcid.org/0000-0001-9643-0299) (Professor, Gynecology and Obstetrics) provided input into the drafting of the initial manuscript.

Ahmet A Baschat (https://orcid.org/0000-0003-1927-2084) (Professor, Gynaecology and Obstetrics) provided input into the drafting of the initial manuscript.

Renato T Souza (https://orcid.org/0000-0002-9075-9269) (Research Fellow, Maternal and Perinatal Health) provided input into the drafting of the initial manuscript.

Fionnuala Mone (https://orcid.org/0000-0002-0718-7547) (Clinical Lecturer, Maternal Fetal Medicine) provided input into the drafting of the initial manuscript.

Diane Farrar (https://orcid.org/0000-0002-5625-761X) (NIHR postdoctoral research fellow, Maternal Health) provided input into the drafting of the initial manuscript.

Seppo Heinonen (https://orcid.org/0000-0001-5949-0874) (Professor, Obstetrics and Gynaecology) provided input into the drafting of the initial manuscript.

Kjell Å Salvesen (https://orcid.org/0000-0002-1788-4063) (Professor, Clinical and Molecular Medicine) provided input into the drafting of the initial manuscript.

Luc JM Smits (https://orcid.org/0000-0003-0785-1345) (Professor, Epidemiology) provided input into the drafting of the initial manuscript.

Sohinee Bhattacharya (https://orcid.org/0000-0002-2358-5860) (Senior Lecturer, Medicine) provided input into the drafting of the initial manuscript.

Chie Nagata (https://orcid.org/0000-0003-4897-2119) (Chief of Clinical Research, Maternal and Child Health) provided input into the drafting of the initial manuscript.

Satoru Takeda (https://orcid.org/0000-0001-9547-1435) (Professor, Obstetrics and Gynaecology) provided input into the drafting of the initial manuscript.

Marleen MHJ van Gelder (https://orcid.org/0000-0003-4853-4434) (Assistant Professor, Perinatal Pharmacoepidemiology) provided input into the drafting of the initial manuscript.

Dewi Anggraini (https://orcid.org/0000-0001-7507-2445) (Consultant, Pediatrics) provided input into the drafting of the initial manuscript.

SeonAe Yeo (https://orcid.org/0000-0002-0721-0997) (Professor, Nursing) provided input into the drafting of the initial manuscript.

Jane West (https://orcid.org/0000-0002-5770-8363) (Professor, Public Health) provided input into the drafting of the initial manuscript.

Javier Zamora (https://orcid.org/0000-0003-4901-588X) (Professor, Biostatistics) developed the protocol.

Hema Mistry (https://orcid.org/0000-0002-5023-1160) (Associate Professor, Clinical Trials and Health Economics) developed the protocol and conducted the economic evaluation.

Richard D Riley (https://orcid.org/0000-0001-8699-0735) (Professor, Biostatistics) developed the protocol and conducted the data analysis.

Shakila Thangaratinam (https://orcid.org/0000-0002-4254-460X) (Professor, Maternal and Perinatal Health) developed the protocol, undertook the literature searches, study selection, drafted the manuscript, acquired Individual Participant Data and led the project.

All authors critically appraised and provided feedback and input into the drafts and final version of the report.

Ethics statement

Ethics approval was not required because the IPD meta-analysis involved secondary analysis of existing anonymised data.

Patient and public involvement

PPI members provided input to the running of the project via participation in the steering committee and project management groups. Katie’s team members which include mothers, pregnant women, carers and family members with an interest in improving the quality of research within women’s health, contributed to the fine-tuning of the primary outcomes of the project proposal, by providing feedback on what they would consider to be an important outcome. Development of the prediction model also took into account input from service users on the acceptability of predictors included in the model. A lay member of the Hilda’s group (a public advisory group for women’s health research) wrote the Plain language summary of the report. Dissemination of findings will be done in collaboration with Katie’s team, the Hilda’s, Sands charity and other interested groups.

Data-sharing statement

All data requests should be submitted to the corresponding author for consideration. Access to available anonymised data may be granted following review and appropriate agreements being in place.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme (NIHR award ref: 17/148/07) and is published in full in Health Technology Assessment; Vol. 27, No. 33. See the NIHR Funding and Awards website for further award information.

Disclaimers

This report presents independent research funded by the National Institute for Health and Care Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, the HTA programme or the Department of Health and Social Care. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, the HTA programme or the Department of Health and Social Care.