Development and validation of Prediction models for Risks of complications in Early-onset Pre-eclampsia (PREP): a prospective cohort study

Article history

The research reported in this issue of the journal was funded by the HTA programme as project number 09/22/163. The contractual start date was in November 2011. The draft report began editorial review in June 2015 and was accepted for publication in October 2015. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

Jon Deeks is a member of the Health Technology Assessment Commissioning Board.

Copyright statement

© Queen’s Printer and Controller of HMSO 2017. This work was produced by Thangaratinam et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

Burden of pre-eclampsia

Pre-eclampsia is a multisystem disorder in pregnancy associated with hypertension and proteinuria. 1–3 Hypertension is defined as systolic blood pressure (BP) of ≥ 140 mmHg and diastolic BP of ≥ 90 mmHg on two occasions between 4 and 6 hours apart. 1–3 Proteinuria is defined as ≥ 300 mg of protein in a 24-hour urine collection period or urine dipstick of 1+ or more in two samples collected 6 hours apart or a spot urine protein-to-creatinine ratio (PCR) of at least 30 mg/mmol. 2–4 Hypertensive diseases in pregnancy remain one of the leading causes of direct maternal deaths in the UK and account for 20% of all stillbirths. 5 In 1% of pregnant women, pre-eclampsia develops before 34 weeks’ gestation and thus is called early-onset pre-eclampsia. 6,7

Early-onset pre-eclampsia is considered to be a pathophysiologically different disease from late-onset pre-eclampsia, with considerably increased risk of maternal complications, including a 20-fold higher maternal mortality. 8–10 The only known cure for this condition is delivery of the baby and placenta. In women with early-onset pre-eclampsia, decisions on the timing of delivery can be difficult, as fetal and neonatal benefits from prolongation of pregnancy beyond preterm gestation need to be balanced against the risk of multisystem dysfunction in the mother. Preterm delivery accounts for 65% of neonatal deaths and 50% of neurological disability in childhood. 11 Many of the current practice guidelines do not consider gestational age at presentation as a criterion for diagnosis, severity or subclassification to stratify risk in women with pre-eclampsia. 2,12

The complexity of the treatment in early-onset pre-eclampsia gives rise to high health-care costs. 6,7 Women are often admitted to a tertiary care facility, and one-third experience complications, which may necessitate admission to an intensive care facility. 13 Infants usually need prolonged intensive care treatment for the management of complications, including lifelong handicaps arising as a result of prematurity. The additional cost to the NHS of caring for a preterm baby born before 33 weeks’ gestation is £61,509 and and of a baby born before 28 weeks’ gestation is £94,190. 14 Each year, the care of preterm babies costs the NHS £939M, largely accounted for by neonatal care such as incubation and hospital readmissions. 14 Delaying premature births by a week could potentially save £260M a year. 14

One of the key recommendations in the last Confidential Enquiries into Maternal and Child Health (CEMACH) report for policy-makers, service commissioners and providers, and health-care professionals (now known as the Centre for Maternal and Child Enquiries, CMACE) is the need to adopt an early-warning system to help in the timely recognition, referral and treatment of women who have or are developing critical conditions. 5 This applies to women with early-onset pre-eclampsia, as early recognition of women at risk of adverse outcomes will allow timely transfer from a secondary to a tertiary unit to enable care in a high-dependency unit or neonatal intensive care unit if needed.

Timely prediction of complications in women with early-onset pre-eclampsia involves the use of a combination of patients’ characteristics, symptoms, physical signs and investigations;15 these ‘tests’ are performed routinely in all obstetric units, but, in the absence of a structured approach, somewhat haphazardly. Gestational age is the most important determinant of perinatal outcome with more than half the chance of intact fetal survival when the gestational age is > 27 weeks and the birthweight is > 600 g. 16 Clinicians are hesitant to advocate expectant management because of uncertainties about the scale of maternal risk. Development of a prediction model for adverse maternal and fetal outcomes will help clinicians make appropriate decisions, after discussion with the parents.

Existing evidence

Objectives

• To develop, and internally validate, a prediction model in women admitted with early-onset pre-eclampsia from 20⁺⁰ weeks to 33⁺⁶ weeks’ gestation, for assessment of the risk of adverse maternal outcome by discharge and at various time points after diagnosis.

• To externally validate and update the model through two external data sets of patients with a diagnosis of early-onset pre-eclampsia.

• To assess the risk of adverse fetal and neonatal outcomes at birth and at any time until discharge, and to summarise the unadjusted and adjusted prognostic ability of a set of candidate predictor variables.

Study methods

The study protocol was developed according to existing recommendations on prognostic research, model development and validation, and prediction rule development,32–34 and reported in line with the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement. 35 The study received ethics approval from the National Research Ethics Service Committee West Midlands (approval number 11/WM/0248).

Study design and conduct

We undertook a prospective, observational cohort study to develop the prediction model(s). All consecutive women with suspected or confirmed diagnosis of early-onset pre-eclampsia were approached to take part in the study. Women were recruited from December 2011 to April 2014 based on a set of prespecified eligibility criteria and the follow-up of the last participant was completed in July 2014. Potential mothers were identified and recruited by research midwives and clinicians from the antenatal clinics, wards, day assessment units and delivery suites. We obtained information routinely collected as part of the antenatal booking process in the UK such as maternal age, ethnicity, smoking, alcohol intake and substance misuse. The ethnicity classification was applied using the NHS criteria. 36,37

Setting

The multicentre study was conducted in 53 obstetric units within secondary and tertiary care hospitals in England and Wales.

Participants

Women with suspected or confirmed diagnosis of early-onset pre-eclampsia (before 34 weeks’ gestation) were recruited to the study. Only women with confirmed early-onset pre-eclampsia were included in the final models.

Patient and public involvement

The Action on Pre-Eclampsia Charity (APEC) was vital in providing important input. The charity was involved from the very start with the development and design of the study protocol and will help with the dissemination of findings from the study. A member of the organisation sat as an independent member of the study steering committee and contributed to the overall supervision and management of the research project. They were also involved with the development of study materials, including the informed consent forms and patient information sheets. The APEC was involved in promoting the study to midwives and clinicians, attending the study days and meetings.

Inclusion criteria

The inclusion criteria were gestational age between 20⁺⁰ and 33⁺⁶ weeks; maternal age of ≥ 16 years; and a diagnosis of new-onset or superimposed pre-eclampsia. We also included women with a diagnosis of haemolysis, elevated liver enzymes, low platelets (HELLP) syndrome with no proteinuria or hypertension and those with one episode of eclamptic seizures but no hypertension or proteinuria. 9,38 All women provided written informed consent and were capable of understanding the information provided. We used an interpreter if required.

The definitions for diagnosis of pre-eclampsia are provided in Table 1.

Exclusion criteria

Women were excluded if the outcome (including recurrent eclamptic seizures) occurred prior to the tests or if there was insufficient time for gaining informed consent or if the mother did not comprehend spoken and written English adequately. A flow chart of study conduct is shown in Figure 1.

FIGURE 1.

Flow chart of the Prediction of Risks in Early-onset Pre-eclampsia (PREP) study conduct. a, Only reduced PREP logistic model validated.

Predictors

Outcome

The primary outcome was composite adverse maternal outcome that included at least one of the components in Table 2. In addition to maternal complications, prior to the analysis, we added delivery before 34 weeks’ gestation as an additional component to the composite maternal adverse outcome to minimise bias caused by treatment paradox. The components of the composite outcome were developed through Delphic consensus and had previously undergone piloting and validation in the Canadian cohort of patients in the PIERS (Pre-eclampsia Integrated Estimate of RiSk) study. 9 A composite measure for fetal outcome was also developed by the Delphi consensus9 (Table 3).

When more than one outcome occurred in the same woman, we chose the adverse outcome that occurred first for the purpose of the survival model. A panel of clinicians with expertise in pre-eclampsia and prognosis research ranked the maternal outcomes for their importance to clinical care (see Appendix 1).

Sample size

We aimed to examine about 10 candidate predictor variables for inclusion in the model(s). Simulation studies examining predictor variables for inclusion in logistic regression models suggest that approximately 10 events are necessary for each candidate predictor to avoid overfitting. 43–45 Therefore, to examine 10 candidate predictors, we required at least 100 women with adverse maternal outcomes in our cohort. From our systematic reviews,20–23 20% of women (100 of 500) with early-onset pre-eclampsia were expected to have adverse maternal outcomes at any time before discharge. Thus, the original target sample size was 500 women with confirmed pre-eclampsia. As the event rate was lower than predicted, we revised the sample size to continue recruitment until 100 women had experienced adverse events. Appendix 2 lists the changes compared to the original protocol submitted to the National Institute for Health Research Health Technology Assessment programme.

Prior to analysis, after discussion with the steering committee, the study group additionally classified delivery before 34 weeks’ gestational age as an adverse maternal outcome to avoid treatment paradox from delivery. The sample size criteria remained the same. With the increased number of outcomes, we were able to consider about 20 candidate predictors but we maintained at least 10 events per predictor in the modelling process.

Data sets for external validation: PIERS and PETRA studies

The PREP model was externally validated in two external independent data sets from the PIERS9 and the Pre-Eclampsia TRial Amsterdam (PETRA)46 studies.

Analysis plan development

We convened a panel of 24 experts in the field of pre-eclampsia and prognostic research, to explore the challenges and potential solutions in the development of a prediction model. The panel focused its discussion on methods to reduce the risk of bias in the PREP models as a result of treatment paradox. 24 After consideration of various methods, it was decided to include effective treatments such as antihypertensive drugs and magnesium sulphate as predictors to avoid bias. The appropriateness of the population, predictors and outcomes was discussed. The methodological issues pertinent to the analysis, such as the choice of model (logistic or survival), were considered and the panel suggested the development of both models.

Statistical analysis

We used a transparent process with appropriate prognostic research methodology for our analysis, and reported using TRIPOD recommendations. 35 We developed and externally validated two models: a logistic model (PREP-L) for overall risk of any adverse outcome by discharge, and a survival model (PREP-S) to assess the risk of adverse outcomes at various time points from diagnosis of pre-eclampsia until 34 weeks’ gestation.

Model development for adverse maternal outcomes

We applied the modelling process to women with confirmed diagnosis of early-onset pre-eclampsia and had complete outcome data (Figure 2).

FIGURE 2.

Flow of women recruited in the PREP study for development of the prediction model(s) for adverse maternal and fetal outcomes.

Definition of survival time

For survival analysis, the end of follow-up was defined as the time of occurrence of the first adverse outcome or end of 34 weeks’ gestation, whichever occurred first. A woman was considered to be at risk from the time of diagnosis of early-onset pre-eclampsia and the failure event was defined as maternal adverse outcome occurring before 34 weeks’ gestation. The survival data information was described in the original data set and copied into all five imputed data sets. The survival information was independent of the multiple imputation, as no outcome was imputed and the time of diagnosis data were available for all women.

Survival model: flexible parametric model

A flexible parametric survival model was used via the Royston–Parmar approach,49–51 with the cumulative baseline hazard scale modelled using restricted cubic splines (implemented as the stpm2 package in Stata12). We chose this approach over a Cox regression as it allowed us to explicitly model the baseline hazard rate allowing non-linear functions via cubic splines, which are very flexible and relatively simple to work with. Simpler parametric models may not be flexible enough to adequately represent the hazard function.

Splines are flexible mathematical functions defined by piecewise polynomials, with some constraints to ensure that the overall curve is smooth. The points at which the polynomials join are called knots. Royston and Lambert50 explain that the stpm2 uses restricted cubic splines which force the function to be linear before the first knot and after the final knot. Let s(x) be the restricted cubic spline function. Defining m interior knots, k₁, . . ., k_m, and also two boundary knots, k_min and k_maxs(x), can be written as a function of parameters γ and some newly created variables z₁, . . ., z_m + 1 giving:

⁽¹⁾ s ( x ) = γ 0 + γ 1 z 1 + γ 2 z 2 + ⋯ + γ m + 1 z m + 1 .

The derived variables z_j (also known as the basis functions) can be calculated as follows:

⁽²⁾ z 1 = x z j = ( x − k j ) 3 − λ j ( x − k min ⁡ ) 3 − ( 1 − λ j ) ( x − k max ⁡ ) 3 ,

where j = 2, . . ., m + 1,

⁽³⁾ λ j = k max ⁡ − k j k max ⁡ − k min ⁡ .

When choosing the location of the knots for the restricted cubic splines, it is useful to have some sensible default locations. In stpm2, the default knot locations are at the centiles of the distribution of uncensored log-event times.

Survival null model

We identified the number of knots to go into the model by fitting the null model with an increasing number of knots. The number of knots was chosen based on the lowest Akaike information criterion/Bayesian information criterion (AIC/BIC) and visual inspection of the change in fitted shape, with preference for simplicity (i.e. fewer knots) to avoid overfitting. AIC and BIC are measurements of model fit.

Univariable model, full model and variable selection process

Univariable analyses were performed for both models on all 22 candidate predictors in their linear (log-transformed, if applicable) form. These were fitted in each imputed data set and combined using Rubin’s rules. 48 Univariable analyses were performed only to summarise the unadjusted associations in the data, and were not used to inform the selection of predictors in the final multivariable models. Where applicable and computationally possible, all analyses were performed with the imputed data sets and the results were combined appropriately. A backwards selection procedure was applied to both full models as previously described. Maternal age and gestational age at diagnosis were forced into the model. The model was refitted after dropping each individual predictor.

Flow of participants in the study

Between December 2011 and April 2014, we screened 3302 pregnant women from 53 maternity units for inclusion in the PREP study. Of these, 2099 did not meet the inclusion criteria: 882 did not have raised proteinuria, 650 did not have raised BP readings, 457 were classed as other, 53 had underlying comorbidities, 36 were participating in a clinical trial of an investigational medicinal product and 21 did not understand English and an interpreter could not be used at the time of recruitment. Of the 1203 eligible women, 1101 were recruited to the study with a suspected or confirmed diagnosis of early-onset pre-eclampsia. Of those recruited, 954 women had confirmed pre-eclampsia, 142 women had a suspected diagnosis of pre-eclampsia that was not subsequently confirmed, baseline information data were not available in five participants and nine were lost to follow-up. The final maternal prediction models included data from 946 women and the fetal prediction model included data from 945 pregnancies (see Figure 2).

Baseline characteristics of women included in the PREP study

Table 5 shows the women recruited into the study according to the various inclusion criteria. Over 90% (866/954) of all participants had a diagnosis of new-onset pre-eclampsia, 75 women (75/954, 7.9%) had superimposed pre-eclampsia, 10 (10/954, 1.0%) had HELLP syndrome and three women (3/954, 0.3%) had a single episode of eclamptic seizure in the absence of raised BP or proteinuria.

The mean age of participants was 30.2 years [standard deviation (SD) 6.1 years] (Table 6). Two-thirds of women identified themselves as European (631/950, 66%), one-fifth as South or South-East Asian (18%, 172/950) and about one-tenth (81/950, 9%) as African. Around 3% of all women were from the Caribbean (31/950), 1% from the Far East (8/950) and 1% from the Middle East (6/950). Ninety-one per cent (866/954) of all pregnancies were singletons, while twins and triplets accounted for 9% (83/954) and 1% (5/954) of pregnancies, respectively. More than half of all women were nulliparous (551/954, 58%). Recurrent miscarriage (three or more) had occurred in 42 women (4.4%). About one-tenth (87/943, 9%) of women reported smoking in pregnancy at booking appointment and alcohol intake in pregnancy was reported by 5% (47/937) of all participants.

Predictor characteristics in women with early-onset pre-eclampsia

The values of the various candidate predictors of women in the PREP study are shown in Table 7. The mean gestational age at which the diagnosis of early-onset pre-eclampsia was made was 30.5 weeks (SD 2.9 weeks) and there were no missing values for gestational age at diagnosis.

Maternal and fetal adverse outcomes in women with early-onset pre-eclampsia

Outcome data were available for 99% (946/954) of all participants in the PREP study. The rates of individual components of the composite adverse maternal and fetal outcomes are provided in Tables 9 and 10, respectively.

Overall, 66.9% (633/946) of all women with early-onset pre-eclampsia experienced at least one adverse maternal outcome and 74.3% (702/945) had at least one adverse fetal outcome. The most frequently reported outcome was preterm delivery before 34 weeks’ gestation, occurring in 61.3% (580/946) of women. The second most common outcome was postpartum haemorrhage (7.8%, 74/946), followed by transfusion of any blood products (5.4%, 51/946) and abruptio placentae (2.6%, 25/946). The least reported maternal complications were need for positive inotrope support (0.1%, 1/946), posterior reversible encephalopathy (0.2%, 2/946) and a Glasgow Coma Scale score of < 13 (0.3%, 3/946). When preterm delivery was excluded as a component of the composite outcome, 15.5% of all women (147/946) had at least one adverse maternal outcome.

Modelling continuous predictors

Maternal age, systolic and diastolic BPs, haemoglobin level and platelet count were normally distributed, and hence we did not apply any transformation. Concentrations of ALT, AST, serum uric acid, serum urea and serum creatinine and the PCR were strongly right skewed, and we log-transformed these values. As gestational age at diagnosis was an inclusion criterion and limited to 34 weeks, a log transformation was applied to decrease the range. We were not able to fully evaluate serum uric acid concentration as a predictor because of data coding issues in this variable at the time of model development. Subsequent to model development being completed, and after the data coding issues were resolved for this variable, we calculated how the c-statistic changed after adding log-transformed serum uric acid concentration to the final models to assess whether or not this variable improved model performance (see Apparent performance and internal validation of the PREP-L model).

Development of PREP-L model: predictor selection

Table 11 shows the univariable and multivariable analysis for association of predictors and adverse maternal outcomes. The models were fitted in each of the imputed data sets and the results combined using Rubin’s rules. In the univariable analysis, lower gestational age at diagnosis, symptoms of epigastric pain and/or nausea and vomiting, clonus, exaggerated tendon reflexes, raised systolic and diastolic BPs, urine dipstick-detectable proteinuria, high levels of haemoglobin, low platelet counts, raised concentrations of ALT, serum urea, serum uric acid and creatinine, increased urine PCR, management with antihypertensives and use of magnesium sulphate were significantly associated with adverse maternal outcomes (p < 0.05). Relevant medical history of one or more conditions such as chronic hypertension, diabetes mellitus, renal disease, autoimmune disease and a history of pre-eclampsia in previous pregnancy were associated with a reduced risk of complications.

Predictor variables were dropped stepwise based on the largest p-value. The final list of predictors for the logistic model were maternal age, log-transformed gestational age at diagnosis, summary score for medical history, systolic BP, platelet count, log-transformed serum urea concentration, log-transformed PCR, baseline treatment with any antihypertensive and baseline treatment with magnesium sulphate.

Transformation of predictors for the final PREP-L model

We considered the following continuous variables for non-linear terms: maternal age, log-transformed gestational age at diagnosis, systolic BP, platelet count, log-transformed serum urea concentration and log-transformed PCR.

Appendix 5 shows the FP terms identified within each multiply imputed data set and the p-value for the test of deviance comparing the FP model with the model including linear terms only. Non-linear terms were identified as significant at the 1% level for log-transformed gestational age at diagnosis and serum urea concentration.

Final PREP-L model before adjusting for optimism

Table 12 shows the final logistic model after multiple imputation, including FP terms, and prior to adjustment for optimism.

The final PREP-L model identified that maternal age, early gestational age at diagnosis of pre-eclampsia, raised systolic BP, high urine PCR, high serum urea concentration, low platelet counts, need for treatment with antihypertensive drugs and administration of magnesium sulphate were associated with increased risk of adverse maternal outcomes. A positive medical history for pre-existing medical conditions or a previous history of pre-eclampsia was associated with a reduced risk of complications.

Apparent performance and internal validation of the PREP-L model

The apparent c-statistic for the PREP-L model (averaged across all multiply imputed data sets) was 0.84 (95% CI 0.82 to 0.87) and after adjustment for optimism it was 0.82 (95% CI 0.80 to 0.84). A sensitivity analysis showed that when all predictors were added to the final model, the c-statistic increased by < 0.01. Another sensitivity analysis of the model using oral and parenteral antihypertensive drugs separately showed no change in the c-statistic; therefore, the combined antihypertensive variable was retained. The addition of log-transformed serum uric acid concentration increased the c-statistic by < 0.004.

The predicted risk was grouped into tenths defined by centiles of predicted risk. Table 13 shows the proportions of outcomes observed within each centile of risk. Figure 3 shows predicted versus observed risk in the model development data set PREP.

FIGURE 3.

Predicted versus observed risk for maternal complications in the PREP-L model. Reproduced from Thangaratinam S, Allotey J, Marlin N, Dodds J, Cheong-See F, von Dadelszen P, et al. Prediction of complications in early-onset pre-eclampsia (PREP): development and external multinational validation of prognostic models. BMC Med 2017;15:68. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium, provided that appropriate credit to the original authors and the source is given. 40

Final adjusted PREP-L model for adverse maternal outcomes in women with early-onset pre-eclampsia

Based on the optimism in calibration, the predictor effect estimates of the developed model coefficient were reduced by the uniform shrinkage factor of 0.862. Appendix 6 shows the coefficient for the final logistic model adjusted for optimism.

Based on the women’s characteristics the probability of adverse outcome by discharge is:

where:

β₁–β_n are the coefficients for predictors in Appendix 6. Written formally, the equation used to derive individual risk predictions by discharge is as shown in Box 3.

The predicted probability of an outcome is exp(X)/(1 + exp(X)), where X is the predicted logit-p.

Application of the PREP-L model

We have shown examples of application of the PREP-L model below for two women recruited in the PREP study.

Sensitivity analysis of the PREP-L model in participants with unconfirmed diagnosis of pre-eclampsia

Of the 142 participants recruited with a suspected diagnosis of pre-eclampsia, 138 had a 1+ urine dipstick. There were two perfect predictions by baseline treatment with magnesium sulphate and these two observations were dropped. The optimism-adjusted logistic model, as described in Appendix 6, was applied to 136 women with an unconfirmed diagnosis of pre-eclampsia. The apparent c-statistic was 0.68 (95% CI 0.58 to 0.79) and the calibration slope was 0.64 (95% CI 0.25 to 1.04).

Modelling continuous predictors

We modelled the continuous predictors as shown in Chapter 4, Modelling continuous predictors.

Development of the PREP-S model: predictor selection

Table 16 shows the hazard ratios for adverse pregnancy outcomes for various candidate predictors by univariable and by multivariable analysis summarised across using the multiply imputed data sets. After dropping the candidate predictor variables stepwise based on the largest p-value, the following were included as the final list of predictors in the survival model: maternal age, log-transformed gestational age at diagnosis, summary score for medical history, systolic BP, clonus, exaggerated tendon reflexes, oxygen saturation, platelet count, log-transformed ALT concentration, log-transformed serum urea concentration, log-transformed serum creatinine concentration, log-transformed PCR, baseline treatment with any antihypertensive drug and baseline treatment with magnesium sulphate.

We identified non-linear terms for log-transformed gestational age at diagnosis and serum urea concentration using FPs. We chose terms with a p-value of > 0.001 to avoid overfitting. After including non-linear terms for gestational age at diagnosis and serum urea concentration, clonus then filled the criterion for exclusion at a p-value of > 0.150 and was therefore removed. Running the multivariable FP procedure without clonus confirmed that the FP terms identified originally were still valid (results not shown).

In the full multivariable model, the risk of adverse maternal outcomes were significantly increased at the 5% level with lower maternal age, greater gestational age at diagnosis, lower number of components of medical history, raised systolic BP, lower platelet count, raised ALT concentration, raised serum urea concentration, increased urine PCR and administration of magnesium sulphate.

We applied the variable selection process and included the non-linear terms for gestational age at diagnosis and serum urea concentration. The PREP-S survival model prior to adjustment for optimism is shown in Table 17.

Apparent performance of the PREP-S model

The apparent c-statistic of the developed survival model was 0.78 (95% CI 0.76 to 0.80). Estimation of risks in the five groups defined by the 10th, 25th, 75th and 90th centiles of predicted risk showed that the probability of adverse outcome was around 52% in the highest risk group at 48 hours after diagnosis, 95% by 1 week, and 100% by 1 month. Ninety-seven per cent (91/94) of women in the > 90th centile group experienced an adverse maternal outcome, with a mean follow-up time of 1.2 days (SD 1.3 days). In the lowest risk group (≤ 10th centile), the probability of outcome-free survival was around 97% at 48 hours after diagnosis, 87% by 1 week, and 56% by 1 month. In this group, 24% (23/95) had an adverse outcome, with a mean follow-up time of 28.9 days (SD 23.1 days) (Table 18).

In women in the groups defined by the highest centiles of predicted risk, there was good agreement between those with observed and predicted adverse outcomes. About 97% (91/94) of women with risks above the 90th centile had an adverse outcome; 83% (118/143) of women with predicted risks between the 75th and 90th centiles had complications; and 61% (289/471) with risks between the 25th and 75th centiles experience an adverse outcome.

Figure 4 shows the model-based mean survival curves for the five prognostic groups compared with their observed Kaplan–Meier survival curves, for 1 month after diagnosis-days. Agreement is generally excellent, perhaps with the exception of the lowest risk group, although this has fewest events and so more uncertainty about the mean predicted curve. CIs are not shown on the figure for aesthetic reasons.

FIGURE 4.

Mean survival curves for groups of prognostic index compared with their observed Kaplan–Meier survival curves up to 30 days from diagnosis. Reproduced from Thangaratinam S, Allotey J, Marlin N, Dodds J, Cheong-See F, von Dadelszen P, et al. Prediction of complications in early-onset pre-eclampsia (PREP): development and external multinational validation of prognostic models. BMC Med 2017;15:68. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium, provided that appropriate credit to the original authors and the source is given. 40

Internal validation and shrinkage of estimates for the final PREP-S model

The bootstrap approach showed an optimism of 0.019 (SD 0.010) in the c-statistic and 0.138 (SD 0.002) in the calibration slope. Based on the optimism in calibration, the predictor effect estimates of the developed model coefficient were reduced by the uniform shrinkage factor: 1 – 0.138 = 0.862. The intercept of the baseline spline term was re-estimated to ensure perfect calibration-in-the-large. The optimism-adjusted Harrell’s c-statistic of the survival model was 0.75 (95% CI 0.73 to 0.78).

Appendix 8 shows the coefficients of the final PREP-S model and the baseline hazard after adjusting for optimism. Table 19 gives the baseline survival at various time points to calculate the predicted survival probability for a woman diagnosed with early-onset pre-eclampsia.

Based on the woman’s characteristics, the survival probability at time point t is:

where β₁–β_n are the coefficients for predictors shown in Appendix 8, and X₁–X_n are the predictor values for the patient. Written formally, the equation used to derive individual risk predictions over time is shown in Box 4.

Positive regression coefficients suggest an increase in the risk of adverse maternal outcome with increasing values of continuous predictors or the presence of dichotomous predictor variables, and vice versa for negative coefficients.

Application of the PREP-S model

We have provided examples of calculating the individual risk of adverse maternal outcomes at 48 hours using the PREP-S model. The predictor values of two women are provided in Table 20. The calculation of risk can easily be amended to a different time point by replacing the value for baseline survival with the baseline survival for the desired time point from Table 18.

Sensitivity analysis of survival model in participants with unconfirmed diagnosis of pre-eclampsia

Of the 142 participants recruited with a suspected diagnosis of pre-eclampsia, 138 had a 1+ urine dipstick. For one woman the time of outcome was missing. The optimism-adjusted survival model, as described in Appendix 8, was applied to this population. The apparent c-statistic was 0.64 (95% CI 0.53 to 0.76) and the calibration scope was 0.88 (95% CI 0.17 to 1.58).

Inclusion criteria and availability of data in external data sets

Characteristics of women with early-onset pre-eclampsia in the PIERS and PETRA studies

There were no significant differences in the mean gestational age at diagnosis of pre-eclampsia, which was around 30 weeks. The PETRA study included only singleton pregnancies, while around 91% (870/954) of pregnancies in the PREP study and 85% (542/634) in the PIERS study were singletons. Two-thirds (601/953, 63%) of women in the PREP study did not have any significant medical history, such as pre-existing medical conditions or previous history of pre-eclampsia, compared with 45% (284/634) and 84% (182/216) in the PIERS and PETRA studies, respectively.

The PETRA study did not have any data on symptoms or examination findings such as deep-tendon reflexes or clonus. In addition, the study did not report any tests for proteinuria, oxygen saturation and serum creatinine concentration that were reported in the other two cohorts. Table 22 compares patient characteristics and candidate predictor variables in the PREP development data set and the PIERS and PETRA validation data sets.

Risk of adverse outcomes in the PIERS and PETRA cohorts

Overall, 67% (633/946) of women with early-onset pre-eclampsia in the PREP study had adverse maternal outcomes by discharge, compared with 77% (489/634) and 86% (185/216) in the PIERS and PETRA cohorts, respectively. The date and time of occurrence of adverse maternal outcomes was consistently reported in the PIERS data set and not in the PETRA study. Maternal and fetal composite outcomes not reported in the PIERS and PETRA data sets are provided in Tables 23 and 24.

External validation of the models

As not all predictors in the PREP models were available in the PIERS and PETRA data sets, we externally validated a slightly reduced version of our final models, with the model parameters re-estimated with a reduced set of predictors. We re-estimated the coefficients and intercept terms of the model, and then adjusted for optimism as before. We validated the survival model in only the PIERS data set, because of the non-availability of time of outcome occurrence in the PETRA cohort.

External validation of the PREP-L model in the PIERS data set

Complete records on the predictors considered were available for 437 of 654 women in whom pre-eclampsia was diagnosed at < 34 weeks’ gestation.

Obtaining the reduced PREP-L model

Serum urea concentration was identified as a predictor in the PREP data, but it was not recorded in the PIERS data set. We obtained the reduced PREP-L (rPREP-L) model and adjusted for optimism after excluding serum urea concentration (see Appendix 9). The calibration slope for this optimism-adjusted rPREP-L model was 1.01 (95% CI 0.86 to 1.15) when averaged across all imputed data sets. The apparent c-statistic was 0.82 (95% CI 0.80 to 0.85).

Application of the reduced PREP-L model in the PIERS data set

The optimism-adjusted c-statistic of the rPREP-L model was 0.81 (95% CI 0.77 to 0.85), indicating a good discrimination in the external validation data set. The calibration slope was 0.93 (95% CI 0.72 to 1.13), indicating very good calibration and model fit in the PIERS data on average across all individuals (Figure 5).

FIGURE 5.

Validation plot of the predicted vs. observed risk for adverse maternal outcome using the rPREP-L model in the PIERS cohort. Reproduced from Thangaratinam S, Allotey J, Marlin N, Dodds J, Cheong-See F, von Dadelszen P, et al. Prediction of complications in early-onset pre-eclampsia (PREP): development and external multinational validation of prognostic models. BMC Med 2017;15:68. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium, provided that appropriate credit to the original authors and the source is given. 40

The predicted risk was grouped into centiles of predicted risk. Table 25 shows the risk of outcome for each centile of predicted risk. When the intercept term was recalibrated to the PIERS data, the calibration slope and c-statistic remained the same.

TABLE 25 - Comparison of the predicted vs. observed risk for adverse maternal outcome using the rPREP-L model in the PIERS cohort

Groups of predicted risk	Women with predicted outcomes, n	Women with observed outcomes, n (%)
< 10th centile	0	–
10–20th centile	3	0 (0)
20–30th centile	20	6 (30)
30–40th centile	24	8 (33)
40–50th centile	33	16 (48)
50–60th centile	34	21 (62)
60–70th centile	38	19 (50)
70–80th centile	58	42 (72)
80–90th centile	72	59 (82)
> 90th centile	155	147 (95)

External validation of the reduced PREP-L model in the PETRA cohort

Complete records on the predictors considered were available for 211 of 216 women in whom pre-eclampsia was diagnosed at < 34 weeks’ gestation .

Harrell’s c-statistic was 0.75 (95% CI 0.64 to 0.86), indicating a moderate discrimination in the external validation data set. The calibration slope was 0.90 (95% CI 0.48 to 1.3), indicating some slight miscalibration, with observed risk generally higher than predicted. However, predictions showed reasonably close agreement at predicted risks above 0.7 (Figure 6). Table 26 shows the risk of outcome for groups defined by tenths of predicted risk.

FIGURE 6.

Validation plot of the predicted vs. observed risk for adverse maternal outcome using the rPREP-L model in the PETRA cohort. Reproduced from Thangaratinam S, Allotey J, Marlin N, Dodds J, Cheong-See F, von Dadelszen P, et al. Prediction of complications in early-onset pre-eclampsia (PREP): development and external multinational validation of prognostic models. BMC Med 2017;15:68. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium, provided that appropriate credit to the original authors and the source is given. 40

TABLE 26 - Comparison of predicted vs. observed risk for adverse maternal outcome using the rPREP-L model in the PETRA cohort

Groups of predicted risk	Women with predicted outcomes, n	Women with observed outcomes, n (%)
< 10th centile	0	–
10–20th centile	0	–
20–30th centile	4	2 (50)
30–40th centile	1	1 (100)
40–50th centile	11	4 (36)
50–60th centile	13	8 (62)
60–70th centile	22	18 (82)
70–80th centile	30	25 (83)
80–90th centile	74	70 (95)
> 90th centile	56	52 (93)

Recalibration of the intercept to the PETRA data did not improve the calibration slope. Table 27 shows the performance of the reduced PREP models in the derivation cohorts and external validation data sets.

TABLE 27 - Performance of the rPREP-L and rPREP-S models in the derivation cohorts and external validation data sets

rPREP-S, reduced PREP-S.

Model performance	PREP	rPREP (for PIERS)	PIERS	rPREP (for PETRA)	PETRA
PREP-L model
Number analysed	946	946	437	946	211
Number of outcomes	633	633	318	633	180
Apparent c-statistic (95% CI)	0.84 (0.82 to 0.87)	0.82 (0.80 to 0.85)	0.81 (0.77 to 0.85)	0.81 (0.79 to 0.84)	0.75 (0.64 to 0.86)
Optimism-adjusted c-statistic (95% CI)	0.82 (0.80 to 0.84)	–	–	–	–
Calibration slope (95% CI)	1	1	0.93 (0.72 to 1.13)	1	0.90 (0.48 to 1.32)
PREP-S model
Number analysed	946	946	339	–	–
Number of events	584	584	239	–	–
Apparent c-statistic (95% CI)	0.77 (0.75 to 0.79)	0.76 (0.74 to 0.78)	0.71 (0.67 to 0.75)	–	–
Optimism-adjusted c-statistic (95% CI)	0.75 (0.73 to 0.78)	–	–	–	–
Calibration slope (95% CI)	1	1	0.67 (0.56 to 0.79)	–	–

External validation of the PREP-S model in the PIERS data set

In the PIERS data set, 634 women were diagnosed with pre-eclampsia before 34 weeks’ gestation. Four hundred and sixty-one failures occurred during follow-up, six of which occurred on the same day as diagnosis and were included by adding a fraction of a day. One hundred and thirty-two had failures by 48 hours, 332 by 1 week and 458 by 30 days after diagnosis. We evaluated the reduced PREP-S (rPREP-S) model in 339 women with complete predictor value data. The total analysis time was 5425 days, with the last observed exit at 89 days of follow-up.

Obtaining the reduced PREP-S model

As serum urea concentration and exaggerated tendon reflexes were identified to be a predictor in the PREP-S model but were not recorded in the PIERS data set, we refitted the PREP-S model. Appendix 10 shows the coefficients of the rPREP-S prediction model after excluding serum urea concentration and exaggerated tendon reflexes, adjusted for optimism. Harrell’s c-statistic of the optimism-adjusted rPREP-S model was 0.76 (95% CI 0.74 to 0.78).

Applying the reduced PREP-S model in the PIERS data set

The rPREP-S model with coefficients as described in Appendix 10 was fitted to the PIERS data set. Figures 7 and 8 compare the predictions made by the PREP model in four prognostic groups of the PIERS data until 34 weeks’ gestation up to 30 days after diagnosis, respectively.

FIGURE 7.

Validation of the PREP-S model in the PIERS data set up to 30 days from diagnosis.

FIGURE 8.

Validation of the PREP-S model in the PIERS data set up to 7 days from diagnosis.

The c-statistic in the PIERS cohort was 0.71 (95% CI 0.67 to 0.75), which was slightly lower than that of the rPREP-S model. However, the four risk groups are still noticeably distinct and ordered appropriately for the majority of the 30-day period, with the exception of the two intermediate-risk groups before 3 days. The calibration slope of the rPREP-S model in the PIERS data set was 0.67 (95% CI 0.56 to 0.79), suggesting large overprediction of the reduced PREP model, and this was observed predominantly in the third of four groups (which had the largest patient numbers), especially after 5 days. Importantly, those identified as ‘high risk’ by the PREP model were still in the high-risk category in the PIERS cohort, but the observed absolute risk values were lower than expected from the reduced PREP model.

Calibration slope for each of the four risk groups are:

• ≤ 15th centile (n = 59): 0.21 (95% CI –0.51 to 0.92)

• 15–50th centile (n = 70): 0.65 (95% CI –0.80 to 2.10)

• 50–85th centile (n = 123): 0.25 (95% CI –0.34 to 0.84)

• > 85th centile (n = 87) 0.73 (95% CI 0.42 to 1.00).

Recalibration of the intercept of the baseline hazard function in the rPREP-S model to the PIERS data set did not improve calibration. Figure 9 shows that the agreement between observed and predicted survival was much improved in the high-risk group. However, it was noticeably worse in the other risk groups.

FIGURE 9.

Validation of the rPREP-S model in the PIERS data set after recalibration up to 30 days from diagnosis. Reproduced from Thangaratinam S, Allotey J, Marlin N, Dodds J, Cheong-See F, von Dadelszen P, et al. Prediction of complications in early-onset pre-eclampsia (PREP): development and external multinational validation of prognostic models. BMC Med 2017;15:68. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium, provided that appropriate credit to the original authors and the source is given. 40

Performance of the PREP-L model for adverse fetal outcomes

We assessed the performance of the PREP-L model (see Table 12) for the fetal composite outcome, using the same predictors. Table 28 and Figure 10 show the proportions of outcomes observed within each centile of risk. The c-statistic was 0.76 (95% CI 0.73 to 0.79) and the calibration slope was 0.77 (95% CI 0.63 to 0.91), indicating overprediction of risk, especially for prediction of 0.6 or below. In women predicted to be at high risk (> 80th centile), around 90% had adverse fetal outcomes.

FIGURE 10.

Calibration plot of the predicted vs. observed risk for adverse fetal outcome using the PREP-L model.

Predictive value of tests for adverse fetal and neonatal outcomes

We evaluated the prognostic value of all candidate predictors associated with maternal outcomes and the following five additional predictors: ultrasound (uterine artery Doppler in second trimester, expected fetal weight and liquor volume), CTG findings and use of steroids within or before 24 hours of diagnosis of pre-eclampsia. Table 29 shows the descriptive values of the candidate predictors and their crude and multivariate association with adverse fetal outcomes.

Association of maternal and fetal characteristics with adverse fetal outcomes

In the multivariable analysis of predictors, increased gestational age at diagnosis of pre-eclampsia reduced the odds of fetal complications (OR 0.09, 95% CI 0.01 to 0.61). A medical history of pre-existing chronic hypertension, diabetes mellitus, autoimmune disease, renal disease or a history of pre-eclampsia in previous pregnancies reduced the odds of composite adverse fetal outcomes for one pre-existing medical complication (OR 0.65 95% CI 0.44 to 0.98) and for two or more pre-existing medical complications (OR 0.43 95% CI 0.25 to 0.77).

The odds of fetal complications were significantly increased in women with raised urine PCR (OR 1.29, 95% CI 1.11 to 1.50), serum urea concentration (OR 1.72, 95% CI 1.07 to 2.76), treatment with antihypertensives (OR 1.56, 95% CI 1.04 to 2.37), treatment with magnesium sulphate (OR 2.40, 95% CI 1.04 to 5.57), abnormal uterine artery Doppler (OR 1.94, 95% CI 1.08 to 3.51) and when expected fetal weight was less than the 10th centile by ultrasound (OR 2.54, 95% CI 1.46 to 4.40).

Strengths and limitations

A well-performing prediction model is one that is relevant, accurate, validated in populations and data sets external to those used to develop the model and applicable to clinical practice. With these properties, it has the potential to improve clinical outcomes by helping clinicians and patients make more informed decisions.

The PREP models were developed in a sample of women with early-onset pre-eclampsia, a condition that is considered to be pathophysiologically different from late onset disease,8–10 and with a high proportion of adverse outcomes. We used prospective cohorts with high-quality data for both model development and validation, and with standardised definitions of variables and outcomes. The model was developed with data from 53 units in the UK, making the results as generalisable as possible within the NHS.

We ensured that all routinely performed tests in clinical practice were evaluated. The choice of predictors and components of the composite outcome were made by using Delphi surveys of experts in the field. 9,15,41 We chose delivery before 34 weeks as an outcome to further minimise treatment paradox-related bias, as delivery is planned at this gestation only if there are concerns regarding the health of the mother.

Prediction models often evaluate a large number of predictors in a population with few events, making the findings less robust. We ensured that we had adequate sample size for the number of candidate predictors to avoid overfitting. 43–45 The rates of follow-up were very high in our PREP cohort and very few individuals had missing values for most predictors.

One of the main reasons why clinicians lack confidence in applying risk scores in practice is the lack of sufficient evidence to demonstrate the reproducibility and transportability of the model in an external data set. 32 Furthermore, they are less likely to accept the model if it does not include important predictors such as BP. Guided by an a priori expert workshop (see Chapter 2, Analysis plan development), we minimised bias due to treatment by the inclusion of management decisions such as use of antihypertensives and magnesium sulphate as predictors. We transparently reported the development of the model and have provided the regression coefficients to enable clinical use and future validation of the model.

Our prediction study used rigorous statistical methods to develop the model, to assess its accuracy and to formally validate its performance in external data sets. 32–35,39,48–51 We developed two prediction models for the dual purpose of obtaining overall complication risks arising from pre-eclampsia and risk estimates for complications at various time points after diagnosis. A logistic model alone would not have sufficient sample size to provide estimates of adverse outcomes at time points close to diagnosis, such as 48 hours after delivery, given the low rates of serious complications. However, the PREP-S model allowed us to overcome this problem, and is the first to provide individualised risks of adverse maternal outcomes at various time points after the diagnosis of early-onset pre-eclampsia.

We performed geographic, temporal and domain validation of the model. The external data sets of the PIERS and PETRA cohorts were geographically different (Canada and the Netherlands) and were conducted earlier than the PREP study. We validated reduced models in external data sets as fewer predictors were evaluated therein. However, the rPREP-L model validated with good discrimination and calibration for predicting overall risk, and we expect the original PREP-L model to have similar, if not better, performance if fully externally validated.

The aim of the model was to provide reliable, accurate and precise information of risks to the mother and baby based on tests done at the time of diagnosis of early-onset pre-eclampsia. We only evaluated the tests and variables measured routinely in clinical practice. The added value of biomarkers and ultrasound to the accuracy of the model is not known. We refrained from using predictors such as fetal weight estimated by ultrasound, which may have been a significant predictor, as access to ultrasound may not always be available close to the diagnosis of pre-eclampsia in most units. We arbitrarily chose the components of relevant medical history and scored them. Inclusion of a different set of medical conditions may have altered the results. Women with a medical history score appeared to have a reduction in the risk of complications. It is likely that specialists in joint obstetric specialist clinics closely monitor these mothers resulting in early diagnosis of pre-eclampsia. Targeted and intense follow-up of these women may have led to prolongation of pregnancy beyond 34 weeks and with low complications. When developing our PREP models, we were unable to properly examine serum uric acid concentration as a predictor because at the time of the model development there was a coding error in the data for this variable. However, subsequent to the PREP models being developed and this coding error being corrected, we examined if the inclusion of serum uric acid concentration was important and found that the c-statistic barely changed for either the PREP-L or PREP-S model.

Women with earlier diagnosis of pre-eclampsia appeared to be at lower risk of maternal complications in the PREP-S model. This is likely, as the primary outcome is largely driven by delivery before 34 weeks. In women who were close to 34 weeks’ gestation, clinicians may have a lower threshold for delivery in the next few days or weeks. However, if the diagnosis was made much earlier in the pregnancy, clinicians would aim to prolong gestation as long as possible, leading to a longer survival time.

Our primary outcome was a composite of maternal complications. A different choice of outcomes may have identified a different set of predictors. However, given the rarity of individual complications in women with early-onset pre-eclampsia, we felt that our approach to include delivery before 34 weeks is a close representative measure for the severity of the disease. However, we did not separately report iatrogenic preterm deliveries from spontaneous preterm deliveries. As it is difficult to accurately identify the cause of spontaneous preterm delivery, which could still be related to pre-eclampsia such as small abruption, we grouped them together as one outcome.

The external data sets were limited in the number of variables evaluated, and hence we were unable to validate our full PREP model in either of them. The reduced PREP models, especially rPREP-L, showed good performance in the development and validation data sets, although the rPREP-S model showed reduced performance. Although the overall values of predictors may be similar to the PREP cohort, the management of women with early-onset pre-eclampsia may be different in the various health-care systems of the external cohorts; for example, magnesium sulphate treatment was provided in 51% of all women with early-onset pre-eclampsia in the PIERS cohort, compared with only 15% in the PREP cohort. Furthermore, the variation in the proportion of women giving birth before 34 weeks’ gestation, which was the major component of the composite outcome, may have contributed to the reduced performance of the model in the external data sets. The narrow spectrum of diseases in individuals in the PETRA cohort may have contributed to the reduced performance of the PREP model.

Comparison with existing evidence

So far, systematic reviews have not been able to produce robust estimates of accuracy of the individual tests. 20,22,23,39,53,54 Tests widely used in clinical practice, such as measurement of BP and proteinuria, suffered from treatment paradox and those such as clonus and deep-tendon reflexes were not studied in sufficient detail. This study is the first to address the above deficiencies.

The PREP study is the first to develop and validate the models for predicting adverse maternal outcomes specifically in women with early-onset disease. Previously, the PIERS and mini-PIERS models have provided estimates for overall risk of adverse outcomes in women with pre-eclampsia of any onset. Their sample sizes were too small to predict complications in women with early-onset pre-eclampsia. This is also the first study to provide individualised risk estimates for adverse outcomes at various time points after diagnosis of early-onset pre-eclampsia. Although the PIERS model included predictors such as gestational age at diagnosis, and concentrations of liver enzymes (AST) and serum creatinine to predict a composite maternal outcome, other important variables, such as BP and proteinuria, were not included.

The performance of any prediction model is influenced by effective treatment measures, such as antihypertensive drugs, magnesium sulphate and delivery, which reduces the probability of adverse outcomes. To avoid such bias, we included management strategies such as need for antihypertensive drugs and magnesium sulphate as predictors. Furthermore, as delivery is considered to be the cure for the condition, we incorporated preterm delivery before 34 weeks’ gestation as a component of the maternal composite adverse outcome. We considered early preterm delivery to be indicative of the severity of the condition, as clinicians usually aim to prolong pregnancy beyond 34 weeks’ gestation to reduce prematurity-related complications in the neonate unless there are overwhelming concerns about the health of the mother. This approach has led to the inclusion of important tests such as measurement of BP, proteinuria and the need for antihypertensives and magnesium sulphate as significant predictors in the final PREP models.

To ensure that the prediction model could be applied in clinical practice, the findings at the time of baseline (i.e. at diagnosis) should be used to assess the risk. Many models include the worst value of the predictor in the same time period used for outcome ascertainment. This is likely to overestimate the predictive performance of the model. Rule of thumb indicates that at least 10 events should be available per candidate predictor for model development. Compared with the PIERS model, that evaluated 34 predictors for 106 outcomes at 48 hours, the survival analysis approach taken with the PREP-S model allowed us to have sufficient sample size to predict complications accurately at various time points, including 48 hours after diagnosis. The PIERS model did not provide estimates of overall risk of complications in pregnancy and reported a discrimination index of just > 0.7 for prediction of adverse outcomes by 1 week. The PREP-L model showed good discrimination of > 0.8 to predict overall risk until discharge and the PREP-S model had an estimate of > 0.75. The performance validated well in the external data set with good discrimination and calibration. Given the potentially rapid changes in predictor values over time in pregnancy, these measures are impressive for their predictive power at the time of diagnosis of the condition.

Implications for clinical practice

The PREP models were developed with the explicit purpose of providing relevant information to mothers and clinicians on individualised risks at the time of diagnosis of pre-eclampsia. The Microsoft Excel file is user-friendly and easily accessible. It should be emphasised that the PREP models should not be used to choose between administration or non-administration of antihypertensives and magnesium sulphate, which are predictor variables. The risk estimates provided by the model are relevant to women who are managed as per the current clinical guidelines. 55 For example, when managing women admitted with very high BP, the clinicians are expected to manage the mother as per current guidelines with antihypertensives and, if appropriate, magnesium sulphate, and not to base treatment on the probability of risk provided by the model. However, the PREP-S model will be useful in providing the mother’s individualised risk of adverse outcomes at various time points, such as 48 hours after diagnosis, given the clinical characteristics and the choice of treatment.

The PREP-L model provides mothers with the overall risk of experiencing an adverse outcome by the time of discharge. The PREP-L model had high discrimination and calibration estimates in both development and validation data sets, and thus appears accurate and transportable to the non-UK populations examined. Clinicians and mothers should be informed that the model provides overall risks by discharge and should not be used to plan immediate management.

The calibration of the PREP-S model was good in the development data set, as expected, and reasonable in the validation cohort, especially for those with high risk, which suggests that women deemed to be at high risk of outcomes are also more likely to experience the outcome. The PREP-S model can be used as a tool for triaging mothers with a diagnosis of pre-eclampsia before 34 weeks’ gestation to decide on the optimal place of delivery. In women identified as high risk by the model, efforts should be made for early transfer of the mother to a tertiary unit for neonatal care, in addition to care for the mother. As the tool provides risk estimates at various time points, resources can be mobilised appropriately when required for transfer of the mother. The risk estimates will identify mothers who require prophylactic corticosteroids when preterm delivery is anticipated. In addition, this tool will allow neonatologists to provide individualised prognostic estimates for the baby after delivery, depending on the predicted risk of complications and for that gestational age.

Research recommendations

The PREP models assessed the risk of composite adverse outcomes. Individual patient data meta-analysis of the studies evaluating the prediction of the accuracy of tests for complications would provide an increased sample size to assess the risk of individual outcomes or outcomes grouped by the organ system involvement. We have undertaken the first two stages (model development and external validation) in prognostic research aimed at improving patient care. A final evaluation of the impact of PREP models on clinical practice is required, especially in the improvement of health outcomes for the mother and baby, but is beyond the remit of this project. In other words, further research may examine the impact of implementing the PREP-S and PREP-L models into clinical practice, in terms of their uptake by clinicians and their impact on patient outcomes. This might be in the form of a cluster randomised trial, for example, with practices randomised to either using or not using the PREP-S and PREP-L models.

Contributions of authors

Shakila Thangaratinam (Professor of Maternal and Perinatal Health) designed and led the project, provided clinical direction and prepared and edited the final report.

John Allotey (Senior Trials Co-ordinator) oversaw the day-to-day management of the study, ensured the study protocol was followed at participating hospitals and assisted in preparing the draft and final version of the manuscript.

Nadine Marlin (Statistician) performed the statistical analysis of the study and contributed to Chapter 2 of the final report.

Ben W Mol (Professor of Obstetrics and Gynaecology) provided clinical direction and edited the final version of the report.

Peter Von Dadelszen (Professor of Maternal Fetal Medicine) provided the PIERS data set used for validation of the PREP model.

Wessel Ganzevoort (Gynaecologist, Fellow of Perinatology) provided the PETRA data set used for validation of the PREP model.

Joost Akkermans (Research Physician, Resident Obstetrician and Gynaecologist) provided the PETRA data set used for validation of the PREP model.

Asif Ahmed (Pro-Vice-Chancellor for Health) contributed to the protocol and study development.

Jane Daniels (Deputy Director/Senior Research Fellow) contributed to the write-up of the final report.

Jon Deeks (Professor of Biostatistics, Joint School Research Lead) contributed to the protocol and study development.

Khaled Ismail (Professor of Obstetrics and Gynaecology) contributed to the protocol and study development.

Ann Marie Barnard (CEO Action on Pre-eclampsia Charity) helped draft and review the lay summary, contributing to the writing of the final report.

Julie Dodds (Senior Clinical Trials Manager) provided management direction by supervising the study co-ordinator (John Allotey), ensuring the implementation of the protocol, preparing the data for analysis and editing the final report.

Sally Kerry (Reader in Medical Statistics) provided guidance and management to the statistician (Nadine Marlin), oversaw the statistical analysis and edited the final report.

Carl Moons (Professor of Clinical Epidemiology) provided statistical guidance on analysis of data and presentation of the results and contributed to the write-up of the final report.

Richard D Riley (Professor of Biostatistics) provided statistical guidance, contributed to the analysis plan for the study and edited the final report.

Khalid S Khan (Professor of Women’s Health and Clinical Epidemiology) designed the project, provided clinical and overall direction, contributed to and edited the final report.

Publication

Thangaratinam S, Allotey J, Marlin N, Dodds J, Cheong-See F, von Dadelszen P, et al. Prediction of complications in early-onset pre-eclampsia (PREP): development and external multinational validation of prognostic models. BMC Med 2017;15:68. https://doi.org/10.1186/s12916-017-0827-3

Data sharing statement

Study data can be obtained from the corresponding author on request.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health.