Assessing prognosis and prediction of treatment response in early rheumatoid arthritis: systematic reviews

Article history

The research reported in this issue of the journal was funded by the HTA programme as project number 14/151/08. The contractual start date was in June 2016. The draft report began editorial review in August 2017 and was accepted for publication in February 2018. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

John Stevens is a shareholder of GlaxoSmithKline plc (GSK House, Middlesex, UK), AstraZeneca plc (Cambridge, UK) and Shire Pharmaceuticals Group plc (Shire plc, Dublin, Republic of Ireland).

Copyright statement

© Queen’s Printer and Controller of HMSO 2018. This work was produced by Archer et al. under the terms of a commissioning contract issued by the Secretary of State for Health and Social Care. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

2018 Queen’s Printer and Controller of HMSO

Description of the health problem

Measurement of key outcomes in rheumatoid arthritis

Major outcomes in RA include measures of joint destruction (e.g. radiographic progression), disease activity [as assessed via, for example, Disease Activity Score-28 (DAS28)] and disability (as assessed via, for example, HAQ scores).

Radiographic progression is frequently measured in clinical trials and observational research. 15 The two main scoring systems available for measuring radiographic progression are the Larsen et al. 16 system and the Sharp et al. 17 system. The van der Heijde18 modification of the Sharp system includes both hands and feet, erosions, joint-space narrowing and a range of joints.

In the UK, monitoring the progression of RA is often undertaken using the DAS28 in terms of swelling (SW28) and of tenderness to the touch (TEN28). The DAS28 also incorporates erythrocyte sedimentation rate (ESR) and a subjective assessment on a scale of 0–100 made by the patient regarding disease activity in the previous week.

The equation for calculating DAS28 is as follows:19

A second version of the DAS28 using C-reactive protein (CRP) levels instead of ESR exists. The DAS28 can be used to classify both the disease activity of the patient and the level of improvement. Patients with a DAS28 of ≤ 3.2 are classed as having inactive disease, patients with a DAS28 of > 3.2 and ≤ 5.1 are regarded as having moderate disease and those with a DAS28 of > 5.1 are regarded as having very active disease. 19

The HAQ is a key measure of patient functional disability. 20 It is a patient-completed assessment resulting in scores ranging from 0 to 3 (with higher scores indicating greater disability).

Background to prognosis research

Background to stratified medicine research

Interpretation of the evidence regarding patient and/or disease characteristics

There are four scenarios that may arise depending on whether a covariate is (or is not) prognostic for at least one treatment and whether a covariate is (or is not) a treatment effect modifier: these are illustrated below for a comparison of two treatments with respect to a single covariate, which may be discrete or continuous.

Scenario 1: prognostic variable for both treatments, but not a treatment effect modifier (Figure 1)

Given two treatments coded such that t₁ = 0 and t₂ = 1, the mean response (i.e. expected value) for a person with a baseline value of x_j who receives treatment t₁ is:

⁽²⁾ E [ y i j ] = β 0 + β 1 x j + β 2 t i .

For treatment t₁ this is:

⁽³⁾ E [ y 1 j ] = β 0 + β 1 x j ,

and for treatment t₂ this is:

⁽⁴⁾ E [ y 2 j ] = ( β 0 + β 2 ) + β 1 x j .

FIGURE 1.

Scenario 1: prognostic variable for both treatments but not a treatment effect modifier.

In this scenario, the treatment effect is β₂, which is constant at each value of the predictor; it is also possible for a baseline variable to be a prognostic and for there to be no treatment effect (in which case, the two regression lines would be superimposed).

Scenario 2: prognostic variable for both treatments and a treatment effect modifier (Figure 2)

Given two treatments coded such that t₁ = 0 and t₂ = 1, the mean response (i.e. expected value) for a person with a baseline value of x_j who receives treatment t₁ is:

⁽⁵⁾ E [ y i j ] = β 0 + β 1 x j + β 2 t i + β 12 t i . x j .

For treatment t₁ this is:

⁽⁶⁾ E [ y 1 j ] = β 0 + β 1 x j ,

and for treatment t₂ this is:

⁽⁷⁾ E [ y 2 j ] = ( β 0 + β 2 ) + ( β 1 + β 12 ) x j .

FIGURE 2.

Scenario 2: prognostic variable for both treatments and a treatment effect modifier.

In this scenario, the treatment effect at x_j is β₂ + β₁₂x_j, which depends on the value of the baseline variable; it is also possible that β₂ = 0.

Scenario 3: not a prognostic variable for one treatment, but a treatment effect modifier (Figure 3)

Given two treatments coded such that t₁ = 0 and t₂ = 1, the mean response (i.e. expected value) for a person with a baseline value of x_j who receives treatment t₁ is:

⁽⁸⁾ E [ y i j ] = β 0 + β 1 x j + β 2 t 1 + β 12 t i . x j .

For treatment t₁ this is:

⁽⁹⁾ E [ y 1 j ] = β 0 ,

and for treatment t₂ this is:

⁽¹⁰⁾ E [ y 2 j ] = ( β 0 + β 2 ) + β 12 x j .

FIGURE 3.

Scenario 3: not a prognostic variable for one treatment, but a treatment effect modifier.

In this scenario, the treatment effect at x_j is β₂ + β₂x_j, which depends on the value of the baseline variable; it is also possible that β₂ = 0.

Scenario 4: not a prognostic variable for either treatment and not a treatment effect modifier (Figure 4)

Given two treatments coded such that t₁ = 0 and t₂ = 1, the mean response (i.e. expected value) for a person with a baseline value of x_j who receives treatment t₁ is:

⁽¹¹⁾ E [ y i j ] = β 0 + β 1 x j + β 2 t i + β 12 t i . x j .

For treatment t₁ this is:

⁽¹²⁾ E [ y 1 j ] = β 0 ,

and for treatment t₂ this is:

⁽¹³⁾ E [ y 2 j ] = β 0 + β 2 .

FIGURE 4.

Scenario 4: not a prognostic variable for either treatment and not a treatment effect modifier.

In this scenario, the treatment effect at x_j is β₂, which is independent of the baseline variable; it is also possible for there to be no treatment effect (in which case, the two regression lines would be superimposed).

Stratified medicine research is concerned with identifying treatment effect modifiers, as illustrated by scenarios 2 and 3.

Review question

The review question as outlined in the Health Technology Assessment (HTA) commissioning brief and final protocol was as follows: what test or combination of clinical, laboratory and imaging tests gives the best assessment of prognosis in RA, and how well do they predict response to treatment?

Assessment structure

The assessment structure consisted of systematic reviews with appropriate predefined subgroup analyses.

It was anticipated in the final protocol that the factors most likely to be of use for prognosis and prediction of treatment response would be assessed through meta-analysis of available aggregate-level data. The de novo development of a specific prediction model and the use of individual participant data (IPD) were outside the remit of this assessment.

Assessment structure

This assessment consisted of two related systematic reviews. The first systematic review (review 1, clinical prediction models) investigated the use of assessment tools and tests in the evaluation of prognosis in early RA patients. The second systematic review (review 2, prediction of treatment response) determined the ability of selected assessment tools and tests to predict the response to specific treatments.

The systematic reviews were informed by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (www.prisma-statement.org/) and current good practice in prognostic reviews, as advocated by the Cochrane Prognosis Methods Group (http://methods.cochrane.org/prognosis/).

A final protocol for this assessment was registered on PROSPERO (as record CRD42016042402).

Scoping of the assessment

The searches for reviews in this assessment were structured in two phases. The phase 1 scoping searches were performed to determine the approximate extent of the evidence base relevant to the assessment and to inform the discussion of prognostic and predictive variables for inclusion in conjunction with clinical advisors (full details are provided in Appendix 1).

Following discussions with two expert clinical advisors who manage patients with early RA in the UK (as described in Acknowledgements), the review team selected variables for inclusion. The selection of prognostic and predictive variables was based on:

• tests and assessment tools (e.g. selected laboratory tests, imaging tests and clinical assessment measures) the variables being readily available and used in UK clinical practice (and, therefore, genetic markers were not included by the review team)

• the clinical experience of advisors in evaluating prognosis/treatment response in patients

• the initial scoping of literature in the area by the review team.

The selected prognostic and predictive variables were used in the development of the full searches for reviews 1 and 2.

Justification of review approach

In view of the anticipated large number of search results, it was necessary for the review team to revise the intended review approach in order to maintain the feasibility of the assessment within the available resources and time scales. Additional details of protocol deviations are provided in Appendix 2.

Therefore, the two related systematic reviews were planned as detailed in the following sections.

Methods for review 1 (clinical prediction models)

Methods for review 2 (prediction of treatment response)

Quantity of research available

Searches for evidence (Figure 5) identified 22 model development studies and one combined model development/external validation study, reporting a total of 39 clinical prediction models for the prediction of major outcomes, including radiographically assessed joint damage, the HAQ score and the DAS28 (note that one publication56 considered the development of models for multiple outcomes; however, because of unclear reporting, the individual models could not be summarised and so have been counted as a single entry for the purposes of the assessment). The study by Bombardier et al. 57 was available as a conference abstract only and a full-text publication was not available. Six external validation studies (including the combined model development/external validation study) of eight previously proposed clinical prediction models for radiographically assessed joint damage outcomes were identified. No studies assessing the impact of the use of clinical prediction models in the clinical management of early RA were identified.

FIGURE 5.

Review 1: study selection represented as an adapted PRISMA flow diagram. 55

Quality of the research available

Potential sources of bias in participant selection, predictors, outcomes, model development and validation are discussed in the following sections and tabulated in Appendix 6.

Types of included prediction model studies

The TRIPOD statement26 was developed with the aim of improving the quality of reporting of prediction model studies. Collins et al. 26 described six categories (listed below) that can be used to classify studies according to their aim (development or/and validation) and the methodology used.

The inclusion criteria for review 1 required that studies developing clinical prediction models apply some form of internal validation to quantify the predictive performance of the developed model (e.g. calibration and discrimination), with the exception of cases in which the clinical prediction model had been subsequently externally validated. In these cases, the external validation results may be considered to be more informative than the omitted internal validation results and it was deemed necessary to include the original development paper for completeness. These studies did not have a category according to the TRIPOD statement, and it was therefore necessary to introduce a new TRIPOD category (Type 0) to allow the description of these studies.

The tailored TRIPOD classification categories used in this review (based on Collins et al. 26) are as follows:

• Type 0 – development of a clinical prediction model in which the predictive performance is not evaluated in the development paper, but an evaluation of the predictive performance has been considered in a separate publication.

• Type 1a – development of a clinical prediction model with an evaluation of the predictive performance using the same data set (apparent performance).

• Type 1b – development of a clinical prediction model using the whole data set and an evaluation of the predictive performance using resampling (e.g. bootstrapping or cross-validation).

• Type 2a – random splitting of data into two groups, the first for clinical prediction model development and the second for the testing of its predictive performance.

• Type 2 – non-random splitting of data into two groups, the first for clinical prediction model development and the second for the testing of its predictive performance.

• Type 3 – development of a clinical prediction model in one data set and an evaluation of the predictive performance on separate data (e.g. from a different study).

• Type 4 – the evaluation of the predictive performance of an existing clinical prediction model using separate data (external validation).

The TRIPOD classification categories of the 28 included studies are presented in Table 1. The PROBAST49 study-type classifications of the included studies are also presented in Table 1.

Of the 23 clinical prediction model development studies, the vast majority (n = 16) were TRIPOD type 1a {i.e. Berglin et al. ,59 Visser et al. 79 [Behandelings Strategie (BeSt)], Combe et al. 62 (Combe A), Combe et al. 63 (Combe B), de Vries-Bouwstra et al. ,65 Degboé et al. ,66 Dirven et al. ,67 Drossaers-Bakker et al. ,56 Fautrel et al. 69 [Études et suivi des polyarthrites indifférenciées récentes (ESPOIR)], Forslind et al. ,70 Graell et al. ,71 Houseman et al. ,72 Saevarsdottir et al. 73,74 [Swedish Farmacotherapy (SWEFOT)], Sanmartí et al. ,75 Syversen et al. 76 and van Steenbergen et al. 77}, with validation conducted in exactly the same data as those used for clinical prediction model development. Fewer studies were categorised as type 1b (n = 1; i.e. Bansback et al. 58), using techniques to evaluate performance and optimism of the developed model, type 2a (n = 2; i.e. Brennan et al. 60 and Dixey et al. 68), using a data-splitting approach for development and validation, or type 0 {n = 2; i.e. Bombardier et al. 57 [Study Of New-Onset Rheumatoid Arthritis (SONORA)] and Centola et al. 61}, with no internal validation presented for Bombardier et al. 57 Although internal validation was conducted for Centola et al. ,61 it was developed for a different outcome measure and so the study is categorised as type 0 for the purpose of the current review. One study {n = 1; i.e. Vastesaeger et al. 78 [Active controlled Study of Patients receiving Infliximab for the treatment of Rheumatoid arthritis of Early onset (ASPIRE)]} considered validation in an external population in addition to model development (type 3); however, the external validation population comprised patients with established RA and, therefore, was outside the scope of this assessment. The methods of internal validation and reporting of models in the included development studies are described in greater detail in Model development.

Five additional included studies were external validations of clinical prediction models (TRIPOD class 4). Three of these studies (i.e. De Cock et al. ,80 Granger et al. 82 and Heimans et al. 84) externally validated a total of seven clinical prediction models (i.e. ASPIRE CRP,78 ASPIRE ESR,78 BeSt,79 ESPOIR,69 SONORA,57 SWEFOT73 and Syversen76). The remaining two external validation studies (Hambardzumyan et al. 83 and Markusse et al. 85) evaluated the use of a multibiomarker disease activity (MBDA) test61 (developed as a measure of disease activity) in the prediction of eligible clinical outcomes. In addition to the seven purely external validation studies, the de Punder model development paper64 (TRIPOD category 1b/4) also externally validated the BeSt79 and ESPOIR69 models.

Two additional external validation studies were identified but not included. 73,86 The study reported by Durnez et al. 86 is considered to be a precursor study to the work by De Cock et al. ,80 and so the work by De Cock et al. 80 is utilised in this assessment as being more recent and comprehensive in the coverage of validated models. The application of the BeSt model79 was considered by Saevarsdottir et al. 73 (TRIPOD category 1a/4) using the SWEFOT data set. However, the presented results did not provide sufficient data to be considered as informative to the current review, as no summary statistic of overall performance (e.g. c-statistic) was reported. Saevarsdottir et al. 73 presented the allocation of individuals to risk matrix categories, combined for the whole validation sample rather than separately by treatment group.

Description of data sources for the development of clinical prediction models and external validation

Moons et al. 31 advocated the use of cohort study data in the development of clinical prediction models, which are preferably prospective in design to allow for the greater completeness of data collection. 31 Trial data are also an appropriate data source, although it has been noted that trial eligibility criteria may result in more restricted and less generalisable data than registry cohort studies. 31 All identified clinical prediction models for review 1 were developed in either observational cohort or registry data or in existing cohorts from intervention trials in early RA populations and, therefore, can be considered to have been developed in appropriate data sources. Studies were longitudinal in design, with potential predictors measured in an early RA population at baseline or in an early disease stage before the measurement of a specified outcome at a subsequent time point. There was one main exception to this, Houseman et al.,72 which is discussed further in Description of predictors. The characteristics of the data sources used in the development and external validation of the included clinical prediction models are tabulated in Table 2.

Four included clinical prediction model development studies used RCT data. 67,73,78,79 The models by Vastesaeger et al. 78 were developed in the ASPIRE RCT. 87 Both the models by Visser et al. 79 and Dirven et al. 67 were generated using data from the BeSt RCT. 89 Data from the SWEFOT RCT90 were used by Saevarsdottir et al. 73 to build the SWEFOT clinical prediction model.

The Leiden Early Arthritis Clinic cohort was a common data source, being used in the development of models by de Vries-Bouwstra et al. ,65 Drossaers-Bakker et al. 56 and van Steenbergen et al. 77 The ESPOIR cohort was used to build models by Degboé et al. 66 and Fautrel et al. 69 Combe et al. developed models for the prediction of radiographic damage/progression (Combe A)62 and HAQ score (Combe B)63 in French cohort data. The Graell71 and Sanmartí75 clinical prediction model development studies used the same Spanish cohort data source. The models by Bansback et al. ,58 Berglin et al. ,59 Brennan et al. ,60 Centola et al. ,61 de Punder et al. ,64 Bombardier et al. 57 (SONORA), Dixey et al. ,68 Forslind et al. ,70 Houseman et al. 72 and Syversen et al. 76 were developed in cohort study data sets.

The five included external validation studies were also performed in appropriate data sources, being either intervention trials {i.e. Hambardzumyan et al. 83 (SWEFOT), Heimans et al. 84 [Induction therapy with MTX and Prednisone in Rheumatoid Or Very Early arthritis Disease (IMPROVED)], Markusse et al. 85 (BeSt)} or cohort study data [i.e. De Cock et al. 80 (Verschueren cohort), Granger et al. 82 (ESPOIR)].

Description of population characteristics

As outlined in Table 3, the majority of clinical prediction models were developed in populations that had satisfied the 1987 ACR criteria for the diagnosis of RA (i.e. ASPIRE,78 Bansback et al. ,58 Berglin et al. ,59 BeSt,79 Brennan et al. ,60 Centola et al. ,61 Combe (A),62 Combe (B),63 de Punder et al. ,64 de Vries-Bouwstra et al. ,65 Dixey et al. ,68 Dirven et al. ,67 Forslind et al. ,70 Graell et al. ,71 Houseman et al. ,72 Sanmartí et al. ,75 SWEFOT,73 Syversen et al. 76 and van Steenbergen et al. 77). Degboé et al. 66 used the 2010 ACR/EULAR criteria in their diagnosis of patients, whereas in the ESPOIR model data source, patients had received a RA diagnosis from their rheumatologist (and the proportion of patients meeting the ACR/EULAR 2010 criteria was reported). The RA diagnosis method used by Drossaers-Bakker et al. 56 was not specified. However, all patients were described as having RA, and two other included studies (i.e. de Vries-Bouwstra et al. 65 and van Steenbergen et al. 77), which were also developed using data from the Leiden Early Arthritis Clinic, both had used the 1987 ACR criteria (and, therefore, the review group has assumed a similar use of the 1987 ACR criteria by Drossaers-Bakker et al. 56). The amount of information available in the conference abstract by Bombardier et al. 57 describing the development of the SONORA clinical prediction model was limited. It was stated that patients included in the SONORA study were diagnosed with RA by a rheumatologist. The populations used in the external validation studies by Hambardzumyan et al. 83 and Markusse et al. 85 were diagnosed with RA according to the 1987 revised ACR criteria. The patients of the ESPOIR cohort, included in the Granger et al. 82 external validation study, were diagnosed with RA by their rheumatologist (and the proportion meeting the ACR/EULAR 2010 criteria was reported). Heimans et al. 84 reported that the ACR/EULAR 2010 criteria were used for the diagnosis of RA in the mixed RA/undifferentiated arthritis population used for their external validation study. 84 In the De Cock et al. 80 external validation study, all patients (in the Verschueren cohort) were described as being newly diagnosed with early RA.

As stated in Study selection, early RA was defined in conjunction with clinical experts as being within 2 years of the onset of symptoms. Included studies reported either the duration of symptoms or the duration of disease at baseline (with disease duration at baseline considered by the review team to be equivalent to symptom duration in the absence of further information). The studies set an upper threshold of symptom/disease duration for study eligibility and/or reported the median/mean symptom/disease duration at baseline, as presented in Table 3. Compared across studies, the baseline median/mean symptom/disease duration for included populations was typically approximately 5–6 months. The ESPOIR,69 Combe (A)62 and Combe (B)63 models were derived from patients with earlier RA (mean duration of disease of 15 weeks, 3.3 months and 3.6 months, respectively). The populations in the models by Drossaers-Bakker et al. 56 (median symptom duration of 1 year) and Syversen et al. 76 (mean disease duration of 2.3 years) had disease of a longer duration. It is noted that the population used to develop the Syversen et al. 76 clinical prediction model was outside the 2-year limit used to define early RA. However, this study was eligible for inclusion because it was externally validated in a population that did meet the inclusion criteria (i.e. De Cock et al. 80). The populations in the data sources used for the six included external validation studies were all of ≤ 2-year symptom/disease duration at baseline.

The DAS28 disease severity score at baseline was tabulated when available. A DAS28 of ≥ 5.1 is classed as severe disease, whereas a DAS28 of 3.2–5.1 is regarded as moderate disease severity. 19 Unfortunately, the DAS28 at baseline was not available for the majority of studies. Five clinical prediction models (i.e. ESPOIR,69 Forslind et al. ,70 Graell et al. ,71 Sanmartí et al. 75 and SWEFOT73) were developed in populations that were defined as having severe disease activity based on the DAS28. Two external validation studies were performed in populations with severe DAS28 (i.e. Granger et al. 82 and Hambardzumyan et al. 83). In some studies, the DAS28 at baseline was reported only for subgroups (rather than for the total population), but indicated a severe DAS28. The cohort used in the De Cock et al. 80 external validation was borderline between the moderate and severe DAS28 categories [mean DAS28-CRP of 4.91 (SD 1.22)]. 80 Therefore, unfortunately, sufficient data were not available to evaluate the performance of predictors across moderate and severe DAS28 populations.

The treatment histories and concurrent treatments in included studies are described in Table 3. Additional population characteristics at baseline are presented in Appendix 7, Table 30.

The data sources and participant selection in the included studies were assessed using domain 1 of the PROBAST tool49 (see Appendix 6).

Appropriate data sources were considered to have been used for all model development and external validation studies. Inclusions and exclusions of participants were considered by the review authors to be acceptable overall. However, it was noted that the Drossaers-Bakker model56 included only female participants, which may affect the generalisability of the model. As explained previously, the Syversen model76 was developed in a population with a longer mean disease duration [mean 2.3 years (SD 1.2 years)]. Although the Syversen model76 has been externally validated in a relevant population, this may affect the performance of the model.

It was stated in the final protocol that, for review 1, patients should have been diagnosed with early RA according to established criteria. The methods used for RA diagnosis in included studies are presented in Table 3. The lack of clarity around the method of RA diagnosis used in the populations in the Drossaers-Bakker56 and SONORA57 clinical prediction models and the De Cock external validation study80 may contribute to a potential source of bias. The De Cock external validation study80 was based on a cohort recruited at the University of Leuven, Belgium. The original report for this study cohort described patients as being ‘newly diagnosed with RA.’ Although the specific method used for diagnosis was not reported, the entire cohort was described as having a diagnosis of RA and the duration of symptoms meets our definition of early RA. Therefore, in light of the fact that the De Cock external validation study80 presents key informative evidence comparing the performance of multiple clinical prediction models and is described as being based on an early RA cohort, this study has been included. 80

Three model development studies were rated as having an unclear risk of participant selection bias on PROBAST domain 1 [i.e. Drossaers-Bakker et al. ,56 Bombardier et al. (SONORA)57 and Syversen et al. 76]. All other remaining model development studies were rated as being at a low risk of participant selection bias. Two external validation studies were rated as having an unclear risk of bias for participant selection (i.e. De Cock et al. 80 and Heimans et al. 84). All other external validation studies were categorised as having a low risk of bias for participant selection.

Description of predictors

The candidate predictors considered in the development of clinical prediction models are presented in Table 4. The candidate predictors were interpreted by the review team to be all predictors considered in the model development process (and were not limited to those included in the multivariable analysis) in line with the definition applied in CHARMS (i.e. Moons et al. 48).

The timings of the measurement of candidate predictors in the included model development studies were noted. Measurement was at baseline for all studies, although some studies (i.e. Bansback et al. ,58 Berglin et al. ,59 Dixey et al. 68 and Houseman et al. 72) also measured predictors at later time points. Anti-CCP positivity was measured at the extension visit (at 8.2 years of follow-up) in the Houseman et al. study72 and was also included in the final clinical prediction models. The inclusion of predictors measured at substantially later time points is potentially problematic to the validity of the resulting clinical prediction model. Variables measured at later time points are problematic: they cannot be used to make predictions about outcomes in patients with newly diagnosed RA in clinical practice because the observation will not be available until after the decision has been made to treat the patient. The outcome will depend on the specific sequence of treatments administered, and such observations will bias the model relative to models that use only baseline information. None of these models has been externally validated, which highlights the difficulty in their application.

Given the incomplete reporting in the conference abstract, it was unclear which candidate predictors were considered in the SONORA model. 57

We considered whether or not all relevant candidate predictors were analysed (i.e. whether or not all variables selected for inclusion in conjunction with clinical advisors were considered in the model development studies as candidate predictors). When selected variables of interest were not examined as candidate predictors, these studies were rated as being at an unclear risk of bias (as it was not clear what potential contribution the untested predictors may have had).

Description of outcomes

The outcomes assessed by the included studies are presented in Table 5 for the model development studies and in Table 6 for the external validation studies.

Among the clinical prediction model development studies, the majority assessed radiographic joint damage as an outcome. Six models were developed to predict radiographic progression [modified Sharp–van der Heijde score (SHS)] at 1 year: ASPIRE,78 BeSt,79 de Vries-Bouwstra,65 Degboé,66 ESPOIR69 and SWEFOT. 73 The SONORA model57 assessed radiographic progression (original Sharp score) at 1 and 2 years. Combe (A)62 predicted radiographic score and radiographic progression (modified SHS) at 3 years. Van Steenbergen et al. 77 and Houseman et al. 72 studied radiographic progression (modified SHS) at 6 and 8.2 years, respectively. The Syversen model76 predicted radiographic progression (modified SHS) over the longest follow-up period (i.e. 10 years). Five studies evaluated radiographic joint damage/progression, measured using the Larsen score (i.e. Brennan et al. 60 at 1 year; Berglin et al. ,59 Forslind et al. 70 and Sanmartí et al. 75 at 2 years; and Dixey et al. 68 at 3 years). De Punder et al. 64 assessed radiographic progression (according to the Ratingen score) at 3 years.

Five other clinical prediction models were designed for the prediction of other eligible outcomes. Dirven et al. 67 assessed short-term functional disability (HAQ score of ≥ 1) after 3 months of treatment. The outcome used in the Graell model71 was the modified HAQ (MHAQ) at 2 years. Bansback et al. 58 and Combe (B)63 both modelled the HAQ score at 5 years [with Combe (B)63 also modelling the 3-year HAQ score]. Drossaers-Bakker et al. 56 included a broad range of outcomes, categorised as (1) body functions and structure (impairment), (2) activities at the individual level (disability), (3) participation in society (handicap) and (4) disease course. However, because of unclear reporting, the performance of the individual models could not be summarised, and so this study was counted as a single model in the combined total of 39 included models.

As summarised above, the majority of studies considered binary outcomes, based on dichotomising an observed continuous variable. Although these outcomes are considered to be standard practice, it is worth noting that dichotomising continuous variables has previously described disadvantages. 92,93 The magnitude of the association will depend on the cut-off point used to dichotomise the variable.

As described in Table 6, all five external validation studies and the study by de Punder et al. ,64 which considered development and external validation, considered radiographic progression at 1 and/or 2 years as the outcome variable.

The external validation studies considered a total of eight previously developed clinical risk prediction models, and conducted a total of 31 external validations. In addition to describing the outcome considered in the external validation study, Table 6 lists which clinical prediction models were considered and whether or not the outcome considered by the external validation study was consistent with that in the original model development paper.

For 17 of the 31 external validations that were conducted, the outcome considered by the external validation was different to that used for the model development. Five of these studies (labelled as N^d in Table 6) externally validated the MBDA clinical prediction model (originally developed to predict the DAS28) for the prediction of radiographic progression outcomes. Ten external validations (labelled as N^a and N^b in Table 6) externally validated risk models using the outcomes at a different time point than that defined in the original model development paper. Two external validations (labelled as N^c and N^e in Table 6) externally validated risk models using different thresholds to define events, compared with that used in the original model development study.

The handling of outcomes in included studies was critically appraised using domain 3 of the PROBAST tool (see Appendix 6). The included studies were rated as being at a low or unclear risk of bias.

Model development

The methods used in the development of the included clinical prediction models are summarised in Table 7. Clinical prediction models were typically developed in the included studies using data from patients with complete cases only [i.e. having complete predictor and outcome data (e.g. Bansback,58 Brennan,60 Combe A,62 Combe B,63 de Punder,64 de Vries-Bouwstra,65 Degboé,66 Dirven,67 Dixey,68 Drossaers-Bakker,56 Forslind,70 Graell,71 Sanmartí,75 SWEFOT73 and Syversen76)].

Continuous variables were frequently converted to dichotomous/trichotomous variables during the clinical prediction model development process. This was the case for seven of the externally validated models developed for the prediction of radiographic progression (all externally validated models, with the exception of MBDA). This conversion is not considered to be good practice. 31 The categorisations used for the variables included in the final models are described in Table 8.

Clinical prediction models were developed using multivariable regression, with the majority of studies using logistic regression [i.e. ASPIRE78, Bansback et al. ,58 Berglin et al. ,59 BeSt,79 Brennan et al. ,60 Combe et al. (A),62 de Punder et al. ,64 Degboé et al. ,66 Dixey et al. ,68 Dirven et al. ,67 Drossaers-Bakker et al. ,56 ESPOIR,69 Forslind et al. ,70 Graell et al. ,71 Houseman et al. ,72 Sanmartí et al. ,75 SWEFOT et al. 73 and Syversen76], as the outcome variables were binary. Backwards selection was reported for three models (i.e. Berglin et al. ,59 de Punder et al. 64 and Dirven et al. 67) as being used in the selection of the final variables in the model, whereas forwards selection was used in four model development studies (i.e. Drossaers-Bakker et al. ,56 Forslind et al. ,70 Graell et al. 71 and Houseman et al. 72). There were variations in how the selection was implemented (e.g. levels of statistical significance and combination with other techniques, such as bootstrapping, see Table 7).

The BeSt model development (by Visser et al. 79) and the model by Graell et al. 71 were the only two model development studies to specifically describe having considered the presence of interactions between variables. As Visser et al. 79 also considered the effect of treatment, this is of relevance to review 2 and is discussed in further detail in Results: synthesis of primary studies.

The majority of the included clinical prediction model development studies used a limited form of internal validation, with apparent predictive performance tested within the original development data set (TRIPOD statement type 1a). Testing the apparent performance of a clinical prediction model using the same data in which the model was developed can lead to optimistic estimates of performance. 26 This is attributed to ‘overfitting’, whereby there are too few outcome events in relation to the number of candidate predictors. Therefore, the performance estimates derived from the included TRIPOD statement type 1a studies should be viewed with caution in the absence of more robust validation.

Ideally, clinical prediction model development studies should include a method of internal validation to quantify any optimism in the predictive performance. Two studies (i.e. Bansback et al. 58 and de Punder et al. 64) used bootstrapping for estimating model performance (i.e. TRIPOD statement type 1b). A further two studies (i.e. Brennan et al. 60 and Dixey et al. 68) used a data-splitting approach.

One study tested the performance of its developed clinical prediction models on separate data from a different study (TRIPOD statement type 3). Vastesaeger et al. 78 tested the performance of their ASPIRE model for the prediction of radiographic progression in data from ATTRACT (Anti-TNF trial in rheumatoid arthritis with concomitant therapy). As stated previously, as the ATTRACT trial was conducted in patients with established RA, this validation is outside the scope of this assessment and so is not considered further.

The form in which the clinical prediction models were presented determines the ease with which they may be used in clinical practice, and not all of the model development studies presented their models in sufficient detail for them to be applied to other populations. In order for clinicians to directly apply a model, one option is to present the full model regression coefficients (including the intercept term), which allows the model to be used to generate absolute risks. Alternatively, the absolute risks for each individual combination of variables (e.g. in a risk matrix format) could be presented. Of the 39 clinical prediction models, 18 were presented in a useable format.

Of the eight externally validated risk models, the majority were presented as risk matrices in the published report: ASPIRE78 (ESR and CRP), BeSt,79 Dirven et al. 67 and ESPOIR. 69 Two externally validated clinical prediction models (MBDA61 and Syversen et al. 76) presented the full model regression coefficients. Syversen et al. 76 also presented the final model graphically using a histogram. The SWEFOT73 models were not presented in a useable format in the model development publication, and for this reason they were not tested in the Granger et al. 82 external validation. However, the main SWEFOT model was considered in the De Cock et al. 80 external validation, based on results that are available in conference proceedings. 73

Some studies were noted as reporting ‘OR only’ in Table 7, indicating that the coefficient of the intercept term in the regression was not provided and, therefore, the model would not be able to be used to provide absolute risks.

The final predictors selected for use in the included clinical prediction models are presented in Table 8. The prognostic variables selected for consideration in the assessment in conjunction with clinical advisors were very commonly incorporated as predictors in the final clinical prediction models. However, the continuous variables were generally categorised, with the models applying a range of differing thresholds in their construction.

Performance results: clinical prediction model development studies

The performance of the included clinical prediction models, as reported in the original model development studies, is presented in Table 9. Discrimination (using the c-statistic) was recorded when available, in addition to information relating to the model calibration and overall model goodness of fit. Additional performance measures presented by the studies are shown in Appendix 7. Unless otherwise indicated, sample sizes and the number of events refer to the sample used for the final multivariable prediction models.

As previously discussed, good discriminative ability in the population used to develop the model would be expected. The external validation of all models in the same data set or multiple data sets is required to objectively compare the reported c-statistics. The results of these external validations are presented in Results: external validation studies.

Models predicting radiographic joint damage

Discrimination was reported using the c-statistic by six of the studies,64,66,69,72,77,79 which considered a total of 13 clinical prediction models. c-statistics for models predicting radiographic joint damage are summarised in a forest plot in Figure 6.

FIGURE 6.

Forest plot of predictive performance of the included clinical prediction models for radiographic joint damage based on internal validation. Anti-MCV, antimutated citrullinated vimentin; TT, treatment and traditional factors; TTG, treatment, traditional and genetic factors.

The BeSt model79 for the prediction of radiographic progression at 1 year demonstrated reasonably good discrimination, which was broadly comparable between the CRP and ESR models (CRP model c-statistic 0.81, 95% CI 0.77 to 0.86; ESR model c-statistic 0.80, 95% CI 0.75 to 0.85).

The de Punder extended and simplified models64 predicted radiographic progression at 3 years. The c-statistics obtained indicated moderate predictive performance (extended c-statistic 0.77, 95% CI 0.72 to 0.81; simplified c-statistic 0.75, 95% CI 0.70 to 0.80).

Four clinical prediction models were produced by Degboé66 for the prediction of radiographic progression at 1 year. The apparent performance (TRIPOD statement type 1a) of these models was reported. The c-statistics were lower than for other models predicting radiographic progression over 1 year [anti-CCP c-statistic 0.65, 95% CI 0.60 to 0.70; antimutated citrullinated vimentin (anti-MCV) c-statistic 0.63, 95% CI 0.58 to 0.68; anti-human citrullinated fibrinogen anti-body (AhFibA) c-statistic 0.65, 95% CI 0.60 to 0.71; high-level ACPA titre c-statistic 0.65, 95% CI 0.60 to 0.70].

The ESPOIR clinical prediction model69 for radiographic progression at 1 year yielded a c-statistic of 0.754 [95% CI not reported (TRIPOD statement type 1a)].

Two clinical prediction models were constructed by Houseman et al. 72 The apparent performances (TRIPOD statement type 1a) were reported. The first model (for radiographic progression at 8.2 years) showed good performance (with a c-statistic of 0.87). The second model (for absolute radiographic outcome at 8.2 years) showed similarly good discriminatory abilities (with a c-statistic of 0.84). It was noted that anti-CCP positivity was measured at the 8.2-year follow-up point, as opposed to baseline or in the early stages of disease, as discussed in Description of predictors.

c-Statistics were presented for the clinical prediction models by van Steenbergen et al. 77 for predicting radiographic progression at 6 years. For this study, an outcome variable with three categories was considered and the reported c-statistics were computed using Harrell’s c-statistic for ordinal data. 37 The model that included treatment plus traditional (i.e. selected demographic, clinical and biomarker) variables had a moderate performance (c-statistic 0.78, 95% CI 0.73 to 0.82). The addition of genetic factors to the model (treatment and traditional and genetic) increased the discrimination of the model slightly (c-statistic 0.82, 95% CI 0.77 to 0.86).

The model goodness of fit (R²) was only rarely reported. Low Nagelkerke’s R² values were obtained for the Berglin models (model 1: 0.21; model 2: 0.24);59 Forslind model 1 (radiological damage)70 yielded a Nagelkerke’s R² value of 0.5, whereas the fit for model 2 (radiological progression) was slightly poorer at 0.4. For the de Vries-Bouwstra model,65 the fit (R²) was 0.36. The R² for the van Steenbergen models77 increased when genetic factors were added to treatment plus traditional factors (0.43 from 0.36, an increase of 0.07; p = 0.06, adjusted R² increase = 0.03). Although Nagelkerke’s R² was reported in several publications and used to compare the overall model fit of competing models, it does not provide an assessment of the models’ predictive performance. A model may have a low R² value while still providing a useful prediction tool.

Model calibration was not frequently reported. De Punder et al. 64 assessed calibration using the Hosmer–Lemeshow test, with p-values being large (p > 0.4) for all clinical prediction models, indicating that there was no evidence to suggest a poor model fit. Calibration slopes and shrinkage factors were also presented for the de Punder extended model64 (shrinkage factor 0.90, 95% CI 0.88 to 0.92; calibration slope 1.0, 95% CI 0.82 to 1.21) and the de Punder simplified64 models (shrinkage factor 0.98, 95% CI 0.96 to 1.0; calibration slope 1.1, 95% CI 0.89 to 1.33). For the ESPOIR model,69 the use of the Hosmer–Lemeshow test was described but not presented.

Results: external validation studies

The performance results of the external validation studies are summarised in Table 10. Six external validation studies (including one combined model development/external validation study) were identified that tested the performance of a total of eight previously developed clinical prediction models. All studies focused on the prediction of radiographic joint damage.

The De Cock et al. 80 external validation study considered six previously developed clinical prediction models (i.e. ASPIRE94 CRP, ASPIRE94 ESR, BeSt,79 Syversen,76 SWEFOT73 and ESPOIR69). Note that the Syversen prediction model76 was referred to as SWEFOT 1 in the De Cock et al. external validation,80 whereas the SWEFOT model by Saevarsdottir et al. 73 was denoted as SWEFOT 2. De Cock et al. 80 presented results for three different time intervals: first year (baseline to year 1); second year (year 1 to year 2); and over 2 years (baseline to year 2). For this final assessment, progression was defined as total Sharp score (TSS) of > 10 over 2 years, as opposed to a TSS of > 5 over 1 year and, as such, differs from the outcome definition used in the model development studies.

The Granger external validation study82 presented results from four clinical prediction models that had been developed in other populations. Results were also presented for the ESPOIR model69 and an updated mESPOIR82 model; however, given that the ESPOIR model69 was initially developed in this population, this should not be considered as an external validation. The results are presented here for completeness, but are not used for the subsequent meta-analysis. Granger et al. 82 considered two versions of each model, with the first being based on the published model and the second recalibrated with a new estimate of the model intercept, but fixing the regression coefficients to be the same. The models were recalibrated on a subsample of one-third of the ESPOIR69 population and then tested on the remaining two-thirds of the population. Details of how the calibration sample was chosen were not provided.

The BeSt79 and ESPOIR69 models were also externally validated in the report of the de Punder models. 64 Heimans et al. 84 externally validated the BeSt model; however, the only version of the c-statistic that was reported was based on a different definition of the outcome (i.e. a SHS of > 0.5 rather than SHS of > 5) than that was used to derive the model and to assess performance in the other external validation studies. Hambardzumyan et al. 83 and Markusse et al. 85 externally validated MBDA.

The application of the BeSt model79 was considered by Saevarsdottir et al. 73 using the SWEFOT data set. However, the presented results did not provide sufficient details to be considered to be informative to the current review, as no summary statistic of overall performance (e.g. the c-statistic) was provided.

Results of the evidence synthesis

In order to provide a comprehensive overview of the available evidence, a meta-analysis was considered for risk models that were externally validated in more than one study. For predicting RRP (SHS of > 5 in 1 year), three clinical prediction models (i.e. BeSt79, ASPIRE CRP78 and ASPIRE ESR78) were each externally validated in two studies (i.e. De Cock et al. 80 and Granger et al. 82) that provided data in a suitable format. A meta-analysis was conducted using the c-statistic as a measure of discrimination. Although it would also be possible to synthesise other performance measures (e.g. the O : E ratio), other outcomes were not widely reported.

The predictive performance results are presented in Table 11 for all external validations that considered RRP (SHS of > 5 in 1 year) as an outcome, along with the results of the FE and RE meta-analysis for the models that were assessed in more than one external validation study.

Of the models that were externally validated in more than one population, the BeSt model79 shows the highest overall predictive performance (overall c-statistic 0.72, 95% CI 0.20 to 0.96), followed by ASPIRE ESR78 and then ASPIRE CRP. 78 However, there is considerable uncertainty in the pooled estimates. For all three clinical prediction models, the 95% CI of the pooled estimate from the RE model contains 0.5, indicating that there is no confidence that the c-statistic is better than would be expected by chance. The 95% prediction intervals are not presented for the RE model as a result of the limited number of studies contributing to the analyses (therefore the prediction intervals returned by the model are the same as the CIs). The highest predictive performance in an individual study assessing the performance of models for the prediction of RRP at 1 year was shown by MBDA, with a c-statistic of 0.77 (95% CI 0.64 to 0.90), followed by SONORA with a c-statistic of 0.76 (95% CI 0.68 to 0.83); however, neither of these models were externally validated in more than one population (using the same outcome definition and presenting a suitable performance measure).

Consideration of results according to baseline disease severity

Given the limited reporting of the DAS28 at baseline in the included model development and external validation studies (see Table 3), it was not possible to draw conclusive inferences regarding the development or performance of clinical prediction models in populations with moderate versus severe disease activity based on the DAS28 at baseline.

Discussion

Twenty-two model development studies and one combined model development/external validation study describing 39 clinical prediction models for the assessment of prognosis in early RA patients were identified across a range of major outcomes (radiographic joint damage, HAQ score and DAS28). The large majority of these model development studies (n = 16) tested apparent performance only (TRIPOD statement type 1a). A total of six external validation studies (including the combined model development/external validation study) were identified that tested the performance of a total of eight previously developed clinical prediction models. All included external validation studies focused on radiographic joint damage outcomes.

Variation was observed in the methods used to develop the included clinical prediction models, such as those used to select or reject candidate predictors from the final model (e.g. informed by univariable analysis, statistical selection or clinical judgement) and handling of continuous predictors (e.g. division into categories). Some studies presented a ‘matrix model’, and so continuous variables were categorised to allow this presentation format, even though the categorisation of continuous covariates is generally not recommended for prognostic model development. 35,36 Model development studies also generally failed to assess interactions between predictors, including interactions with the treatment group, and so did not generate truly treatment-specific models. This is considered in further detail in Chapter 5.

There was also inconsistency in the measures of predictive performance reported from internal validation. The c-statistic was reported for 8 of the 23 model development studies, although sensitivity and specificity (eight studies), accuracy (seven studies) and PPV and/or NPV (12 studies) were also commonly reported. However, even if consistent approaches had been used for internal validation, comparing the performance of clinical prediction models that have been developed and internally validated in different populations would still be limited, as good discriminative ability in the population used to develop the model would be expected. External validation is required to provide an objective comparison.

Of the eight studies that reported predictive performance in internal validation using the c-statistic, the results were variable. The c-statistics for radiographic progression outcomes ranged between 0.63 (i.e. Degboé et al. ,66 predicting a ΔSHS of ≥ 5 at 1 year) and 0.87 (i.e. Houseman et al. ,72 predicting a ΔSHS of ≥ 10.5 at 8.2 years). The Houseman72 models were notable in that they were among the few models that included MMP-3 in their final selection of predictors (MBDA also includes MMP-3). Unfortunately, the Houseman models72 were assessed using apparent performance only and were not externally validated, and so it is not possible to verify the strong apparent predictive performance suggested in the study report. Furthermore, the Houseman model72 included anti-CCP measured at the 8.2-year follow-up visit, which would bias the performance results relative to models that use only baseline information. Two studies predicting the HAQ score reported c-statistics [0.78 (i.e. Dirven et al. ,67 HAQ score of ≥ 1 at 3 months) to 0.82 (i.e. Bansback et al. ,58 HAQ score of ≥ 1.5 at 5 years)], which also showed promising discrimination, albeit derived from internal validation.

Although the 39 included clinical prediction models present predictive performance results based on internal validation, 31 of these have not been externally validated. This could be largely explained by the fact that 21 of the clinical prediction models were not published in a useable format. Of the eight clinical prediction models that were externally validated, the majority were presented as risk matrices in the published report,67,69,78,79 whereas two models were presented using the full model regression coefficients. 61,76 The SWEFOT73 models were not presented in a useable format in the model development publication; however, the main SWEFOT model was considered in the De Cock et al. 80 external validation based on the results that are available in conference proceedings. 73 The SONORA model was externally validated by Granger et al. ,82 despite being available only in a non-useable format in abstracts.

For models that have been externally validated, predictive performance was observed to vary widely. The highest predictive performance of a clinical prediction model for the prediction of radiographic progression (ΔSHS of ≥ 5 at 1 year) in an individual external validation study was shown by MBDA, which was externally validated by Markusse et al. 85 (c-statistic 0.77, 95% CI 0.64 to 0.90), followed by SONORA, which was externally validated by Granger et al. 82 (c-statistic 0.76, 95% CI 0.68 to 0.83). However, neither of these models was externally validated for the prediction of radiographic progression in more than one population. The lowest predictive performance of a clinical prediction model for the prediction of radiographic progression (ΔSHS of ≥ 5 at 1 year) in an individual external validation study was shown by Syversen et al. ,76 which was externally validated by De Cock et al. 80 (c-statistic = 0.34) followed by ESPOIR (c-statistic = 0.35). For both of these models, the external validation results suggest a performance that is poorer than would be expected by chance (c-statistic < 0.5). However, the performance of ESPOIR was much more promising when externally validated by de Punder et al. ,64 in which c-statistics were reported to range from 0.76 to 0.79 using a different outcome definition (≥ 5 Ratingen points at 3 years).

Three clinical prediction models (i.e. ASPIRE CRP, ASPIRE ESR and BeSt) were externally validated in more than one population using the same outcome definition (ΔSHS of ≥ 5 at 1 year). The results of the RE meta-analysis implied that the most favourable performance across external validations was observed using the BeSt model (c-statistic 0.72, 95% CI 0.20 to 0.96), followed by ASPIRE ESR (c-statistic 0.62, 95% CI 0.44 to 0.78) and ASPIRE CRP (c-statistic 0.55, 95% CI 0.13 to 0.91). However, the favourable performance of BeSt was largely informed by the high c-statistic observed in the Granger et al. external validation,82 and the BeSt model performed less favourably than either of the ASPIRE models in the De Cock et al. external validation. 80 There is considerable heterogeneity for all three models, with the wide CIs suggesting substantial uncertainty in the expected predictive performance in a new sample of patients. The 95% CIs of the pooled estimates contain 0.5 for all three clinical prediction models, indicating that there is limited confidence that the performance of the models is better than would be expected by chance.

Further external validations were identified in populations outside the scope of the assessment. These external validations were in populations with established RA. Vastesaeger et al. 78 tested the performance of their ASPIRE model78 in data from the ATTRACT RCT {median disease duration of 8.4 years [interquartile range (IQR) 4.3–14.7 years]}. Lillegraven et al. 95 also tested the performance of the ASPIRE CRP,78 BeSt79 and SWEFOT73 clinical prediction models in patients with established RA [median disease duration of 12 years (IQR 4–23 years)] and reported low predictive performance (c-statistics: ASPIRE, 0.59; BeSt, 0.65; and SWEFOT, 0.57).

No clinical prediction model performed consistently better than any other, and the current evidence therefore does not support the recommendation of one clinical prediction model over another. The inconsistent results generated by the clinical prediction models on external validation indicates that there is heterogeneity in the populations in which the models are being tested that is not explained by the currently proposed models. However, in this assessment, it was not possible to explore the reasons for the observed heterogeneity, because of the small number of external validations.

The external validation of a clinical prediction model may not be straightforward if the treatment regimens used in the external validation population differ from those used in the original model development population. The authors of the external validation studies included in this review aimed to select the published clinical prediction matrices that provided the closest approximation to the treatment strategy in their external validation population. The issue of treatment practices changing over time, with the resulting differences in treatment between model development and external validation populations, was acknowledged by Steyerberg et al. 23 in their contribution to the PROGRESS series of publications. However, these authors also noted that it was important to consider whether or not existing models could be updated and their predictive performance improved by, for example, recalibration or the addition of new predictor variables. Moons et al. 31 also advocated the value of updating existing prediction models in order to improve their predictive performance in new settings or populations. Recalibration was considered only in the external validation by Granger et al. 82 However, there was evidence that recalibration may not have resulted in a better model fit.

In order to assess the comparative performance of the competing models more thoroughly, further external validations would be required. These should be conducted in clinically appropriate populations with early RA, previously untreated and with sufficient variation (or case mix) in the population to ensure that the results are generalisable to the target population.

Although further external validation would help to clarify whether or not selected models are likely to perform better than others, as previously discussed, there were limitations identified in the methods used to develop the clinical prediction models in many of the development studies. Limitations included the absence of potentially important candidate predictors, inconsistent selection procedures, incorrect handling of continuous predictors and failure to test for interactions between predictors. It is therefore likely that the most clinically useful prediction model would contain predictors from across more than one of the reviewed clinical prediction models and/or consider alternative handling of key predictive variables. Access to IPD would allow harmonisation (both in terms of model development and validation) beyond that which is possible in the current assessment, which considers only aggregate-level data. 96 Guidance on IPD meta-analyses of clinical prediction modelling studies, including the advantages and challenges of undertaking such projects, is given by Debray et al. 97 Methods for assessing clinical prediction models using IPD from multiple studies are also described by Pennells et al. 98 Although the provided reference focuses on the prediction of cardiovascular disease outcomes using time-to-event data, the key principles are relevant for the current review.

Therefore, despite the availability of a range of clinical prediction models, uncertainty still remains over the most appropriate clinical prediction model(s) for use in clinical practice. Future research efforts should focus on consolidating understanding from existing clinical prediction models and external validation studies, and ensuring that recommended practice for model development and reporting is adhered to.

The meta-analysis was limited by the small number of external validation studies available for analysis. The synthesised estimates are indicative of performance in the observed studies, but cannot be used to provide a definitive conclusion about the performance in future studies or to explore the reasons for the heterogeneity between studies. A bivariate meta-analysis of calibration and discrimination has also been proposed and could potentially be used to increase the precision of summary estimates and avoid the exclusion of studies for which relevant estimates are missing. 99 However, this was not considered in this assessment because of the other described limitations of the evidence base.

Review 1 conclusions

Despite the availability of a range of clinical prediction models, uncertainty still remains over the most appropriate clinical prediction model(s) for use in clinical practice. In order to assess the comparative performance of the competing models more thoroughly, further external validations would be required. However, limitations were observed in the methods used to develop the clinical prediction models. It is likely that the most clinically useful prediction model would contain predictors from across more than one of the reviewed clinical prediction models and/or consider alternative handling of key predictive variables.

Quantity of research available

Only one of the clinical prediction models involving multiple treatments that were reported in review 1 investigated potential interactions between treatments and predictor variables. 79

The study selection process for review 2 is depicted in Figure 7. A total of 12 studies were included,78,100–111 two of which used the same data set. 78,100

FIGURE 7.

Review 2 study selection represented as a PRISMA flow diagram. 55

It was stated in the final protocol that any potential overlap would be highlighted between this report and the National Institute for Health Research (NIHR) HTA Technology Assessment Review (TAR) report number 14/16/01. One study included in review 2 (i.e. Taylor et al. 101) was also included in the HTA TAR report number 14/16/01. This study examined the effect of baseline synovial vascularity assessed by PDUS on radiographic progression at 54 weeks by MTX versus MTX and IFX. Other studies in the NIHR HTA TAR report number 14/16/01 were screened for relevance to the current review, but were excluded for not examining an early RA population and/or not examining the interaction between baseline patient and/or disease characteristics and treatments.

A summary of the 12 included studies that are used to assess the prediction of treatment response by patient and/or disease characteristics is presented in Table 12. All included studies were post hoc subgroup analyses of RCTs. 78,100–111 The geographical locations of the studies included the Netherlands,103 the USA,105,110 Finland,106 Italy,107 Germany108 and the UK,101,109 with three studies examining data from two multinational trials78,100,104 and one study not reporting location. 102 Sample sizes ranged from 24 to 1004 participants; most studies were funded by the pharmaceutical industry,78,100–105,107,110 two studies reported funding from local or national health-care agencies106,109 and one study did not report the funding source. 108

Overall quality of research available

All included studies were rated as being at a moderate risk of bias, according to the QUIPS assessment tool54 (Table 13). Generally, most studies were rated as being at a moderate risk of bias for study participation. This was mainly because most studies did not report the source of the target population and the methods used to recruit patients to the study, with many not reporting or partially reporting whether or not there was adequate participation in the study. Most studies reported the recruitment period, inclusion and exclusion criteria and baseline characteristics.

The majority of studies were also rated as being at a moderate risk of bias for study attrition. This was mainly because most studies did not report the proportion of the baseline sample that provided data for analysis or what attempts were made to collect information on participants who dropped out. Most studies did not report, or partially reported, the key characteristics for participants who dropped out or the comparative characteristics of those who dropped out and those who provided follow-up data. However, reasons for the loss of follow-up data were typically reported.

Most studies were also rated as being at a moderate risk of bias for prognostic factor measurement. This was mainly because of a lack of clarity or reporting surrounding the proportion of data on the prognostic factor that was available for analysis and the method used for dealing with missing data in most studies. The method and setting of prognostic factor measurement was reported by approximately half of the included studies, and most studies reported a definition and also valid and reliable measurement of the prognostic factor.

Most studies were rated as being at a low risk of bias for outcome measurement. This was mainly attributable to most of the studies reporting a clear definition of the outcome, valid and reliable measurement of the outcome and the method and setting of the outcome being the same for all participants.

Study confounding was also rated as being at a moderate risk of bias for most studies. This was mainly attributable to a lack of reporting or partial reporting among the majority of studies on whether or not important confounders were measured and the definition and valid and reliable measurement of these, and also the method and setting of confounding measurement and whether or not important confounders were accounted for in the study design and analysis.

Most studies were also rated as being at a moderate risk of bias for statistical analysis. This was mainly because of uncertainty over whether or not the presentation of the analytical strategy and model development strategy and the avoidance of selective reporting of results were adequate, because of a lack of a clear description in most studies.

Treatments, baseline variables and outcomes examined of the included studies

Table 14 summarises the treatments, patient and/or disease characteristics and outcomes examined. Numerous treatment comparisons were made, including:

• ETN versus MTX102

• sequential monotherapy with initial MTX versus step-up monotherapy with initial MTX versus MTX and SSZ and tapered prednisone versus MTX and IFX103

• ABT (subcutaneous administration) and MTX versus ABT (subcutaneous administration) versus MTX104

• MTX and ETN immediate therapy versus MTX and SSZ and HCQ immediate therapy versus MTX and ETN step-up therapy versus MTX and SSZ and HCQ step-up therapy105

• MTX and HCQ and SSZ in a treat-to-target (TTT) regime versus cDMARD sequential monotherapy in a TTT regime, starting with SSZ106

• ciclosporin A (CsA) versus cDMARDs107

• MTX versus gold sodium thiomalate (GSTM)108

• MTX versus MTX and CsA versus MTX and prednisolone versus MTX and CsA and prednisolone (in a factorial design)109

• MTX versus MTX and IFX (3 mg/kg or 6 mg/kg)78,100

• MTX and ABT (subcutaneous administration) versus MTX and ADA (subcutaneous administration),110 and

• MTX versus MTX and IFX. 101

Eligible baseline variables were ACPA status,104,106,109,110 smoking status,105 erosions,100,107 RF status,100,111 CRP level,100–102 ESR,100,108 SJC,100 BMI103 and vascularity of synovium detected using PDUS. 101

Eligible outcomes were erosions/radiographic progression,100–102,106–109 disease activity,105,109 physical function,109,110 remission103,110 and a DAS of > 2.4. 103 Four studies assessed outcomes at 6 months,105,108–110 11 studies assessed outcomes at 1 year,100–103,106–110 one study assessed outcomes at 18 months,109 four studies assessed outcomes at 2 years,105,106,109,110 and one study assessed outcomes at 3, 4 and 5 years. 106

Description of population characteristics

The key population characteristics are summarised in Table 15 (and additional population characteristics are summarised in Appendix 9, Table 34).

The majority of included studies used the 1987 revised ACR criteria for RA diagnosis. 100–103,105–107,109,110 One study did not report the diagnosis criteria104 and one study reported using the ACR case criteria. 108 Five studies defined early RA as being active RA or the presence of symptoms for a duration of < 2 years,103,104,106,109,110 four studies defined early RA as being active RA or the presence of symptoms for a duration of < 3 years100–102,105 and two studies used other definitions (active RA for ≥ 6 months but ≤ 4 years,107 or by radiographic definition, i.e. patients without advanced disease108); in all cases, baseline mean/median disease/symptom duration met the inclusion criterion for early RA (i.e. being within 2 years of the onset of symptoms) for this review. The duration of RA was reported in some studies as disease duration and in others as symptom duration. Disease duration ranged from a mean of 1.4 months107 to a mean of 17.9 months,101 and from a median of 1.0 month109 to a median of 24.0 months. 110 Symptom duration ranged from a mean of 6.0 months to a mean of 7.1 months,104 and from a median of 5.3 months to a median of 6.0 months. 103 Baseline DAS28, when reported, ranged from a mean of 5.3101,104 to a mean of 6.7,100 and one study reported DAS, with a baseline mean of 4.4,103 indicating severe RA in all populations for which the DAS28 or the DAS at baseline was reported. Three studies did not report the DAS28 or the DAS at baseline. 102,107,108 Participants were DMARD naive in two studies103,106 and MTX naive in three studies. 100,102,108 One study reported that participants were DMARD naive or treated with a maximum of one DMARD, which could be an antimalarial agent (30% of the sample) or auranofin (22% of the sample). 107 One study reported prior DMARD use in 14% of the sample,109 one study reported that participants had to be bDMARD naive with an inadequate response to MTX110 and one study reported that participants had to be receiving a stable dose of MTX. 101 One study reported that participants were bDMARD naive, with prior MTX use in 21% of the sample,105 and one study reported that participants were MTX naive or had received a low dose of MTX previously for ≤ 4 weeks with a 1-month washout period. 104

Results: synthesis of primary studies

Treatment prediction models presents the findings for clinical prediction models developed according to specific treatments. Prediction of treatment effect by anticitrullinated protein/peptide anti-body status at baseline: erosions presents information regarding patient and/or disease characteristics that are potential treatment effect modifiers.

Tables 16 and 17 provide the estimates of the within-treatment responses and interaction effects by baseline variable for continuous outcomes. Appendix 10 presents plots of the estimates of the within-treatment responses by baseline variable. The results are presented by study because the combination of outcomes and predictors generated no studies that shared any treatment in common, so that a formal synthesis was not possible.