Modelling disease progression in relapsing remitting onset multiple sclerosis using multilevel models applied to longitudinal data from two natural history cohorts and one treated cohort

Article history

The research reported in this issue of the journal was funded by the HTA programme as project number 10/55/01. The contractual start date was in August 2011. The draft report began editorial review in August 2015 and was accepted for publication in May 2016. The authors have been wholly responsible for all data collection, analysis and interpretation, and for writing up their work. The HTA editors and publisher have tried to ensure the accuracy of the authors’ report and would like to thank the reviewers for their constructive comments on the draft document. However, they do not accept liability for damages or losses arising from material published in this report.

Declared competing interests of authors

Kate Tilling has received non-financial support from the Multiple Sclerosis (MS) Trust during the conduct of the study, and had her expenses paid by the MS Trust to attend meetings of the UK MS risk-sharing scheme (RSS) scientific advisory group in order to outline the plan for these analyses. Michael Lawton has had his expenses paid by the MS Trust to attend meetings of the UK MS RSS scientific advisory group in order to outline the plan for these analyses, and also his travel and accommodation expenses for visiting Vancouver to analyse the British Columbia MS data set. Neil Robertson has received travel grants and honoraria from Biogen, Novartis, Serono, Sanofi, Genzyme and Bayer and holds grants for unrelated work from Genzyme and Novartis. Helen Tremlett is funded by the MS Society of Canada (Don Paty Career Development Award), Michael Smith Foundation for Health Research and is the Canada Research Chair for Neuroepidemiology and MS. She has received research support from the National MS Society, the Canadian Institutes of Health Research and the UK MS Trust; and speaker honoraria and/or travel expenses to attend conferences from the Consortium of MS Centres (2013), the National MS Society (2012, 2014), Bayer Pharmaceuticals (2010), Teva Pharmaceuticals (2011), European Committee for Treatment and Research in MS (2011, 2012, 2013 and 2014), UK MS Trust (2011), the Chesapeake Health Education Program, US Veterans Affairs (2012), Novartis Canada (2012), Biogen Idec (2014) and the American Academy of Neurology (2013, 2014 and 2015). Unless otherwise stated, all speaker honoraria are either donated to a MS charity or to an unrestricted grant for use by her research group. Yoav Ben-Shlomo has had his expenses paid by the MS Trust to attend meetings of the UK MS RSS Scientific Advisory Group in order to outline the plan for these analyses, and has a relative with MS who is currently on treatment for the disease. He is a member of the UK MS RSS Scientific Advisory Group.

Copyright statement

© Queen’s Printer and Controller of HMSO 2016. This work was produced by Tilling et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

Multiple sclerosis treatment

There is no cure for MS, but treatments are available and generally fall into four categories, with currently the first two categories being available only to those with RR onset MS:

1. drugs that aim to modify the disease process (DMTs)

2. corticosteroids to accelerate recovery from a relapse

3. symptomatic drug treatment, to help ease symptoms of MS

4. other non-pharmacological therapies and general support to minimise the impact of disability.

The DMTs are also known as ‘immunomodulatory agents’ and are currently licensed only for patients with relapsing-onset MS (RR or SPMS). In 2002, there were four licensed DMTs: two forms of interferon beta-1a [Avonex^® (Biogen Idec Ltd) and Rebif^® (Merck Serono Ltd), with high- and low-dose formulations available, 44 µg and 22 µg, respectively], one form of interferon beta-1b [Betaferon^® (Bayer Plc)] and glatiramer acetate [Copaxone^® (Teva UK Ltd)] (first generation of this drug). At that time, the UK’s National Institute for Health and Care Excellence appraised the evidence for cost-effectiveness of these DMTs and concluded that these drugs were not cost-effective over a 10- or 15-year period. 8 However, it also concluded that, because the relevant clinical trials were based on only short-term outcomes, there was not enough information to assess the longer-term implications of these DMTs. To try to address these concerns, the UK MS risk-sharing scheme (UK MS RSS) was established in 2002. 9,10 Under the scheme, the UK’s Department of Health agreed to pay for the DMTs (see UK multiple sclerosis risk-sharing scheme cohort), conditional on treated patients being included in a 10-year monitoring study aiming to assess progression of the disease. The monitoring study aimed to compare progression of MS within this treated group of patients with progression in an untreated cohort. Initially, a cohort of untreated patients from London, Ontario, Canada, were selected [a subset of 314 patients judged to have fulfilled the Association of British Neurologists (ABN) criteria, from the London, Ontario, cohort of 1043 patients recruited between 1972 and 1984]. The first report from the UK MS RSS, based on the 2-year follow-up data, was unable to find evidence for the effectiveness of DMTs, but the authors were hesitant in drawing any firm conclusions because of methodological limitations. 9 The main concerns raised related to the use of the London, Ontario, cohort, which only permitted patients to have stable or deteriorated disability scores; this is not what is observed in actual clinical practice, in which scores naturally fluctuate and can also improve. 11 This use of the London, Ontario, cohort forced a complex algorithm to be used in the analysis, creating an artificial ‘no-improvement’ rule. This caused multiple problems, resulting in uncertainty in the findings, with the sensitivity analysis revealing qualitatively different results from the main analysis (suggesting that the DMTs might actually have a beneficial effect). In addition, there were concerns about missing data and a simple best-case/worst-case approach was taken so that either disability scores were assumed to be unchanged at follow-up or a simple linear extrapolation was done. The authors highlighted these problems9 and suggested that future analyses should seek a different, more powerful cohort in which raw (observed) rather than smoothed and adjusted disability scores could be used. They also highlighted the possibility of exploring other analytical models to complement the Markov model that was used for that analysis. In addition, they noted that, given potential secular trends in the natural history of MS (e.g. an improvement in prognosis over calendar time, as a result of other treatments for MS being made available or better overall management of the disease), a more contemporary cohort would be preferable. However, there is little evidence of change in disability progression between 1975 and 1995,12 indicating that a historical cohort could potentially be used.

Outcomes in multiple sclerosis clinical trials and observational studies

Commonly used outcomes in MS studies include measures of relapse rate, magnetic resonance imaging metrics and disability measures, such as the Expanded Disability Status Scale (EDSS). 13 Relapses are typically defined as periods of worsening of neurological symptoms in terms of severity or duration (e.g. present for at least 24 hours), in the absence of fever or infection. Relapses can be subject to recall bias and have been shown to be sensitive to the placebo effect. 13 Disability is usually measured by a disease-specific scale, with the most commonly used being the EDSS,14 although others have been developed, such as the MS Functional Composite. 15 The EDSS score can range from 0 (normal neurological exam) to 10 (death due to MS), and the scale is shown in Table 1. The scale is ordinal, and was not designed for use as a cardinal scale; thus, changes of 1 point at lower levels on the scale do not equate to 1-point changes at higher scores. The MS Functional Composite15 was developed to overcome some of these problems, being a metric (rather than ordinal) scale, and includes a measure of cognitive function that is not directly captured by the EDSS. 16 However, it is more complex to administer than the EDSS, may not be any more sensitive to change, and is highly affected by visual/speech deficits and learning effects. 16 Therefore, the EDSS continues to be the tool of choice for measuring outcome in many MS clinical trials and observational studies, and was used in the original pivotal clinical trials of the DMTs; it is also the only outcome-measuring tool available for the UK MS RSS.

Methods in multiple sclerosis observational studies assessing disability progression

Studies of progression in MS tend to use survival analysis techniques to model outcomes over time, for example, examining time to a sustained and a confirmed EDSS score of 6 points (often chosen as it represents a ‘hard’ outcome and a turning point in the MS disease course, indicating that the individual can no longer walk without an aid, such as a cane or crutch) or age at which an EDSS score of 6 points is reached. 17 Such analyses have several disadvantages:

1. Outcome data are usually available only at discrete intervals (e.g. annually) and thus the exact time to the outcome may not be known.

2. The analysis cannot take into account further deterioration (or improvement) once the outcome is achieved.

3. The analysis cannot take into account the actual disability either before or after the outcome is achieved: a person with annual EDSS scores of 4, 4, 4 and 6 points would contribute the same information to the analysis as someone with annual EDSS scores of 1, 3, 5, 7 and 9 points, and yet clearly the progression of disease is very different in these two cases.

4. The method for dealing with missing data is unclear: for example, given a patient with annual scores of 1, 3, missing and 7 points or 6, 6, 7, 8 and 9 points (i.e. who had already reached the outcome of interest by their first assessment), in both cases the exact time to the outcome would be uncertain.

Alternatives have been suggested, including discrete Markov models18 and the construction of percentile charts. 19,20 Discrete Markov models can relate the probabilities of transitions between disease states to covariates, including baseline characteristics and time. Such models have been applied to the UK MS RSS cohort,9 but, at least in the 2-year analysis, which used the London, Ontario, comparator cohort, the models showed such sensitivity to missing data and model assumptions that firm conclusions could not be drawn. 9 One disadvantage of these discrete models is that the repeated measures need to be at similar (discrete) time intervals on all patients for common transition probabilities to be estimable. There are more complex ‘continuous’ Markov models, which can overcome this disadvantage. Percentile charts have been constructed for the development of EDSS score over time since diagnosis. 19,20 However, these were essentially cross-sectional in nature, being either empirical or smooth functions of the percentiles of EDSS score at each year post onset or post diagnosis, and thus do not always reflect how a given individual changes over time. For this method also, all patients are required to be assessed at the same, discrete time intervals. In addition, the relationship between these centiles and patient characteristics cannot easily be modelled.

Predictors of progression in multiple sclerosis

There is wide variation in disease trajectory between MS patients, with limited consensus over the factors affecting progression. 17 A systematic review of studies of MS disability progression (carried out in 2004–5) identified 27 published studies related to RRMS. 21 The prognostic factors identified were male sex (evidence mixed, but a suggestion of poorer prognosis for males), age at onset (all but one study related poorer prognosis to older age at onset), type of symptoms at onset (little evidence for any of the symptoms examined except for bowel/bladder involvement, which was associated with poorer prognosis), incomplete recovery from the first attack (consistently associated with poorer prognosis) and shorter interval between first and second attacks (consistently associated with poorer prognosis). 21 An examination of time to SPMS in the BCMS cohort also identified poorer prognosis for males and those with an older age at onset, and found no evidence of associations between onset symptoms and progression. 22

Summary

In the case of a disease whose median duration may be as long as 30–40 years from onset,23,24 with outcomes that vary greatly and are largely unpredictable, ranging from so-called ‘benign’ to ‘malignant’ disease, it is of considerable importance to be able to better predict long-term outcomes. This could potentially facilitate the prognosis given to patients as well as informing therapeutic and management choices made by clinicians and patients. This is particularly relevant when recent developments in therapeutics may allow significant impact on natural history but with costs in terms of associated adverse events. Patients and their carers increasingly request access to relevant prognostic information to help plan both professional and family life decisions, although this information is often not readily available.

Objectives

1. To develop and apply multilevel models to repeated measures (longitudinal) data on disease progression in patients with relapsing-onset MS (RR and secondary progressive) in the UoWMS and BCMS cohorts. This will allow us to determine average progression trajectory, individual deviations from the average and accuracy of prediction of individual trajectories.

2. To examine the impact of informative censoring using joint modelling of the EDSS trajectory and time to dropout and, if necessary, use multiple imputation methods as part of a sensitivity analysis.

3. To apply the multilevel model developed above to data from the UK MS RSS to estimate average progression trajectory, individual deviations from the average, difference between individual progression trajectories in the UK MS RSS and progression predicted by the BCMS model; and, thus, estimate the effect of the MS drugs on disability progression.

4. To apply patient-derived utilities to the differences in EDSS score as a result of DMTs and, thus, obtain an estimate of the value put on EDSS score-related progression associated with DMT by patients.

Data sets

In this study, we have used data from two cohorts of patients with relapsing-onset MS to develop and validate models for the prediction of EDSS score. These cohorts are the UoWMS cohort and the BCMS cohort. We used data from a third cohort, the UK MS RSS, to examine the ability of the model for EDSS score to identify changes in progression in a treated (rather than natural history) cohort.

Data from the BCMS cohort can be analysed only in Canada. Therefore, we used the UoWMS cohort for initial exploratory analysis, before travelling to Canada to analyse the BCMS cohort and develop a natural history model independently using this large cohort. This BCMS model was validated on the UoWMS cohort (which is smaller). The BCMS model was then used as an untreated comparison arm for the UK MS RSS.

Patient eligibility and selection of the analysis data sets are presented below under each cohort heading, and in Eligibility.

Modelling the natural history of multiple sclerosis in the Welsh and British Columbia cohorts

The EDSS scores on the same person measured over time are likely to be correlated. Therefore, we used multilevel models with two levels: measurement occasion (level 1) clustered within person (level 2). 28 These longitudinal methods have several advantages over approaches previously used: they take into account the correlation of observations within an individual; they use all available data (i.e. all available EDSS scores); time can be modelled continuously and thus all patients do not need to have measures at rigidly fixed intervals (which mimics well-patient visits in clinical practice) and important covariates can be included, such as age at disease onset and disease duration. In addition, patients with missing data should not bias the analysis, provided that data are missing at random. Furthermore, the impact of any data which are not missing at random (e.g. informative censoring) can be examined using extensions to this method (see objective 2). Although the usual multilevel models treat the EDSS score as a continuous variable, extensions to these methods could also consider transformations of the EDSS score, or model it as an ordinal outcome, if there was evidence of poor model fit.

We modelled the EDSS scores of individuals using repeated measure multilevel models. Our model had two levels: the level 2 unit was the individual and the level 1 unit was the observation. A simple multilevel repeated measure model is a linear random intercept and random slope model. This ‘random intercept, random slope’ model assumes that there is a linear change in the outcome (y) for the population, and the population average change is described by Equation 1. However, the model does not assume that every individual will follow that pattern of change exactly. Instead, each individual is allowed their own intercept (the ‘random intercept’), for which the individual intercepts are normally distributed around the population average. Similarly, each individual is allowed their own slope (the ‘random slope’), for which the individual slopes are normally distributed about the population average slope. In addition, the individual intercepts and slopes are allowed to be correlated, for example if people who have a lower intercept then progress faster, the model would allow a negative correlation between individual intercept and individual slope.

The random intercept, random slope model is detailed below:

where y_ij is the EDSS score for the ith individual on the jth time point and t_ij is the time variable for the ith individual’s jth time point.

where N₂(0, D) is a multivariate normal distribution with degree 2, mean 0 and covariance matrix D, and N(0, σ²) is a univariate normal distribution, with mean 0 and variance σ².

The u_ki (the ‘random intercept’ and ‘random slope’) are often referred to as the level 2 (individual-level) random effects and the e_ij as the level 1 (observation-level) random effects.

For all the following multilevel repeated measure models the following assumptions were made:

1. Individual-level random effects are correlated with each other (with freely estimated covariances), as are observation-level random effects.

2. The observation-level and individual-level random effects are uncorrelated with each other.

We explored different time axes (e.g. age, time since onset, time since ABN eligibility, etc.), using Akaike information criterion (AIC)29 and the variation of the differences between fitted and observed EDSS scores to identify the best-fitting model.

Individuals were (right) censored when a DMT was initiated to avoid a potential treatment-effect bias, although we recognised this was at the cost of introducing a potential indication bias [whereby DMT initiation may be triggered by changes (i.e. worsening) in disability (EDSS score)]. 30 We tested the model assumption that the patient-level and individual-level residuals are normally distributed, and if necessary would allow for non-normal random effects. 31

Modelling the impact of disease-modifying therapy on disease progression in the UK Multiple Sclerosis Risk-Sharing Scheme

Statistical software

All computations were carried out using Stata software (StataCorp LP, College Station, TX, USA), and all multilevel models estimated using the runmlwin command.

The natural history data sets

Table 3 shows the patient characteristics for the Welsh (UoWMS) and British Columbia (BCMS) natural history cohorts.

The BCMS cohort contained information on more than twice as many individuals as the UoWMS cohort, and had a larger number of observations (EDSS scores) per person. However, the patient characteristics were remarkably similar for sex, age at ABN eligibility, age at MS symptoms onset, disease duration at ABN eligibility and the proportion of patients with SPMS at ABN eligibility. The age at onset was 2 years lower in the BCMS cohort than in the UoWMS cohort, although the longer disease duration at ABN eligibility meant that the average age at ABN eligibility in the BCMS cohort was only 1.3 years lower than in the UoWMS cohort. A higher proportion of the BCMS cohort reached secondary-progressive disease during follow-up and this is probably related to the longer duration of follow-up in this cohort. There were slightly more relapses in the 2 years prior to, and a moderately higher EDSS score at, ABN eligibility in the UoWMS cohort than in the BCMS cohort.

Model development using the University of Wales Multiple Sclerosis cohort

Complex measurement error

In order to assess the distribution of the measurement error, we plotted the level 1 residuals from the best-fitting model (time since onset, including time and the log of time since onset) against time since onset (Figure 1).

FIGURE 1.

Level 1 residuals plotted over time since MS symptom onset. Source: adapted under the Open Government Licence from Lawton et al. 25

Figure 1 shows some evidence that the variance of the level 1 residuals in the chosen model are decreasing over time.

Fractional polynomials of degree 2 in the level 1 random effects did not significantly improve the fit of the model compared with fractional polynomials of degree 1 (p = 0.08). Adding a linear time term to the level 1 random effects resulted in a clear improvement in the model fit (p < 0.001). This was not the best level 1 random effect of time, as including the square root of time gave a better fit than the model with linear time. However, the difference between the model with linear time and square root of time as a level 1 random effect was not large (difference in AIC = 1.1 units). The model with linear time as a level 1 random effect was chosen for ease of convergence of the model and also for parsimony of not including different powers of time in the level 1 random effects compared with those in level 2 random effects and fixed effects.

We adjusted the level 1 time variance term to be equal to zero, as its 95% CI included zero. Hence we allowed only the covariance term between the constant and level 1 time term to be freely estimated. This constraint did not affect the model fit considerably (difference in AIC = 5.39).

Model for natural history of multiple sclerosis using the University of Wales Multiple Sclerosis cohort

The best-fitting model to the UoWMS data included time since onset and the log of time since onset, allowed these to vary randomly between individuals, and also allowed measurement error to vary with time. The model included only observations taken at least 3 months after a relapse. The coefficients for this model are shown in Table 10.

Assessing model fit

The main assumption of the multilevel models was that the level 1 and level 2 residuals were normally distributed. This was tested using quantile–quantile (Q–Q) plots of the level 1 and level 2 residuals for our chosen model. The Q–Q plots are shown in Figure 2 and were close to normality.

FIGURE 2.

The Q–Q plots for all residuals from the final UoWMS model. (a) Level 1 (constant); (b) level 1 (time); (c) level 2 (constant); (d) level 2 (time); and (e) level 2 (log-time). Source: adapted under the Open Government Licence from Lawton et al. 25

Model development with the British Columbia Multiple Sclerosis cohort

Model development using the BCMS data gave very similar results. First, time since onset was found to be a better time axis than age. Second, linear and log-time for the individual trajectories gave a good fit to the data and fractional polynomials of degree 3 showed little improvement. The AICs for the top three models are shown in Appendix 1 (see Table 20). Third, adding complex measurement error (i.e. a linear time term to the observation-level random effects) gave a model with better fit, and fractional polynomials of degree 2 for the measurement error term showed little improvement over this.

The only difference from the UoWMS model was that minimal changes were observed in the model when the 1-, 3- and 6-month post-relapse windows were explored. Hence we choose to use the BCMS model, which included only observations taken at least 1 month after a relapse.

The two models in Table 11 showed a significant difference only in the level 1 constant random effect and the constant fixed effect. A difference in the constant fixed effect was expected, given that, from the patients’ characteristics (see Table 3), the first ‘ABN eligible’ EDSS score was, on average, higher in the UoWMS cohort. The difference in the level 1 constant random effect shows that there was less variation around the initial EDSS score for the UoWMS cohort than for the BCMS cohort. This could be partly because of the lower estimated EDSS score at onset in the BCMS cohort, if measurement error is higher when assessing lower EDSS scores. Model fit was assessed by examining the distributions of the residuals. Figure 8 (see Appendix 1) indicates that these were reasonably normally distributed.

Final model relating Expanded Disability Status Scale to time since Association of British Neurologists eligibility, and including age at onset

For comparison with the Markov modelling approach to the UK MS RSS cohort,46 we then fitted a model including only one covariate: the binary indicator of whether or not the age at onset was above or below the median age of 27.9 years (Table 13). There was not a strong association between the binary indicator for age at onset and EDSS trajectory, with some indication that those younger at onset had higher EDSS scores [by 0.14 (UoWMS)/0.096 (BCMS) points].

Validation of the British Columbia Multiple Sclerosis model relating Expanded Disability Status Scale to time since Association of British Neurologists eligibility, using the University of Wales multiple sclerosis cohort

We examined portability of the model between populations by using the BCMS model to predict EDSS score for individuals in the UoWMS cohort (when the UoWMS cohort included only observations taken at least 3 months after a relapse). Table 14 is based on 253 individuals who have at least two EDSS scores, and shows the differences between observed EDSS score in the UoWMS cohort and EDSS score predicted by the BCMS model (conditional on the first observed EDSS score, i.e. using the first observed EDSS score as the baseline). The average EDSS score over time and the average predicted by the BCMS model are shown in Figure 3. There is reasonable evidence that the BCMS model can be used to predict EDSS scores in the UoWMS cohort. There does appear to be slight evidence of a pattern, in that differences early in follow-up are positive (the model overpredicts) and those later in follow-up tend to be negative (the model underpredicts). However, the average differences are small in magnitude, compared with the spread of the differences, the measurement error and variation of EDSS score.

TABLE 14 - Observed EDSS score for the UoWMS cohort, and EDSS score predicted by the BCMS model relating EDSS score to time since ABN eligibility and including age at onset as a binary covariate

SD, standard deviation.

Time since eligibility (years)	EDSS score
	Observed in the UoWMS			Predicted using the BCMS model		Predicted – UoWMS observed
	Number of observations (number of individuals)	Mean	SD	Mean	SD	Mean	SD
0–0.5	88 (72)	3.76	1.71	3.77	1.63	0.01	0.99
0.5–1.5	292 (153)	4.18	2.02	4.26	1.81	0.08	1.14
1.5–2.5	265 (118)	4.82	1.78	4.87	1.59	0.05	1.14
2.5–3.5	202 (101)	5.27	1.61	5.19	1.62	–0.08	1.32
3.5–4.5	182 (92)	5.61	1.77	5.58	1.67	–0.03	1.53
4.5–5.5	142 (72)	5.74	1.73	5.86	1.42	0.12	1.46
5.5–6.5	138 (54)	6.07	1.77	5.85	1.35	–0.22	1.57
6.5–7.5	75 (39)	6.09	1.79	5.98	1.38	–0.11	1.68
7.5–8.5	52 (30)	6.10	1.92	6.27	1.38	0.17	1.69
8.5–9.5	29 (14)	7.33	1.22	7.22	1.67	–0.10	1.48
9.5–10.5	30 (10)	7.48	1.16	7.45	1.80	–0.03	1.49
Total	1495 (253)	5.20	1.97	5.20	1.80	–0.001	1.35

FIGURE 3.

Observed and predicted EDSS score in the UoWMS cohort using the model relating EDSS score to time since ABN eligibility and including age at onset as a binary covariate.

Validation of the British Columbia Multiple Sclerosis model to predict utility in the University of Wales Multiple Sclerosis cohort

Table 15 is again based on 253 individuals who have at least two EDSS scores, and shows the ability of the BCMS model to predict utility scores related to disease progression in the UoWMS population. Here we predicted EDSS score using our model (conditional on first observed EDSS score) and then converted both predicted and observed EDSS score to 1 – primary utility. The overall difference between predicted and observed (predicted – observed) utility related to disease progression for all non-baseline observations between eligibility and a time of 10.5 years (n = 1495) has a mean (standard deviation) of –0.00 (0.16) and hence a 95% CI of –0.01 to 0.00 for the mean difference. The RMSE of the difference is 0.16. The average observed and predicted utility related to disease progression over time are shown in Figure 4. Again, it appears that the BCMS model can be used to predict utility related to disease progression in the UoWMS cohort. There does appear to be slight evidence of a pattern, in that differences early in follow-up are positive (the model overpredicts) and those later in follow-up are negative (the model underpredicts). However, the average differences are small in magnitude compared with the spread of the differences (and the potential measurement error and variation of the EDSS score).

TABLE 15 - Observed utilities for the UoWMS cohort, and utility predicted by the BCMS model relating EDSS score to time since ABN eligibility and including age at onset as a binary covariate

SD, standard deviation.

Time since eligibility (years)	UoWMS observed 1 – utility			1 – utility predicted using the BCMS model		Predicted – observed
	Number of observations (number of individuals)	Mean	SD	Mean	SD	Mean	SD
0–0.5	88 (72)	0.420	0.10	0.429	0.12	0.008	0.09
0.5–1.5	292 (153)	0.448	0.15	0.458	0.14	0.010	0.12
1.5–2.5	265 (118)	0.490	0.13	0.493	0.11	0.003	0.10
2.5–3.5	202 (101)	0.527	0.13	0.535	0.15	0.008	0.13
3.5–4.5	182 (92)	0.567	0.19	0.576	0.18	0.009	0.16
4.5–5.5	142 (72)	0.579	0.18	0.590	0.17	0.012	0.17
5.5–6.5	138 (54)	0.656	0.24	0.582	0.17	–0.074	0.23
6.5–7.5	75 (39)	0.648	0.22	0.607	0.17	–0.041	0.23
7.5–8.5	52 (30)	0.639	0.22	0.649	0.22	0.010	0.23
8.5–9.5	29 (14)	0.843	0.21	0.754	0.24	–0.088	0.23
9.5–10.5	30 (10)	0.830	0.22	0.790	0.29	–0.041	0.25
Total	1495 (253)	0.543	0.19	0.538	0.18	–0.005	0.16

FIGURE 4.

Observed and predicted 1 – utility in the UoWMS cohort using the BCMS model relating EDSS score to time since ABN eligibility and including age at onset as a binary covariate.

Summary of natural history modelling

We developed models using two independent data sets, and found evidence of a non-linear pattern of progression, and also that time since disease milestones (onset, ABN eligibility) provided a better fit to the data than modelling EDSS score progression as a function of age.

As shown in Figures 5 and 6, the predictions (conditional on the first observed EDSS scores) fit the actual data well, with time since eligibility providing a closer fit to the data than time since onset. The model coefficients from the BCMS cohort (based on a larger sample size with longer follow-up) are a better predictor of the UoWMS data than the coefficients derived from the UoWMS cohort itself. When using the BCMS model coefficients to predict observations in the UoWMS cohort there is a slight underestimate to begin with, owing to differences in EDSS score at ABN eligibility between the two data sets. However, at 4 years post eligibility, there is no material difference between observed and expected EDSS score. These figures support for the idea of using a model developed in British Columbia as a non-randomised comparison group for natural history of MS in the UK.

FIGURE 5.

Fit of the UoWMS models (not including covariates) to the UoWMS cohort EDSS score data.

FIGURE 6.

Fit of the BCMS models (not including covariates) to the UoWMS cohort EDSS score data.

Age onset binary models for time since eligibility

Table 18 compares the models with age at onset as a binary variable in the BCMS, UoWMS and UK MS RSS cohorts. The estimated average EDSS score is lowest in the BCMS cohort and highest in the UoWMS cohort, and the association between age at onset and average EDSS score is similar across cohorts.

Figure 7 shows the predicted average EDSS score from ABN eligibility for the BCMS and UoWMS cohorts, and from baseline for the UK MS RSS cohort. Increase in the EDSS score seems to be slightly slower in the UK MS RSS cohort than in the two natural history cohorts. In particular, the UoWMS and UK MS RSS cohorts have similar baseline EDSS values, but quite different trajectories over time. This could be for many reasons, including chance, the different settings and calendar time of the cohorts, confounder distribution, but also possibly because of an effect of DMT treatment on EDSS score progression.

FIGURE 7.

Predicted average EDSS score across time since ABN eligibility for the models from the UoWMS, BCMS and UK MS RSS cohorts.

Using the natural history model to predict progression in UK multiple sclerosis risk-sharing scheme

Table 19 shows the observed progression (EDSS score at up to 6 years minus EDSS score at baseline) for the UK MS RSS cohort, together with progression predicted under the natural history model. Observed progression in the treated UK MS RSS cohort was 0.6 points lower than predicted by the BCMS model (95% CI 0.5 to 0.6 points lower). This translated to a difference in the utility related to disease progression of –0.04 (95% CI –0.05 to –0.04). Thus, the increase in disability (as measured by the EDSS) was slightly less in the DMT-treated UK MS RSS cohort than would have been predicted by the natural history model. There was a corresponding small decrease in utility related to disease progression in the DMT-treated UK MS RSS cohort than would have been predicted by the natural history model.

Key findings

1. Our main study finding was the remarkably similar disability trajectories (EDSS score) over time in the two geographically distinct cohorts of MS patients. Whether the time axis was time since onset or time since ABN eligibility, the Welsh and British Columbia cohorts gave rise to very similar models (aims 1 and 2).

2. The coefficients from the models based on one cohort could be used to predict EDSS score for patients from the second cohort, by conditioning on the first EDSS score observed for an individual. However, the accuracy of these predictions will always be limited by any measurement error in the EDSS score. Thus, when using the BCMS model to predict EDSS scores for individual patients in the UoWMS cohort, conditional on their first two EDSS score observations, 90% of the predictions lay within –2 to 2 points of the observed EDSS score. This compared with an estimated intrarater reliability of 1 point or an inter-rater reliability of 1.5 points, for those scores between 1 and 3.5 points on the EDSS. 39

3. Although the accuracy of these predictions may not be sufficiently good for predicting at an individual level for clinical purposes, these results provide strong evidence of the ability to use one of these cohorts of MS patients as a non-randomised comparison group for another. The overall mean difference between EDSS scores for the UoWMS cohort and those predicted by the BCMS model was –0.001 points, with a standard deviation of 1.35 points, thus giving a 95% CI for the difference of –0.07 to 0.07 points. This was a narrow CI, and thus would be of utility in comparing average effects between two cohorts.

4. We identified several covariates as either being related to the average EDSS score, or to the rate of change in disability (EDSS score) over time. The estimates of these associations were consistent between the two cohorts. Most were consistent with either observations from previously published natural history studies17,21 or with clinical expectations. For instance, consistent with the wider literature,47 those who were older at MS onset tended to have greater disability (higher EDSS scores) and worsened somewhat more rapidly over time. Consistent with clinical expectations, those with SP disease had a higher EDSS score at ABN eligibility. However, of interest was the absence of any obvious difference in subsequent progression compared with those who remained RR. Patients with more relapses in the 2 years prior to ABN eligibility tended to have higher EDSS scores at ABN eligibility. Women had lower EDSS scores, on average, than men but again there were no obvious differences in EDSS score trajectory compared with men.

5. Our study suggests that EDSS trajectories can be modelled using multilevel models. In all models, there was no strong evidence of non-normality of residuals and, therefore, no need to consider further transformations of EDSS. As previously suggested, there was evidence that intra individual variability in the EDSS score and this variability was greater for lower EDSS scores. 39 Our models treat the EDSS as a continuous outcome (rather than ordinal) and we found no evidence that this gave rise to poor prediction or poor fit of the model to the observed data.

6. Finally, our results suggest that progression (as measured by the EDSS) may be slower in a cohort treated with DMT than that predicted by natural history models, with an estimated difference in EDSS score of –0.59 (95% CI –0.64 to –0.54) points at up to 6 years post treatment (aim 3). This translated to a difference in the utility related to disease progression of –0.04 points (95% CI –0.05 to –0.04 points) (aim 4). This is a very small mean difference, particularly when set against the variation in EDSS score both within and between individuals.

Strengths and weaknesses

A major strength of our study was the ability to develop models using one cohort (the BCMS cohort) and to validate their predictive power on a cohort, which was distinct in both time and geography (the UoWMS cohort). The remarkable similarity between the models for these two cohorts indicates that these models may be useful for comparing EDSS score between different populations.

The multilevel model approach meant that coefficients from a model developed on one cohort could be used to predict EDSS score for an individual from a different cohort, and that this prediction could be improved by conditioning on the first (or first two) observed EDSS scores.

Both cohorts contained a reasonably large number of individuals, assessed by MS specialist neurologists at several points over the course of disease. Moreover, many of these data were collected before the DMTs were either widely used or licensed for MS, meaning that most patients were largely unexposed to any drug treatment known to modify the MS disease course. These types of data sets are incredibly valuable and cannot be recreated given the ubiquitous use of DMTs for MS in clinical practice today. However, one limitation is that, for this project, DMTs were considered as a single class, and no attempt was made to model progression differently for different drugs. Another potential source of bias is that only a small proportion of the available data from each cohort was used; the eligibility criteria mean that only those individuals with definite SPMS or RRMS were included, and of those, only those who were ABN eligible, and who had at least two EDSS score measures. These restrictions may have led to selection bias, if underlying progression was related to the chance of being selected for these analyses.

A potential weakness of our study was the inability to capture physician-level information, as we were unable to adjust for the capture of EDSS scores by different physicians. In the BCMS cohort, four core neurologists assessed > 85% of the EDSS scores considered in this current study, thus minimising differences between neurologists. Nonetheless, given the documented variation between measures taken by different raters on the same individual,39 this could limit the precision with which EDSS score can be predicted.

A further issue is that our models treat the EDSS score as a continuous measure. Although the EDSS score is an ordinal measure, and was not designed for use as a continuous outcome, all the models showed good fit to the observed data. Examination of the residuals from the models also showed no evidence of lack of fit of these models. Models treating the EDSS as a continuous outcome are much simpler to fit and interpret than models treating the EDSS as an ordinal scale, and this advantage may outweigh the disadvantage of lack of fit with the original design of the EDSS. We have used only the EDSS (which has a particular focus on walking abilities) as an outcome measure, so have not taken into account other domains of life that are affected by MS (including quality of life).

A further weakness was the censoring of patients after first exposure to a DMT in the UoWMS cohort. In the BCMS cohort, this was largely overcome by limiting the entire cohort to a time period when the DMTs were largely not available (1980–95). If the decision to initiate a DMT was based on one or more of the observed variables which are in the model (e.g. EDSS history after ABN eligibility or relapse rate at ABN eligibility), then this should not have biased the models, and the similarity between the BCMS and UoWMS models would indicate that this was the case.

Our definition of ABN eligibility included the number of relapses in the 2 years prior to a given EDSS score measure. This variable was also one of the covariates in our models. However, a patient in whom data were available for < 2 years since onset, but who had experienced more than two documented relapses in that time, would have satisfied the ABN eligibility criteria, although the number of relapses in the previous 2 years would be based on an observation period of < 2 years. An alternative would have been to calculate an annualised relapse rate for each patient at ABN eligibility; however, we felt that this might give too much weight to an individual who had two relapses in a fairly short time (e.g. two relapses within 6 months would give an annualised relapse rate of four, but this individual is perhaps unlikely to have had another two relapses in the next 6 months). This potential measurement error in our definition of the number of relapses in the 2 years prior to ABN eligibility may have caused overestimation of the association between relapses at ABN eligibility and EDSS score at eligibility.

Another weakness is that patients with SPMS at the time of ABN eligibility were included in the derivation of the natural history models (for both BCMS and UoWMS cohorts), whereas patients with SPMS at baseline were excluded from the UK MS RSS analyses presented here. This could result in the UK MS RSS cohort having lower initial EDSS scores than those from the natural history cohorts; however, given that that natural history models did not identify any association of SPMS with EDSS trajectory, this should not bias our results. Sensitivity analyses presented elsewhere46 confirm that including patients with SPMS at baseline in the UK MS RSS analyses resulted in a tiny attenuation of the difference between UK MS RSS and natural history-predicted values of EDSS score. A similar issue is that in deriving our natural history models, we removed all observations made within 1 month (BCMS cohort) or 3 months (UoWMS cohort) of a relapse. In the UK MS RSS, the clinicians were instructed not to take EDSS score measures during a relapse. Measures made close to (but not during) a relapse in the UK MS RSS could have been higher than usual and thus we could have underestimated the difference between natural history and treated cohorts.

We applied the natural history model to all patients from the UK MS RSS with EDSS score data at 6 years post treatment, and assumed that a difference between observed EDSS score and that predicted by the natural history model could be because of DMT. Patients not remaining on DMT for 6 years in the UK MS RSS may have higher EDSS score values at 6 years than if they had been treated throughout, meaning that we have a conservative estimate of the effect of treatment. We have factored in potential bias because of missing data at 6 years, and this seems unlikely to account for the observed ‘treatment effect’.

The key potential source of bias in this study is that this is a purely observational study, and there could also be other differences between the cohorts that could account for the observed lower EDSS score in the UK MS RSS cohort, rather than this difference being as a result of treatment. Our examination of the portability of the natural history models between the BCMS and UoWMS cohorts gives some confidence that the model from one population can be used as non-randomised comparison group for progression in a different population. However, the difference in EDSS scores is small (–0.6 points) and could at least partially be accounted for by unmeasured differences between the BCMS and UK MS RSS cohorts. For example, if in the UK only those with physician-reported better prognosis, or those who were generally healthier (e.g. with fewer comorbidities), were recruited to the UK MS RSS, this could lead to an apparent better progression in this group than predicted. Similarly, if other improvements in MS care have taken place in the time between the BCMS cohort and the UK MS RSS cohort, this could be responsible for the apparent difference in EDSS score (rather than it being a result of DMT treatment). The agreement between the BCMS model (data for this study collected 1980–95) and the UoWMS model (data for this study collected 1976–2011) might suggest that there has not been a time-dependent improvement in EDSS score; however, with the comparison of only two cohorts, strong conclusions cannot be drawn.

Future research

Our model development opens up several important areas of future research. First, although these models have been developed and validated using two valuable natural history data sets, further validation in other MS cohorts would be useful, the caveats being the relative dearth of suitable data sets and the financial constraints, which may limit long-term follow-up. The ability of these models to predict disability (EDSS score) for the individual (rather than the population average), conditional on their first observed EDSS score and characteristics such as age at onset, needs further investigation and validation. This could include examination of different modelling strategies (such as Markov transition models or different parameterisations of time).

Such research would need to be accompanied by qualitative research into the needs of patients, their families, and clinicians, and to provide evidence around the best way to present prognostic models to these groups. In addition, qualitative research could examine the value put upon the small mean difference in EDSS score by patients, their carers and families, and how this balances against the prospects of long-term treatment. Utilities (as used here) are a relatively crude measure to examine this, and utility may change over time or depending on the treatment regime used. Multivariate models could be used to jointly model outcomes of relevance to patients, in addition to EDSS score, for example modelling EDSS score progression jointly with the occurrence of relapses or a quality-of-life score.

Second, these models were developed in patients who became ABN eligible and, therefore, represent a subset of MS patients. Although these individuals are highly relevant to the UK MS RSS (as these patients represent the only individuals likely to receive these drugs), applying these models to other cohorts with broader prescribing ability might be limited. Similar modelling strategies could be used to model natural history of MS in all patients with RRMS, rather than restricting to this subgroup. The models could also be expanded to look at prognosis in those with SPMS. Focusing follow-up on those who may be likely to progress more quickly (e.g. those with a high EDSS score at baseline and RRMS) would also enable more accurate modelling of factors affecting prognosis for those individuals.

A third key area is to what degree the uncertainty around individual predictions could help decide management decisions around different therapeutic options. Newer agents for MS, such as natalizumab,48 fingolimod or alemtuzumab,49 appear to have greater benefits in terms of reductions in relapse rates, but their impact on longer-term gains in terms of disease progression is uncertain, and any gains are at a higher burden of side effects and toxicity. Thus, careful patient selection for these drugs is important. Further research should examine how incorporation of formal prognostic models may or may not enhance shared clinical decision-making between clinicians and patients.

The same approach as used here could also be applied to assessing the associations of newer DMTs and other potential treatments for MS with progression, to aid in health-care decision-making. While, this approach (using a natural history non-randomised comparison group) remains inferior to randomised controlled trials, it may represent a compromise in terms of the ethical and financial constraints on future funding of randomised controlled trials. Recent developments in network meta-analysis also allow better use of randomised controlled trial data to infer different comparisons among treatments. 50

Here, we used natural history models as a non-randomised comparison group for a treated population, in order to estimate the long-term effect of treatment. More robust estimates of this long-term effect would be provided if it were feasible (ethically, logistically and financially) to conduct long-term randomised controlled trials, and by bringing all available evidence together using systematic review and (when appropriate) meta-analysis. It would also be useful to examine whether or not any treatment effect remains constant over time, and whether or not there are any subgroups of patients (e.g. those with lower baseline EDSS score, those with longer duration of disease) who benefit more or less from DMT treatment.

Other acknowledgements

We would like to thank:

Professor Alasdair Coles, Professor Mark Gilthorpe and Professor Andrew Pickles, who formed the Advisory Group for this project and provided helpful input to the model development.

Dr Jackie Palace, Dr Charles Dobson and Dr Martin Duddy, who led the UK MS RSS, and who provided guidance on clinical interpretation and implications.

Thomas Bregenzer (Paraexcel), who provided the data from the UK MS RSS cohort, and also comparison analyses using Markov models.

Professor Richard Lilford (chairperson) and Professor Richard Gray and Dr Pelham Barton, of the UK MS RSS Scientific Advisory Group, who developed the statistical analysis plan for the main RSS analysis.

The MS Trust, which provided access to the data from the UK MS RSS and sponsored Mr Michael Lawton in applying this work to further analyses of the UK MS RSS data set, in particular Nicola Russell, who facilitated access to the data.

BCMS cohort, we thank all the MS patients for participating in these studies. We gratefully acknowledge the BCMS clinic neurologists who contributed to the study through patient examination and data collection (current members listed here by primary clinic):

University of British Columbia MS Clinic: A Traboulsee (University of British Columbia Hospital MS Clinic Director and Head of the UBC MS Programmes); A-L Sayao; V Devonshire; S Hashimoto (University of British Columbia and Victoria MS Clinics); J Hooge (University of British Columbia and Prince George MS Clinic); L Kastrukoff (University of British Columbia and Prince George MS Clinic); and J Oger.

Kelowna MS Clinic: D Adams; D Craig; and S Meckling.

Prince George MS Clinic: L Daly.

Victoria MS Clinic: O Hrebicek; D Parton; and K Atwell-Pope.

UoWMS cohort: we thank all the MS patients for participating in these studies. We gratefully acknowledge the neurologists who contributed to the study through patient examination and data collection (current members listed here in alphabetical order): M Cossburn; C Hirst; G Ingram; S Luppe; T Pickersgill; and also M Wardle, who wrote the MS database.

Contributions of authors

Kate Tilling (Professor of Medical Statistics, University of Bristol) helped design and obtain funding for the project. She also oversaw the project including all the statistical analyses, and drafted and revised the manuscript.

Michael Lawton (Research Assistant, Medical Statistics, University of Bristol) performed all the statistical analyses, generated the data sets from the UoWMS cohort based on the selection criteria, and assisted in writing and revising the manuscript.

Neil Robertson (Professor of Neurology, Institute of Psychological Medicine and Clinical Neuroscience, Cardiff University) collected and collated clinical data for the UoWMS cohort, gave feedback on the analysis and revised the manuscript.

Helen Tremlett (Professor and Canada Research Chair in Neuroepidemiology and MS, Faculty Medicine, Division of Neurology, University of British Columbia) facilitated and supervised patient selection and data extraction for the BCMS cohort, gave feedback on the analysis and revised the manuscript.

Feng Zhu (Statistician, Department of Medicine, Division of Neurology, University of British Columbia) generated the data sets from the BCMS cohort based on the selection criteria and revised the manuscript.

Katharine Harding (Clinical Lecturer in Neurology, Institute of Psychological Medicine and Neurology, Cardiff University) collected and collated clinical data for the UoWMS cohort, gave feedback on the analysis and revised the manuscript.

Joel Oger (Professor, University of British Columbia) was one of the neurologists entering patients in the BCMS database for over 30 years and has financed the database for over 10 years. He ensured that when new clinics were created in British Columbia they reported to the main database, and negotiated with the local neurologists for the participation of the BCMS database in the UK MS RSS. He has reviewed a draft of the manuscript.

Yoav Ben-Shlomo (Professor of Clinical Epidemiology, University of Bristol) helped design and obtain funding for the project. He was also involved in setting up the collaborations with the UoWMS and BCMS data sets, discussing the analysis plans, commenting on initial results, and reviewing and commenting on the draft and final report.

Publications

Lawton M, Tilling K, Robertson N, Tremlett H, Zhu F, Harding K, et al. A longitudinal model for disease progression was developed and applied to multiple sclerosis. J Clin Epidemiol 2015:68;1355–65.

Palace J, Duddy M, Bregenzer T, Lawton M, Zhu F, Boggild M, et al. Effectiveness and cost-effectiveness of interferon beta and glatiramer acetate in the UK Multiple Sclerosis Risk Sharing Scheme at 6 years: a clinical cohort study with natural history comparator. Lancet Neurol 2015;14:497–505.

Data sharing statement

Requests for access to the natural history data should be addressed to Helen Tremlett (BCMS cohort) or Neil Robertson (UoWMS cohort). Requests for access to the UK MS RSS data should be addressed to Nicola Russell at the Multiple Sclerosis Trust, Spirella Building, Letchworth SG6 4ET, UK.

Disclaimers

This report presents independent research funded by the National Institute for Health Research (NIHR). The views and opinions expressed by authors in this publication are those of the authors and do not necessarily reflect those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health. If there are verbatim quotations included in this publication the views and opinions expressed by the interviewees are those of the interviewees and do not necessarily reflect those of the authors, those of the NHS, the NIHR, NETSCC, the HTA programme or the Department of Health.