Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies

Data

Data were combined as part of the BRANDO (Bias in Randomized and Observational studies) project. The authors of 10 meta-epidemiological studies9,11–15,18–21 (Table 1) agreed to contribute their original data to a combined database that would be used to conduct combined analyses and investigate reasons for differences between the results of the original studies. The combined database was created using Microsoft Access^™ software (Microsoft Corporation, Redmond, WA, USA). Most authors supplied separate tables containing data on included trials and meta-analyses, allowing linkage between these two levels of data. When such information was not supplied, information from publications of relevant systematic reviews was used to link trials and meta-analyses. 15 Citations (of publications from which the data were extracted) were linked to the data sets using matching via source systematic review, author name and publication year. 12–15 Studies by Sampson et al. 20 and McAuley et al. 18 and the part of the study by Egger et al. 9 that was based on published journal articles [referred to hereafter as Egger (journal)] did not include information on study design characteristics in the included trials, whereas the study by Royle and Milne19 did not include outcome data. These studies cannot contribute to meta-epidemiological analyses, but were retained to contribute to other, descriptive analyses.

Assigning identification numbers and creating a unified data set

To enable automated identification of meta-analyses and trials that occurred in more than one meta-epidemiological study, we labelled each trial and review entry in contributing data sets with a MEDLINE, EMBASE or ISI Web of Science unique publication identifier (ID), in respective order of preference. We first searched MEDLINE (through PubMed^™) using bibliographic information for every trial and review included in contributing data sets [e.g. full title, author(s), publication year, journal, volume, pagination] and retrieved their unique MEDLINE identifier (PMID). For references that were not located in MEDLINE, EMBASE was searched (through Ovid^™) in the same way and EMBASE-unique identifiers were retrieved. For references that were not located in EMBASE the search was carried out in the ISI Web of Science database, but only a small number of the remaining non-indexed citations were identified in this way. Trial references not indexed in MEDLINE, EMBASE or ISI were cross-checked for duplication with other references using duplicate search facilities in Reference Manager^™ bibliographic database software (Thompson Reuters, New York, NY, USA), followed by manual checking. All non-indexed, unique references were assigned a unique identification number derived from their ID number in the Reference Manager database.

We defined in the master database the variables that we wished to combine from the contributing studies. An initial combined data set was then created by mapping each variable in each contributed data set to the predefined variable in the master database that it most closely matched. New variables were then added to the master database to capture contributed data that did not fit the a priori variable definitions.

Identification and removal of duplicate meta-analyses and trials

The unique identifiers of trials included in each meta-analysis in the combined data set were compared with those in every other meta-analysis (regardless of whether or not the meta-analyses assessed the same outcome). Using Intercooled Stata^™ version 9 (StataCorp LP, College Station, TX, USA), meta-analyses that contained any trials in common with any other meta-analysis were grouped together. Not all meta-analyses within these sets contained overlapping trials: for example, two meta-analyses with no trials in common might each contain trials in common with a third meta-analysis and these three meta-analyses formed one set.

Meta-analyses that had no overlap with any other meta-analysis in the combined data set were transferred to the final data set. We then considered each set of overlapping meta-analyses in turn. Meta-analyses were excluded from each set until there was minimal overlap between the remaining meta-analyses, according to the following rules:

1. Exclude meta-analyses from the Royle and Milne19 study, which did not include outcome data (entire meta-analysis removed regardless of the extent of overlap).

2. Exclude meta-analyses for which information on study design characteristics was not available in preference to those for which such information was available. If both meta-analyses in a duplicate pair contain no information on study design characteristics, exclude meta-analyses from earlier studies first: in order Egger (journal),9 Sampson et al. ,20 McAuley et al. 18 (entire meta-analysis removed regardless of the extent of overlap).

3. Exclude meta-analyses from the portion of the study by Egger et al. 9 that was based on assessment of study design characteristics in Cochrane reviews in preference to meta-analyses from studies that directly assessed these characteristics.

4. Exclude meta-analyses with fewer assessed study design characteristics in preference to those with more assessed characteristics.

5. Exclude meta-analyses from less recently published systematic reviews in preference to more recently published reviews.

6. Exclude meta-analyses including fewer trials in preference to meta-analyses including more trials.

These rules were used in order of priority, that is, the next rule was applied only if the previous could not yield a decision. We recorded reasons for all decisions to exclude meta-analyses. Pairs of meta-analyses with recorded study design characteristics and results that had only a minimal overlap between them were retained at this stage. Overlap was considered minimal if the number of overlapping trials was no more than 10% of the sum of the numbers of overlapping trials and unique trials from both meta-analyses. We then removed the overlapping trials from the meta-analysis in the duplicate pair that (1) contained more trials or (2) according to the rules 3 to 5 above. If these rules did not yield a clear decision, overlapping trials were removed from one of the meta-analyses at random. At both stages, the choice of meta-analyses or trials to be removed was independent of the assessment of the study design characteristics, and disagreements in these assessments between studies were not considered. Similarly, the decision on removal was also independent of the types of outcome measure assessed in the overlapping meta-analyses.

Some meta-analyses contained multiple results from the same trial, usually because they included multiarm trials in which the same control/comparison group was compared with two different treatment groups. Where appropriate, the two treatment groups were combined. In other cases one of the results was removed at random. We then checked the 2 × 2 results tables from trials in the deduplicated data set against the data in the source review publications. Inconsistencies were clarified with the contributors and corrected where necessary. We recoded the direction of outcome events where necessary so that the coded outcome for each trial corresponded to an adverse (undesirable) event.

Assessing consistency of assessment of reported study design characteristics in different contributing studies

We used trials included in more than one meta-epidemiological study to assess the reliability of the assessment of methodological characteristics between studies. For this analysis we defined such trials as those with the same bibliographic reference occurring in more than one meta-epidemiological study, irrespective of numbers of participants or events, or the type of outcome measure. To assess the inter-rater reliability we compared contributors’ assessments of the following three methodological characteristics: adequacy of the method for generating the random sequence used for allocating participants to treatment groups (sequence generation), adequacy of concealment of treatment allocation from participants and investigators at the point of enrolment into the trial (allocation concealment), and contributors’ assessments of whether or not a trial was double blind or not (blinding). Kappa statistics were calculated for the assessment of sequence generation (inadequate/unclear compared with adequate), allocation concealment (inadequate/unclear compared with adequate) and blinding (not double blind/unclear compared with double blind) of duplicated trials for each pair-wise comparison between contributing meta-epidemiological studies. Only comparisons with at least 10 overlapping trials between any two meta-epidemiological studies were analysed, for each of the three characteristics.

Classification of interventions and outcome measures

We classified the type of experimental intervention, type of comparison intervention and type of outcome measure for each meta-analysis in the final data set. Interventions were classified based on an expanded and modified version of the classification by Moja et al. 23 Comparison interventions were further classified as inactive (e.g. placebo, standard care) or active. When it was not clear which intervention group should be considered experimental and which the comparison, at least two study collaborators made a consensus decision. In cases when a decision could not be reached, such meta-analyses were excluded from analyses of associations between study design characteristics and intervention effect estimates.

We classified outcome measures according to an expanded and modified version of the classification developed by Wood et al. 16 Outcome measures were further grouped as all-cause mortality, other objectively assessed (including pregnancy outcomes and laboratory outcomes), objectively measured but potentially influenced by clinician/patient judgement (e.g. hospital admissions, total dropouts/withdrawals, caesarean section, operative/assisted delivery) and subjectively assessed. When a review reported that different methods of outcome assessment were used in different trials within the same meta-analysis, the meta-analysis was classified according to the most subjective method of outcome assessment. This was the case in 16 meta-analyses. For example, some trials of smoking cessation assessed outcomes using exhaled carbon dioxide or salivary cotinine (classified as objectively assessed), whereas for others it was by patient self-report (classified as subjectively assessed). Classifications of interventions and outcome measures were checked by at least two of the collaborators who were clinicians by training (CG, LLG, BA-N or JP). Classifications of both outcome measures and interventions were based solely on the information provided in the review from which the meta-analysis was extracted. We did not retrieve further information on individual outcome measures from publications of included trials.

Database structure

The database design allowed for multiple results from the same trial to be contained in different meta-analyses, multiple meta-analyses in systematic reviews, overlapping meta-analyses between systematic reviews and multiple references to the same trial or review. The final database structure consisted of six tables (Figure 1). The first five tables, References, Trials, Trial Results, Meta-analyses and Systematic Reviews, contained relevant study characteristics. The sixth table, Relationships, contained the identifiers necessary to link information between all the other tables. Each reference entry was linked to its related trial or systematic review, and further links were established between trials, trial results, meta-analyses and systematic reviews in the corresponding database tables. Figure 1 lists core variables included in each table.

FIGURE 1.

Schematic representation of the final database structure. ICD-10, International Classification of Disease, 10th edition. 30

Removing overlaps

Figure 2 depicts the derivation of the final database through removal of overlaps between meta-analyses. The initial combined data set contained 427 systematic reviews with 454 meta-analyses, 4857 trials and 4874 trial results. Of the 454 meta-analyses, 196 contained at least one trial that overlapped with at least one other meta-analysis, among which we identified 71 sets of meta-analyses containing overlapping trials. The size of these sets varied from 2 to 17 meta-analyses (median 2). We removed 91 entire meta-analyses containing 1354 trial results during the deduplication process. Of the 1354 results removed, 844 were duplicates of trials retained in the database, whereas 510 were unique. Of those that were unique, 340 (67%) were from studies that did not record either methodological characteristics or results, thus 170 potentially informative trials were removed during this process. A further 24 individual trial results were removed from 16 additional meta-analyses for which overlap was minimal (see methods). One trial was removed from 10 meta-analyses, two trials from four meta-analyses, and three trials from two meta-analyses each. An additional 19 trial results were removed as a result of overlaps within nine of the meta-analyses (e.g. where there were two comparisons with a common control group). The final database contained 352 systematic reviews contributing 363 meta-analyses, 3474 trials and 3477 trial results.

FIGURE 2.

Flow diagram depicting the deduplication process.

Table 2 shows an example of a set of four meta-analyses containing overlapping trials. Three meta-analyses (1005, 2456 and 2531) were taken from the same Cochrane review,24 whereas 2097 was from a journal review25 with a similar topic. Table 3 summarises the overlap between the trials in each pair of meta-analyses. To remove overlaps, meta-analyses were dropped in the following order: (1) meta-analysis 2097, because it was contributed by the Egger (journal) study,9 which did not include study design characteristics; (2) meta-analysis 2456, because study design characteristics in this study were extracted from a Cochrane review rather than from primary publications. The overlap between the remaining two meta-analyses (1005, from the study by Contopoulos-Ioannidis et al. 21 and 2531 from the study by Kjaergard et al. 15) was then examined in detail (Table 4). Each meta-analysis contained 10 trials, of which eight overlapped. Of the overlapping trials, seven had the same totals per group (different event numbers), and one trial had different data for the control group. These meta-analyses assessed two different outcomes: that in meta-analysis 1005 was ‘No change in positive and negative syndrome scale (data greater than 20%)’, whereas that in meta-analysis 2531 was ‘dropouts’. This explains why there was no exact correspondence in the 2 × 2 data for any trial. Meta-analysis 1005 was retained because the contributing study provided information on one additional study design characteristic (sequence generation).

Reliability of assessment of reported study design characteristics

Eight of the participating meta-epidemiological studies contained sufficient trials (at least 10) in common to contribute to analyses of the reliability of assessment of reported study design characteristics (Table 5). Overall, there was good agreement between assessments carried out in the different contributing studies. For sequence generation (two comparisons), the percentages of studies in which the assessments were in agreement were 81% and 82% and kappa statistics were 0.56 and 0.64. For allocation concealment (12 comparisons), percentage agreement varied between 52% and 100% and kappa statistics between 0.19 and 1.00 (median 0.58). The lowest kappa (0.19) was observed in the comparison between the Egger et al. 9 and Pildal et al. 14 studies: 10 (48%) studies assessed as having adequate concealment of allocation in the Egger study (which used assessments reported by Cochrane review authors) were assessed as having inadequate or unclear concealment of allocation in the Pildal study. Assessments were most reliable for blinding (nine comparisons): percentage agreement ranged from 80% to 100% (in four comparisons), and kappa statistics ranged from 0.55 to 1.00 (median 0.87).

Interventions and outcome measures

Outcome measures were initially classified into one of 23 categories. Table 6 shows the numbers of meta-analyses with outcomes in each of these categories, together with the final category grouping. The most common outcome measure was all-cause mortality (64 meta-analyses, 18%), followed by clinician-assessed outcomes (51 meta-analyses, 14%).

Experimental and comparison interventions were classified as shown in Table 7. The majority of included meta-analyses (242 meta-analyses, 67%) assessed a pharmacological experimental intervention. The most common comparison intervention was placebo or no treatment (225 meta-analyses, 62%), followed by a pharmacological comparison (54 meta-analyses, 15%). Forty-eight meta-analyses (397 trial outcomes) in which it was unclear what was the experimental intervention were excluded from meta-epidemiological analyses.

Description of the final database

The final data set contained information from 363 meta-analyses containing 3474 trials. Of these, 282 meta-analyses (2572 trials) had 2 × 2 results data available. A total of 186 meta-analyses (1236 trials) had information on randomisation sequence generation, 228 (1840) on allocation concealment and 234 (1970) on blinding. In total, 175 meta-analyses (1171 trials) had information on 2 × 2 results data and these three study design characteristics.

Statistical methods

Intervention effects were modelled as log-odds ratios and outcomes were recoded where necessary so that odds ratios (ORs) < 1 corresponded to beneficial intervention effects. We fitted Bayesian hierarchical bias models using the formulation previously described as ‘Model 3’ by Welton et al. 26 We assumed that the observed number of events in each arm of each trial has a binomial distribution, with the underlying log-odds ratio in trial i in meta-analysis m (LORim) equal to

where X_im = 1 and 0 for trials with and without the reported characteristic. The parameter δ_im represents the intervention effect in trial i of meta-analysis m. These are assumed to be randomly distributed with variance τm2 within each meta-analysis:

Parameter β_im quantifies the potential bias associated with the study design characteristic of interest. We assumed the following model structure:

for trials i with the reported characteristic in each meta-analysis m, and

across meta-analyses.

This allows for three effects of bias. First, mean intervention effects may differ between trials with and without the reported study design characteristic. Estimated mean differences (b₀) were exponentiated and are thus reported as ratios of odds ratios (RORs). Second, variation in bias between trials within meta-analyses is quantified by standard deviation κ; κ² corresponds to the average increase in between-trial heterogeneity in trials with a specified study design characteristic. Third, variation in mean bias between meta-analyses is quantified by between-meta-analysis standard deviation φ. We derived 95% credible intervals (CrI) for each parameter. In presenting results from our primary analyses, we also display the posterior variance of the parameter b₀, denoted by V₀. Use of this value in downweighting results from trials at high risk of bias in future meta-analyses is discussed below. For the results of our secondary analyses we do not present V₀ explicitly, but the posterior uncertainty about b₀ is reflected in the CrI for the ROR.

Data management and cleaning prior to analysis, and graphical displays of results, were carried out using Stata Version 11. Bias models were then fitted using WinBUGS Version 1.4 (MRC Biostatistics Unit, Cambridge, UK),27 assuming vague prior distributions for unknown parameters. Preliminary results indicated that estimated variance components κ and φ were sensitive to the prior distributions assumed for these parameters. Sensitivity to priors for variance parameters is a well-known problem in Bayesian hierarchical modelling. 28,29 This motivated a simulation study in which the performance of a range of prior distributions for variance components was compared, assuming typical values from the BRANDO database (Harris et al. , Health Protection Agency, 2010, personal communication). The prior found to give the best overall performance was a modified Inverse Gamma(0.001, 0.001) prior with increased weight on small values. This prior was therefore assumed for each variance parameter in all analyses. For location parameters (overall mean bias, baseline response rates, treatment effects), Normal(0, 1000) priors were assumed.

Meta-analyses can inform estimates of the effect of a study design characteristic only if they contain at least one trial with and one without the characteristic. We refer to such meta-analyses and trials from these meta-analyses as informative. As it is impossible to estimate both τ² and κ in a meta-analysis with fewer than two studies with and without the study design characteristic of interest, such meta-analyses were prevented from contributing to the estimation of κ by use of the ‘cut’ function in WinBUGS.

We first conducted univariable analyses for each study design characteristic separately using all informative meta-analyses for that characteristic. The primary analysis used dichotomised variables for each characteristic (inadequate/unclear compared with adequate for sequence generation and allocation concealment, and not double blind/unclear compared with double bind). All such analyses were repeated separately for different types of outcome measure (all-cause mortality, other objectively assessed and subjectively assessed). Evidence that effects of bias differed according to the type of outcome was quantified using posterior probabilities that effects for subjective or other objective outcomes were larger than those for mortality outcomes: for example Pr(κ_subjective > κ_mortality). The main univariable analyses were repeated using the meta-meta-analytic approach used in previous analyses8,16 allowing for random effects both within and between meta-analyses.

Further univariable and multivariable analyses were conducted using two data subsets: (1) meta-analyses of trials with information on all three study design characteristics and (2) meta-analyses of trials with information on both allocation concealment and blinding. Subset 2 was used because many studies did not have a recorded bias judgement on sequence generation (see Table 11). We conducted univariable analyses on three composite dichotomous variables: risk of bias due to inadequate/unclear allocation concealment or lack of blinding (using subset 2), any risk of selection bias (inadequate or unclear sequence generation or allocation concealment) and any risk of bias (inadequate or unclear sequence generation or allocation concealment, or not double blind). Analyses of the second two composite variables used subset 1.

Multivariable analyses were based on an extension of ‘Model 3’ of Welton et al. 26 We assumed distinct variance components associated with each study design characteristic. In the main multivariable analyses, we assumed no interactions between the different characteristics. In an additional analysis on subset 2 we allowed for interactions between inadequate allocation concealment and lack of double blinding. Interaction terms were assumed to have the same hierarchical structure as the main effects, again with distinct variance components. The implied average bias in studies with both characteristics was estimated (on the log-odds scale) as the sum of the fitted coefficients representing the average effect of each of the two characteristics and the fitted coefficient representing the average interaction term. A 95% CrI for this sum, accounting for correlations between the three coefficients, was calculated using WinBUGS. These measures were exponentiated in order to express the implied average bias as ROR. For comparison, we also calculated the corresponding implied average bias for the model without interaction terms for subset 2. We repeated all univariable analyses on subsets 1 and 2 to allow comparisons with the results of multivariable analyses.

In additional analyses we estimated separate effects of ‘inadequate’ and ‘unclear’, for each study design characteristic, by fitting models in which these two categories had different average bias (compared with ‘adequate’ trials) and distinct variance components (κ and φ). It was necessary to exclude some of the contributing meta-epidemiological studies from these analyses because their original data coding had not separated unclear from inadequate (see Tables 9 and 11). We conducted separate analyses according to clinical area and type of intervention, and repeated analyses in meta-analyses that were derived from the subset of contributing studies not included in the study of Wood et al. 16 For meta-analyses comparing two active interventions it is not possible to estimate RORs quantifying average bias, or between-meta-analysis heterogeneity in average bias, because there is no clear direction in which bias operates. Instead, we estimated parameters from restricted models to estimate increases in between-trial (within-meta-analysis) heterogeneity in trials with, compared with those without, specified study design characteristics, among meta-analyses containing at least two trials with and without the characteristic of interest.

Welton et al. 26 showed how results from hierarchical bias models can be used to formulate a prior distribution for the bias associated with a study design characteristic in a new fixed-effect meta-analysis that is assumed to be statistically exchangeable with the meta-analyses used to estimate the model parameters. This approach was based on a normal approximation to the distribution of the observed intervention effect y_i in each new trial i, which is assumed to have known sampling variance σ_i². Assuming known b₀, κ, φ and V₀, a posteriori use of the empirically based prior distribution leads to results from trials with the reported characteristic being corrected for the estimated average bias across meta-analyses (estimated b₀), and the variance of such results being increased from σ_i² to (σ_i² + κ² + φ² + V₀). The minimum variance of the estimated intervention effect that can in theory be achieved by an infinitely large trial with the reported characteristic is therefore κ² + φ² + V₀.

It is of interest to quantify the likely magnitude of increases in variance resulting from application of this bias adjustment to future trial results. To do so, for each trial in the BRANDO data set we calculated the observed log-odds ratio y_i and the Woolf estimate of its sampling variance, σ_i². We assumed that these represented a typical range of σ_i²s that may be observed in future trials. For each high/unclear-risk trial result in turn, we calculated the percentage increase in variance that would result from bias adjustment at the trial level, that is:

The calculations used the posterior median values of κ and φ. We summarised these percentage increases in variance by the median and interquartile range (IQR) across trials for each study design characteristic.

Formulae from Welton et al. 26 also allow us to calculate a bias-adjusted summary mean and variance of the intervention effect in a new fixed-effect meta-analysis. Using this approach, results from trials at low risk of bias are assigned the usual inverse variance (1/σ_i²) weight. For trials at high/unclear risk of bias, the bias-corrected estimated effect size is used, and downweighted according to a function of σ_i², κ², φ² and V₀ (interested readers are referred to page 123 of Welton et al. 26). The resulting bias-adjusted estimate of the summary effect size will therefore have a larger variance than the standard unadjusted meta-analytic summary. The magnitude of this increase in variance will depend on the number of trials and the variances of the intervention effect estimates in the new meta-analysis, and the number of trials classified as high/unclear risk of bias. We assumed that the meta-analyses in the BRANDO database are typical in terms of these characteristics, and calculated the percentage increase in variance of the summary log-odds ratio due to bias adjustment for each meta-analysis in turn. These were summarised by the median and IQR across meta-analyses for each study design characteristic.

Influence of reported study design characteristics on intervention effect estimates: univariable analyses of individual characteristics

Figure 3 and Table 14 present results from univariable analyses of the influence of reported study design characteristics on intervention effect estimates, both overall and separately according to type of outcome measure. Compared with Figure 3, Table 14 additionally includes 95% CrIs for the variance parameters κ and φ and displays the numbers of trials, and trials at high risk of bias, included in analyses. Overall, intervention effect estimates were exaggerated by an average of 11% in trials with inadequate or unclear sequence generation (ROR 0.89, 95% CrI 0.82 to 0.96), and between-trial heterogeneity was higher among such trials (κ = 0.16, 95% CrI 0.03 to 0.27). When analyses were stratified according to the type of outcome measure, the average effect of inadequate or unclear sequence generation appeared greatest for subjective outcomes [ROR 0.83, 95% CrI 0.74 to 0.94, posterior probability (PPr) that ROR_subjective < ROR_mortality = 0.73], and the increase in between-trial heterogeneity was also greatest for such outcomes (κ = 0.20, CrI 0.03 to 0.32, PPr that κ_subjective > κ_mortality = 0.78). In contrast, there was little evidence that inadequate or unclear sequence generation was associated with exaggeration of intervention effects for all-cause mortality (ROR 0.89, 95% CrI 0.75 to 1.05) or for other objective outcomes (ROR 0.99, 95% CrI 0.84 to 1.16). For all types of outcome measure there was only limited between-meta-analysis heterogeneity in mean bias (estimated φ between 0.04 and 0.07).

FIGURE 3.

Estimated RORs and effects on heterogeneity associated with reported study design characteristics, according to the type of outcome measure: univariable analyses based on all available data. Random sequence generation: 944 trials from 112 informative meta-analyses; allocation concealment: 1292 trials from 146 informative meta-analyses; blinding: 1057 trials from 104 informative meta-analyses.

Overall, intervention effect estimates were exaggerated by 7% in trials with inadequate or unclear allocation concealment (ROR 0.93, 95% CrI 0.87 to 0.99), and there was evidence that between-trial heterogeneity was increased for such studies (κ = 0.12, 95% CrI 0.02 to 0.23). The influence of inadequate or unclear allocation concealment appeared greatest among meta-analyses with a subjectively assessed outcome measure (ROR 0.85, 95% CrI 0.75 to 0.95, PPr that ROR_subjective < ROR_mortality = 0.97; κ = 0.20, 95% CrI 0.02 to 0.33, PPr that κ_subjective > κ_mortality = 0.85). In contrast, the average effect of inadequate or unclear allocation concealment was close to the null for meta-analyses with mortality (ROR 0.98, 95% CrI 0.88 to 1.10) and other objective outcomes (ROR 0.97, 95% CrI, 0.85 to 1.10). Estimates of both between-trial and between-meta-analyses heterogeneity in bias were lower for such outcomes than for subjectively assessed outcomes.

Lack of, or unclear, double blinding was associated with an average 13% exaggeration of intervention effects (ROR 0.87, 95% CrI 0.79 to 0.96). There was evidence that between-trial heterogeneity was increased for such studies (κ = 0.14, 95% CrI 0.02 to 0.30), and that average bias varied between meta-analyses (φ = 0.14, 95% CrI 0.03 to 0.28). Average bias (ROR 0.78, 95% CrI 0.65 to 0.92), increased between-trial heterogeneity (κ = 0.37, 95% CrI 0.19 to 0.53) and between-meta-analysis heterogeneity in average bias (φ = 0.23, 95% CrI 0.04 to 0.44) all appeared greatest for meta-analyses assessing subjective outcomes (PPr ROR_subjective < ROR_mortality = 0.94, PPr κ_subjective > κ_mortality = 0.99, PPr φ_subjective > φ_mortality = 0.90). Among meta-analyses with subjectively assessed outcomes, the influence of lack of blinding appeared greater than the influence of inadequate or unclear sequence generation or allocation concealment.

Results from univariable analyses of the influence of inadequate or unclear allocation concealment and lack of double blinding restricted to the 228 meta-analyses (1793 trials) that contained information on both of these characteristics were similar (Table 15) to those presented in Figure 3 and Table 14. We also repeated the univariable analyses in a data set restricted to 175 meta-analyses (1171 trials) that had information on all three study design characteristics (sequence generation, allocation concealment and blinding). The results were similar (Table 16), although the influence of inadequate allocation concealment appeared somewhat greater in analyses restricted to the 88 informative meta-analyses (811 trials) that had information on all three characteristics.

Influence of reported study design characteristics: univariable analyses of combinations of characteristics

We conducted further univariable analyses using variables defined based on combinations of the study design characteristics. Figure 4 and Table 16 show the estimated effects of any risk of selection bias (inadequate or unclear sequence generation or allocation concealment, compared with other trials) on intervention effects and heterogeneity, based on 53 informative meta-analyses containing 534 trials, of which 89 (17%) were assessed as being at low risk of selection bias. Risk of selection bias was associated with an average 11% exaggeration of intervention effect estimates (ROR 0.89, 95% CrI 0.78 to 1.00) and with increased between-trial heterogeneity (κ = 0.12, 95% CrI 0.02 to 0.27). Average effects did not differ substantially according to type of outcome measure. Between-meta-analysis heterogeneity was highest for objective outcomes (φ = 0.20, 95% CrI 0.02 to 0.68).

FIGURE 4.

Estimated RORs and effects on heterogeneity associated with combinations of study design characteristics: univariable analyses according to type of outcome measure. Analyses based on subsets of trials with assessments of all three study design characteristics (upper and middle panels), and of allocation concealment and blinding (lower panel). Upper panel: inadequate or unclear sequence generation or allocation concealment (any risk of selection bias); middle panel: inadequate or unclear sequence generation or allocation concealment or lack of or unclear double blinding (any risk of bias); lower panel: inadequate or unclear allocation concealment or lack of or unclear double blinding.

Only 37 informative meta-analyses [409 trials, of which 58 (14%) were assessed as being at low risk of bias] contributed to the analysis of any risk of bias (inadequate or unclear sequence generation or allocation concealment, or lack of or unclear double blinding, compared with all other trials); Figure 4 and Table 16 show estimated effects of any risk of bias from one of the three characteristics (compared with low risk of bias from all three). Any risk of bias was associated with an average 21% exaggeration of intervention effect estimates (ROR 0.79, 95% CrI 0.64 to 0.92). The numbers of informative meta-analyses and trials included in analyses stratified by type of outcome measure were small, but the influence of any risk of bias appeared smallest for all-cause mortality outcomes.

A total of 104 informative meta-analyses [990 trials, of which 259 (26%) were assessed as at low risk of bias] contributed to analyses of the influence of inadequate or unclear allocation concealment or lack of double blinding (compared with adequate allocation concealment and presence of double blinding). Figure 4 and Table 15 show that intervention effects from trials at high risk of bias according to this variable were exaggerated by an average of 12% (ROR 0.88, 95% CrI 0.81 to 0.95). The ROR was closer to 1 (0.95) for all-cause mortality outcomes than for other objective or subjective outcomes (0.84, 95% CrI 0.69 to 1.00 and 0.83, 95% CrI 0.73 to 0.93 respectively). The increase in between-trial heterogeneity appeared greatest for subjective outcomes (κ = 0.17, 95% CrI 0.02 to 0.31).

Univariable analyses: comparison with meta-meta-analytic approach

Table 17 displays results of univariable analyses, based on all available data, conducted using the meta-meta-analytic approach8 employed in previous work. 16 Using this approach, we estimated the ROR and the between-meta-analysis standard deviation φ associated with each study design characteristic. Increases in between-trial variability are not estimated using this approach.

For inadequate or unclear sequence generation, inadequate or unclear allocation concealment and lack of or unclear double blinding, estimated RORs are broadly consistent with those from the hierarchical bias models displayed in Figure 3. For each study design characteristic, the exaggeration of intervention effect estimates is greater for meta-analyses with subjectively assessed outcomes than for meta-analyses with all-cause mortality or other objectively assessed outcomes. Consistent with the small estimated values of φ in the hierarchical models, the between-meta-analysis standard deviation was estimated as zero for each of these characteristics in meta-analyses with all-cause mortality outcomes and also, for sequence generation and allocation concealment, in trials with other objectively assessed outcomes. Between-trial variability in bias was greatest for lack of or unclear double blinding, in meta-analyses with other objective or subjective outcomes. Results from analyses of combined study design characteristics were also broadly consistent with those from the hierarchical bias models.

Influence of reported study design characteristics: multivariable analyses

Table 18 presents results from multivariable analyses of the influence of inadequate or unclear allocation concealment and lack of or unclear double blinding, based on the 169 informative meta-analyses (1456 trials) in which both characteristics were assessed. Results from models without interaction terms are displayed in Figure 5: estimated RORs were similar to, or modestly attenuated compared with, the univariable analyses presented in Table 15. Estimated effects on heterogeneity were also modestly attenuated. Results from the model including interaction terms are displayed in Figure 6. RORs for the interaction between inadequate or unclear allocation concealment and lack of double blinding were close to 1, with wide CIs. Therefore, these results are consistent with the effects of these two characteristics being multiplicative.

FIGURE 5.

Estimated RORs and effects on heterogeneity from multivariable analyses, according to type of outcome measure. Analyses based on trials with assessments of both allocation concealment and blinding. Upper panel, influence of inadequate or unclear versus adequate allocation concealment, adjusted for blinding; lower panel, influence of lack of double blinding or unclear blinding status versus double blinding, adjusted for allocation concealment.

FIGURE 6.

Estimated RORs and effects on heterogeneity from multivariable analyses including interaction terms, according to type of outcome measure. Analyses based on trials with assessments of both allocation concealment and blinding. Upper panel, influence of inadequate or unclear versus adequate allocation concealment, in double-blind trials; middle panel, influence of lack of double blinding or unclear blinding status versus double blinding, in trials with adequate allocation concealment; lower panel, interaction between allocation concealment and blinding.

Results from multivariable analyses of the influence of all three study design characteristics are presented in Table 19 and displayed in Figure 7. Estimated RORs for each study design characteristic were of similar magnitudes to those in the univariable analyses presented in Table 16. For inadequate or unclear sequence generation or allocation concealment, estimated increases in between-trial heterogeneity (quantified by κ) were smaller in multivariable analyses than in the corresponding univariable analyses (see Table 16). Estimates of between-meta-analysis variability in average bias changed little compared with univariable analyses.

FIGURE 7.

Estimated RORs and effects on heterogeneity from multivariable analyses including all three study design characteristics, according to type of outcome measure. Analyses based on trials with assessments of all three study design characteristics. Upper panel, influence of inadequate or unclear versus adequate sequence generation, adjusted for allocation concealment and blinding; middle panel, influence of inadequate or unclear versus adequate allocation concealment, adjusted for sequence generation and blinding; lower panel, influence of lack of double blinding or unclear blinding status versus double blinding, adjusted for sequence generation and allocation concealment.

Analyses according to type of intervention and clinical area.

Results from univariable analyses of the influence of the three study design characteristics according to clinical area are shown in Table 20. For pregnancy and childbirth – the clinical area contributing most meta-analyses to the combined data set – RORs were further from 1 than in the analyses of the whole data set displayed in Figure 3 and Table 14, whereas estimates of effects on heterogeneity were broadly consistent with analyses of the whole data set. For the other two clinical areas, RORs were attenuated towards 1. Only small numbers of meta-analyses contributed to estimation of κ, but estimated values of κ and φ were generally smaller for circulatory system meta-analyses than for the other two clinical areas.

Table 21 displays results of univariable analyses restricted to pharmacological and surgical interventions. The majority of meta-analyses included in the full data set addressed pharmacological interventions; it was therefore unsurprising that overall results restricted to such interventions were consistent with those from the full data set. For surgical interventions, the influence of inadequate or unclear sequence generation and allocation concealment was estimated from only six and nine meta-analyses respectively: CIs were too wide to allow substantive conclusions to be drawn.

Analyses of meta-analyses comparing two active interventions

For meta-analyses comparing two active interventions we estimated increases in between-trial (within-meta-analysis) heterogeneity in trials with, compared with trials without, a study design characteristic of interest (among meta-analyses containing at least two trials with and without the characteristic of interest). Based on eight meta-analyses containing 84 trials, the between-trial standard deviation (corresponding to κ in previous analyses) for trials with inadequate or unclear (compared with adequate) sequence generation was 0.35 (95% CrI 0.02 to 0.88), similar to the estimate for inadequate or unclear (compared with adequate) allocation concealment (0.36, 95% CrI 0.02 to 0.88, based on six meta-analyses containing 52 trials). The estimated increase in heterogeneity was somewhat lower for lack of or unclear (compared with adequate) double blinding (0.24 95% CrI 0.02 to 0.79, based on seven meta-analyses containing 58 trials).

Results after excluding meta-analyses that contributed to the study of Wood et al.16

Data from the three contributing meta-epidemiological studies by Schulz et al. ,11 Kjaergard et al. 15 and Egger et al. 9 were combined in a study previously reported by Wood et al. ,16 which used a meta-meta-analytic approach for statistical analyses. 8 A total of 123 meta-analyses (1066 trials) were contributed by other meta-epidemiological studies. Table 22 shows numbers of contributing meta-analyses and results from univariable analyses of the influence of the three study design characteristics, both overall and separately according to type of outcome measure. Compared with the full data set, estimated RORs tend to be closer to the null, except for the influence of inadequate or unclear (compared with adequate) sequence generation in meta-analyses with subjective outcomes (ROR 0.86, 95% CrI 0.75 to 0.97). However, effects on the between-trial heterogeneity are broadly consistent with those for the main analyses reported in Table 14, with between-trial heterogeneity increased in meta-analyses with subjective outcomes, for all study design characteristics.

Univariable three-category analyses

Table 23 presents results from analyses of all trials in which reported methodological characteristics were assessed in three categories (low, unclear or high risk of bias). The number of trials and meta-analyses contributing to each analysis are shown in Table 24. There were no consistent patterns when comparing RORs for unclear (compared with low) and high (compared with low) risk of bias, which provides some support for combining these effects in the main analyses. Consistent with the main analyses, estimated values of κ were greatest for meta-analyses with subjectively assessed outcomes, both for high and unclear (compared with low) risk of bias.

Downweighting potentially biased evidence in future meta-analyses

Table 25 presents implications of the results from the primary univariable analyses (see Table 14) for downweighting of potentially biased evidence in future meta-analyses, based on formulae from Welton et al. 26 Because estimated values of κ and φ were greatest for meta-analyses with subjectively assessed outcomes, the minimum variance of the estimated intervention effect for a trial at high or unclear risk of bias is greatest for such trials. Across all BRANDO trials with inadequate or unclear sequence generation, bias adjustment led to a median 10% (IQR 4% to 23%) increase in trial-level variance. Downweighting based on results specific to type of outcome measure has the greatest effect in trials with subjectively assessed outcomes [median 20% (IQR 8% to 39%) increase in variance]. Results were broadly similar for downweighting based on inadequate or unclear allocation concealment. The median increase in variance for trials with subjectively measured outcomes that were not double blind or unclearly blinded was 63% (IQR 22% to 138%).

Downweighting all trials with inadequate or unclear sequence generation led to a median 13% (IQR 5% to 32%) increase in the variance of the summary (meta-analytic) intervention effect estimate among informative meta-analyses in the BRANDO database. This is in contrast to a median increase of 217% (IQR 87% to 482%) that results from completely excluding such trials, because only 26% of trials were assessed to have adequate sequence generation. Bias adjustment for meta-analyses with subjectively assessed outcomes led to a median 31% (IQR 11% to 56%) increase in the variance of the summary intervention effect estimate, which was again small compared with complete exclusion of such trials. Results were broadly similar for the other study design characteristics, although differences between the effects of downweighting and excluding trials at high or unclear risk of bias were smaller for double blinding, because 56% of trials from informative meta-analyses were double blind. Even for subjectively assessed outcomes, excluding trials not assessed as double blind led to a greater loss of precision than retaining but downweighting them [median increase in variance 36% (IQR 8% to 72%) for downweighting compared with median 62% (IQR 19% to 140%) for excluding].

Background

Aim

To examine the importance of different types of bias and other sources of variability in estimates of the effect of health care interventions, in different settings and areas of medicine.

Objectives

Research methods

The project will involve the construction and analysis of two large databases to be used for empirical research on bias and other sources of variability in estimates of the effect of health care interventions. The first database will contain information on the characteristics and results of randomised controlled trials included in meta-analyses, and will be the responsibility of the research associate to be based in Bristol and directly supervised by Jonathan Sterne (see Workstream A below). The second database will contain information on the characteristics and results of randomised controlled trials and non-randomised studies that evaluated comparable interventions using comparable outcomes, and will be the responsibility of the research associate to be based in Oxford and directly supervised by Jon Deeks (see Workstream B below). Data analyses, using the methods described in the section on data analysis, will be the responsibility of the project statistician, supervised by Jonathan Sterne, Jon Deeks and Douglas Altman.

Although there appears to be symmetry between Workstreams A and B, in reality there are some important differences. In A we will be looking at effect modifiers (methodological and other) within the class of RCTs to explain variation among RCTs. In B we will examine the magnitude, extent of and reasons for variation between RCTs and NRS, and perhaps also within different types of NRS. Several good studies of effect modifiers in RCTs (the studies in Table 1) have already been published, and we propose to build on this work. In contrast, there is (to our knowledge) no published evidence for B. Although there are several studies comparing RCTs and NRS (Table 2), none looked in detail at the characteristics of the RCTs and NRS included in those reviews, and there is little information on the magnitude of effect modification comparing RCTs and NRS or between different designs of NRS. Given that Workstream B starts from a much lower knowledge base than Workstream A and also is more complex than Workstream A, the objectives for B are more limited than for A.

Statistical methods and data analysis

We first describe statistical methods relevant to the analysis of meta-epidemiological studies. We then state the broad aims of the statistical analyses. Predicted outputs from the analyses appear in a separate section.

Schedule of work

Outputs

People

The team of applicants comprises the leading investigators in the field. All have extensive experience of conducting randomised trials and meta-analyses. Together, the team has links with clinical researchers in many branches of medicine, and covers a breadth of expertise, including clinical medicine and epidemiology (Egger, Ioannidis, Gluud, Pildal), methodological and reporting issues in randomised trials research (Altman, Schulz, Moher, Egger, Ioannidis), Cochrane methodology and Cochrane databases (Gluud, Ioannidis, Jüni, Pildal) and the statistical issues involved in the analysis of meta-epidemiological studies (Sterne, Altman, Ioannidis, Deeks, Schulz). Several applicants (Moher, Altman, Schulz, Egger, Ioannidis) are members of the CONSORT Group,26,27 the QUOROM Group (Moher, Altman) and the STROBE Group (Egger, Sterne, Altman) and most are affiliated to the Cochrane Collaboration. Three of the applicants (Egger, Moher, Sterne) are co-convenors of the Cochrane Reporting Bias Methods Group and two (Altman, Deeks) are co-convenors of the Cochrane Statistical Methods Group.

Dissemination of research results

We will disseminate our findings in journal articles and at national and international meetings. The final project report will be made available to organisations in the UK and worldwide that fund trials and systematic reviews. In particular, we will meet with representatives of the NHS Health Technology Assessment (HTA) programme and the National Institute for Clinical Excellence (NICE) to discuss the implications of our findings for the research that they commission. We will prepare educational articles and materials, and will make particular efforts to disseminate these in specialist areas for which the research identifies problems with the conduct and reporting of trials. A number of the applicants and collaborators run regular workshops at meetings of the Cochrane Collaboration, and these, together with courses on randomised trials and systematic reviews run in Bristol and Oxford, will incorporate the results of the research as it becomes available. We will also discuss, with other colleagues involved in the Cochrane Collaboration, how the Cochrane Reviewers’ Handbook might be updated in the light of our results. Those applicants involved with the development and updating of the CONSORT statement on the reporting of randomised controlled trials will ensure that this is updated to take account of new empirical evidence on aspects of trial quality which emerges from the study. Perhaps the most powerful influence on the conduct and reporting of trials and systematic reviews is medical journals’ publication policies. Here, CONSORT has direct and indirect influence: direct through the membership of several editors of the most influential journals and indirect through endorsement from International Committee of Medical Journal Editors and Council of Science Editors. Similarly, the findings of this research will feed into updates of and extensions to the QUOROM and STROBE Statements.

References

Prioritisation Group

1. Director, NIHR HTA programme, Professor of Clinical Pharmacology, Department of Pharmacology and Therapeutics, University of Liverpool

2. Professor Imti Choonara, Professor in Child Health, Academic Division of Child Health, University of Nottingham

   Chair – Pharmaceuticals Panel

3. Dr Bob Coates, Consultant Advisor – Disease Prevention Panel

4. Dr Andrew Cook, Consultant Advisor – Intervention Procedures Panel

5. Dr Peter Davidson, Director of NETSCC, Health Technology Assessment

6. Dr Nick Hicks, Consultant Adviser – Diagnostic Technologies and Screening Panel, Consultant Advisor–Psychological and Community Therapies Panel

7. Ms Susan Hird, Consultant Advisor, External Devices and Physical Therapies Panel

8. Professor Sallie Lamb, Director, Warwick Clinical Trials Unit, Warwick Medical School, University of Warwick

   Chair – HTA Clinical Evaluation and Trials Board

9. Professor Jonathan Michaels, Professor of Vascular Surgery, Sheffield Vascular Institute, University of Sheffield

   Chair – Interventional Procedures Panel

10. Professor Ruairidh Milne, Director – External Relations

11. Dr John Pounsford, Consultant Physician, Directorate of Medical Services, North Bristol NHS Trust

    Chair – External Devices and Physical Therapies Panel

12. Dr Vaughan Thomas, Consultant Advisor – Pharmaceuticals Panel, Clinical

    Lead – Clinical Evaluation Trials Prioritisation Group

13. Professor Margaret Thorogood, Professor of Epidemiology, Health Sciences Research Institute, University of Warwick

    Chair – Disease Prevention Panel

14. Professor Lindsay Turnbull, Professor of Radiology, Centre for the MR Investigations, University of Hull

    Chair – Diagnostic Technologies and Screening Panel

15. Professor Scott Weich, Professor of Psychiatry, Health Sciences Research Institute, University of Warwick

    Chair – Psychological and Community Therapies Panel

16. Professor Hywel Williams, Director of Nottingham Clinical Trials Unit, Centre of Evidence-Based Dermatology, University of Nottingham

    Chair – HTA Commissioning Board

    Deputy HTA Programme Director

HTA Commissioning Board

1. Professor of Dermato-Epidemiology, Centre of Evidence-Based Dermatology, University of Nottingham

2. Professor of Bio-Statistics, Department of Public Health and Epidemiology, University of Birmingham

3. Professor of Clinical Pharmacology, Director, NIHR HTA programme, Department of Pharmacology and Therapeutics, University of Liverpool

1. Professor Zarko Alfirevic, Head of Department for Women’s and Children’s Health, Institute of Translational Medicine, University of Liverpool

2. Professor Judith Bliss, Director of ICR-Clinical Trials and Statistics Unit, The Institute of Cancer Research

3. Professor David Fitzmaurice, Professor of Primary Care Research, Department of Primary Care Clinical Sciences, University of Birmingham

4. Professor John W Gregory, Professor in Paediatric Endocrinology, Department of Child Health, Wales School of Medicine, Cardiff University

5. Professor Steve Halligan, Professor of Gastrointestinal Radiology, Department of Specialist Radiology, University College Hospital, London

6. Professor Angela Harden, Professor of Community and Family Health, Institute for Health and Human Development, University of East London

7. Dr Joanne Lord, Reader, Health Economics Research Group, Brunel University

8. Professor Stephen Morris, Professor of Health Economics, University College London, Research Department of Epidemiology and Public Health, University College London

9. Professor Dion Morton, Professor of Surgery, Academic Department of Surgery, University of Birmingham

10. Professor Gail Mountain, Professor of Health Services Research, Rehabilitation and Assistive Technologies Group, University of Sheffield

11. Professor Irwin Nazareth, Professor of Primary Care and Head of Department, Department of Primary Care and Population Sciences, University College London

12. Professor E Andrea Nelson, Professor of Wound Healing and Director of Research, School of Healthcare, University of Leeds

13. Professor John David Norrie, Director, Centre for Healthcare Randomised Trials, Health Services Research Unit, University of Aberdeen

14. Professor Barney Reeves, Professorial Research Fellow in Health Services Research, Department of Clinical Science, University of Bristol

15. Professor Peter Tyrer, Professor of Community Psychiatry, Centre for Mental Health, Imperial College London

16. Professor Martin Underwood, Professor of Primary Care Research, Warwick Medical School, University of Warwick

17. Professor Caroline Watkins, Professor of Stroke and Older People’s Care, Chair of UK Forum for Stroke Training, Stroke Practice Research Unit, University of Central Lancashire

18. Dr Duncan Young, Senior Clinical Lecturer and Consultant, Nuffield Department of Anaesthetics, University of Oxford

1. Dr Tom Foulks, Medical Research Council

2. Dr Kay Pattison, Senior NIHR Programme Manager, Department of Health

HTA Clinical Evaluation and Trials Board

1. Director, Warwick Clinical Trials Unit, Warwick Medical School, University of Warwick and Professor of Rehabilitation, Nuffield Department of Orthopaedic, Rheumatology and Musculoskeletal Sciences, University of Oxford

2. Professor of the Psychology of Health Care, Leeds Institute of Health Sciences, University of Leeds

3. Director, NIHR HTA programme, Professor of Clinical Pharmacology, University of Liverpool

1. Professor Keith Abrams, Professor of Medical Statistics, Department of Health Sciences, University of Leicester

2. Professor Martin Bland, Professor of Health Statistics, Department of Health Sciences, University of York

3. Professor Jane Blazeby, Professor of Surgery and Consultant Upper GI Surgeon, Department of Social Medicine, University of Bristol

4. Professor Julia M Brown, Director, Clinical Trials Research Unit, University of Leeds

5. Professor Alistair Burns, Professor of Old Age Psychiatry, Psychiatry Research Group, School of Community-Based Medicine, The University of Manchester & National Clinical Director for Dementia, Department of Health

6. Dr Jennifer Burr, Director, Centre for Healthcare Randomised trials (CHART), University of Aberdeen

7. Professor Linda Davies, Professor of Health Economics, Health Sciences Research Group, University of Manchester

8. Professor Simon Gilbody, Prof of Psych Medicine and Health Services Research, Department of Health Sciences, University of York

9. Professor Steven Goodacre, Professor and Consultant in Emergency Medicine, School of Health and Related Research, University of Sheffield

10. Professor Dyfrig Hughes, Professor of Pharmacoeconomics, Centre for Economics and Policy in Health, Institute of Medical and Social Care Research, Bangor University

11. Professor Paul Jones, Professor of Respiratory Medicine, Department of Cardiac and Vascular Science, St George‘s Hospital Medical School, University of London

12. Professor Khalid Khan, Professor of Women’s Health and Clinical Epidemiology, Barts and the London School of Medicine, Queen Mary, University of London

13. Professor Richard J McManus, Professor of Primary Care Cardiovascular Research, Primary Care Clinical Sciences Building, University of Birmingham

14. Professor Helen Rodgers, Professor of Stroke Care, Institute for Ageing and Health, Newcastle University

15. Professor Ken Stein, Professor of Public Health, Peninsula Technology Assessment Group, Peninsula College of Medicine and Dentistry, Universities of Exeter and Plymouth

16. Professor Jonathan Sterne, Professor of Medical Statistics and Epidemiology, Department of Social Medicine, University of Bristol

17. Mr Andy Vail, Senior Lecturer, Health Sciences Research Group, University of Manchester

18. Professor Clare Wilkinson, Professor of General Practice and Director of Research North Wales Clinical School, Department of Primary Care and Public Health, Cardiff University

19. Dr Ian B Wilkinson, Senior Lecturer and Honorary Consultant, Clinical Pharmacology Unit, Department of Medicine, University of Cambridge

1. Ms Kate Law, Director of Clinical Trials, Cancer Research UK

2. Dr Morven Roberts, Clinical Trials Manager, Health Services and Public Health Services Board, Medical Research Council

Diagnostic Technologies and Screening Panel

1. Scientific Director of the Centre for Magnetic Resonance Investigations and YCR Professor of Radiology, Hull Royal Infirmary

2. Professor Judith E Adams, Consultant Radiologist, Manchester Royal Infirmary, Central Manchester & Manchester Children’s University Hospitals NHS Trust, and Professor of Diagnostic Radiology, University of Manchester

3. Mr Angus S Arunkalaivanan, Honorary Senior Lecturer, University of Birmingham and Consultant Urogynaecologist and Obstetrician, City Hospital, Birmingham

4. Dr Diana Baralle, Consultant and Senior Lecturer in Clinical Genetics, University of Southampton

5. Dr Stephanie Dancer, Consultant Microbiologist, Hairmyres Hospital, East Kilbride

6. Dr Diane Eccles, Professor of Cancer Genetics, Wessex Clinical Genetics Service, Princess Anne Hospital

7. Dr Trevor Friedman, Consultant Liason Psychiatrist, Brandon Unit, Leicester General Hospital

8. Dr Ron Gray, Consultant, National Perinatal Epidemiology Unit, Institute of Health Sciences, University of Oxford

9. Professor Paul D Griffiths, Professor of Radiology, Academic Unit of Radiology, University of Sheffield

10. Mr Martin Hooper, Public contributor

11. Professor Anthony Robert Kendrick, Associate Dean for Clinical Research and Professor of Primary Medical Care, University of Southampton

12. Dr Nicola Lennard, Senior Medical Officer, MHRA

13. Dr Anne Mackie, Director of Programmes, UK National Screening Committee, London

14. Mr David Mathew, Public contributor

15. Dr Michael Millar, Consultant Senior Lecturer in Microbiology, Department of Pathology & Microbiology, Barts and The London NHS Trust, Royal London Hospital

16. Mrs Una Rennard, Public contributor

17. Dr Stuart Smellie, Consultant in Clinical Pathology, Bishop Auckland General Hospital

18. Ms Jane Smith, Consultant Ultrasound Practitioner, Leeds Teaching Hospital NHS Trust, Leeds

19. Dr Allison Streetly, Programme Director, NHS Sickle Cell and Thalassaemia Screening Programme, King’s College School of Medicine

20. Dr Matthew Thompson, Senior Clinical Scientist and GP, Department of Primary Health Care, University of Oxford

21. Dr Alan J Williams, Consultant Physician, General and Respiratory Medicine, The Royal Bournemouth Hospital

1. Dr Tim Elliott, Team Leader, Cancer Screening, Department of Health

2. Dr Joanna Jenkinson, Board Secretary, Neurosciences and Mental Health Board (NMHB), Medical Research Council

3. Professor Julietta Patrick, Director, NHS Cancer Screening Programme, Sheffield

4. Dr Kay Pattison, Senior NIHR Programme Manager, Department of Health

5. Professor Tom Walley, CBE, Director, NIHR HTA programme, Professor of Clinical Pharmacology, University of Liverpool

6. Dr Ursula Wells, Principal Research Officer, Policy Research Programme, Department of Health

Disease Prevention Panel

1. Professor of Epidemiology, University of Warwick Medical School, Coventry

2. Dr Robert Cook, Clinical Programmes Director, Bazian Ltd, London

3. Dr Colin Greaves, Senior Research Fellow, Peninsula Medical School (Primary Care)

4. Mr Michael Head, Public contributor

5. Professor Cathy Jackson, Professor of Primary Care Medicine, Bute Medical School, University of St Andrews

6. Dr Russell Jago, Senior Lecturer in Exercise, Nutrition and Health, Centre for Sport, Exercise and Health, University of Bristol

7. Dr Julie Mytton, Consultant in Child Public Health, NHS Bristol

8. Professor Irwin Nazareth, Professor of Primary Care and Director, Department of Primary Care and Population Sciences, University College London

9. Dr Richard Richards, Assistant Director of Public Health, Derbyshire County Primary Care Trust

10. Professor Ian Roberts, Professor of Epidemiology and Public Health, London School of Hygiene & Tropical Medicine

11. Dr Kenneth Robertson, Consultant Paediatrician, Royal Hospital for Sick Children, Glasgow

12. Dr Catherine Swann, Associate Director, Centre for Public Health Excellence, NICE

13. Mrs Jean Thurston, Public contributor

14. Professor David Weller, Head, School of Clinical Science and Community Health, University of Edinburgh

1. Ms Christine McGuire, Research & Development, Department of Health

2. Dr Kay Pattison, Senior NIHR Programme Manager, Department of Health

3. Professor Tom Walley, CBE, Director, NIHR HTA programme, Professor of Clinical Pharmacology, University of Liverpool

External Devices and Physical Therapies Panel

1. Consultant Physician North Bristol NHS Trust

2. Reader in Wound Healing and Director of Research, University of Leeds

3. Professor Bipin Bhakta, Charterhouse Professor in Rehabilitation Medicine, University of Leeds

4. Mrs Penny Calder, Public contributor

5. Dr Dawn Carnes, Senior Research Fellow, Barts and the London School of Medicine and Dentistry

6. Dr Emma Clark, Clinician Scientist Fellow & Cons. Rheumatologist, University of Bristol

7. Mrs Anthea De Barton-Watson, Public contributor

8. Professor Nadine Foster, Professor of Musculoskeletal Health in Primary Care Arthritis Research, Keele University

9. Dr Shaheen Hamdy, Clinical Senior Lecturer and Consultant Physician, University of Manchester

10. Professor Christine Norton, Professor of Clinical Nursing Innovation, Bucks New University and Imperial College Healthcare NHS Trust

11. Dr Lorraine Pinnigton, Associate Professor in Rehabilitation, University of Nottingham

12. Dr Kate Radford, Senior Lecturer (Research), University of Central Lancashire

13. Mr Jim Reece, Public contributor

14. Professor Maria Stokes, Professor of Neuromusculoskeletal Rehabilitation, University of Southampton

15. Dr Pippa Tyrrell, Senior Lecturer/Consultant, Salford Royal Foundation Hospitals’ Trust and University of Manchester

16. Dr Nefyn Williams, Clinical Senior Lecturer, Cardiff University

1. Dr Kay Pattison, Senior NIHR Programme Manager, Department of Health

2. Dr Morven Roberts, Clinical Trials Manager, Health Services and Public Health Services Board, Medical Research Council

3. Professor Tom Walley, CBE, Director, NIHR HTA programme, Professor of Clinical Pharmacology, University of Liverpool

4. Dr Ursula Wells, Principal Research Officer, Policy Research Programme, Department of Health

Interventional Procedures Panel

1. Professor of Vascular Surgery, University of Sheffield

2. Consultant Colorectal Surgeon, Bristol Royal Infirmary

3. Mrs Isabel Boyer, Public contributor

4. Mr Sankaran Chandra Sekharan, Consultant Surgeon, Breast Surgery, Colchester Hospital University NHS Foundation Trust

5. Professor Nicholas Clarke, Consultant Orthopaedic Surgeon, Southampton University Hospitals NHS Trust

6. Ms Leonie Cooke, Public contributor

7. Mr Seumas Eckford, Consultant in Obstetrics & Gynaecology, North Devon District Hospital

8. Professor Sam Eljamel, Consultant Neurosurgeon, Ninewells Hospital and Medical School, Dundee

9. Dr Adele Fielding, Senior Lecturer and Honorary Consultant in Haematology, University College London Medical School

10. Dr Matthew Hatton, Consultant in Clinical Oncology, Sheffield Teaching Hospital Foundation Trust

11. Dr John Holden, General Practitioner, Garswood Surgery, Wigan

12. Dr Fiona Lecky, Senior Lecturer/Honorary Consultant in Emergency Medicine, University of Manchester/Salford Royal Hospitals NHS Foundation Trust

13. Dr Nadim Malik, Consultant Cardiologist/Honorary Lecturer, University of Manchester

14. Mr Hisham Mehanna, Consultant & Honorary Associate Professor, University Hospitals Coventry & Warwickshire NHS Trust

15. Dr Jane Montgomery, Consultant in Anaesthetics and Critical Care, South Devon Healthcare NHS Foundation Trust

16. Professor Jon Moss, Consultant Interventional Radiologist, North Glasgow Hospitals University NHS Trust

17. Dr Simon Padley, Consultant Radiologist, Chelsea & Westminster Hospital

18. Dr Ashish Paul, Medical Director, Bedfordshire PCT

19. Dr Sarah Purdy, Consultant Senior Lecturer, University of Bristol

20. Dr Matthew Wilson, Consultant Anaesthetist, Sheffield Teaching Hospitals NHS Foundation Trust

21. Professor Yit Chiun Yang, Consultant Ophthalmologist, Royal Wolverhampton Hospitals NHS Trust

1. Dr Kay Pattison, Senior NIHR Programme Manager, Department of Health

2. Dr Morven Roberts, Clinical Trials Manager, Health Services and Public Health Services Board, Medical Research Council

3. Professor Tom Walley, CBE, Director, NIHR HTA programme, Professor of Clinical Pharmacology, University of Liverpool

4. Dr Ursula Wells, Principal Research Officer, Policy Research Programme, Department of Health

Pharmaceuticals Panel

1. Professor in Child Health, University of Nottingham

2. Senior Lecturer in Clinical Pharmacology, University of East Anglia

3. Dr Martin Ashton-Key, Medical Advisor, National Commissioning Group, NHS London

4. Dr Peter Elton, Director of Public Health, Bury Primary Care Trust

5. Dr Ben Goldacre, Research Fellow, Division of Psychological Medicine and Psychiatry, King’s College London

6. Dr James Gray, Consultant Microbiologist, Department of Microbiology, Birmingham Children’s Hospital NHS Foundation Trust

7. Dr Jurjees Hasan, Consultant in Medical Oncology, The Christie, Manchester

8. Dr Carl Heneghan, Deputy Director Centre for Evidence-Based Medicine and Clinical Lecturer, Department of Primary Health Care, University of Oxford

9. Dr Dyfrig Hughes, Reader in Pharmacoeconomics and Deputy Director, Centre for Economics and Policy in Health, IMSCaR, Bangor University

10. Dr Maria Kouimtzi, Pharmacy and Informatics Director, Global Clinical Solutions, Wiley-Blackwell

11. Professor Femi Oyebode, Consultant Psychiatrist and Head of Department, University of Birmingham

12. Dr Andrew Prentice, Senior Lecturer and Consultant Obstetrician and Gynaecologist, The Rosie Hospital, University of Cambridge

13. Ms Amanda Roberts, Public contributor

14. Dr Gillian Shepherd, Director, Health and Clinical Excellence, Merck Serono Ltd

15. Mrs Katrina Simister, Assistant Director New Medicines, National Prescribing Centre, Liverpool

16. Professor Donald Singer, Professor of Clinical Pharmacology and Therapeutics, Clinical Sciences Research Institute, CSB, University of Warwick Medical School

17. Mr David Symes, Public contributor

18. Dr Arnold Zermansky, General Practitioner, Senior Research Fellow, Pharmacy Practice and Medicines Management Group, Leeds University

1. Dr Kay Pattison, Senior NIHR Programme Manager, Department of Health

2. Mr Simon Reeve, Head of Clinical and Cost-Effectiveness, Medicines, Pharmacy and Industry Group, Department of Health

3. Dr Heike Weber, Programme Manager, Medical Research Council

4. Professor Tom Walley, CBE, Director, NIHR HTA programme, Professor of Clinical Pharmacology, University of Liverpool

5. Dr Ursula Wells, Principal Research Officer, Policy Research Programme, Department of Health

Psychological and Community Therapies Panel

1. Professor of Psychiatry, University of Warwick, Coventry

2. Consultant & University Lecturer in Psychiatry, University of Cambridge

3. Professor Jane Barlow, Professor of Public Health in the Early Years, Health Sciences Research Institute, Warwick Medical School

4. Dr Sabyasachi Bhaumik, Consultant Psychiatrist, Leicestershire Partnership NHS Trust

5. Mrs Val Carlill, Public contributor

6. Dr Steve Cunningham, Consultant Respiratory Paediatrician, Lothian Health Board

7. Dr Anne Hesketh, Senior Clinical Lecturer in Speech and Language Therapy, University of Manchester

8. Dr Peter Langdon, Senior Clinical Lecturer, School of Medicine, Health Policy and Practice, University of East Anglia

9. Dr Yann Lefeuvre, GP Partner, Burrage Road Surgery, London

10. Dr Jeremy J Murphy, Consultant Physician and Cardiologist, County Durham and Darlington Foundation Trust

11. Dr Richard Neal, Clinical Senior Lecturer in General Practice, Cardiff University

12. Mr John Needham, Public contributor

13. Ms Mary Nettle, Mental Health User Consultant

14. Professor John Potter, Professor of Ageing and Stroke Medicine, University of East Anglia

15. Dr Greta Rait, Senior Clinical Lecturer and General Practitioner, University College London

16. Dr Paul Ramchandani, Senior Research Fellow/Cons. Child Psychiatrist, University of Oxford

17. Dr Karen Roberts, Nurse/Consultant, Dunston Hill Hospital, Tyne and Wear

18. Dr Karim Saad, Consultant in Old Age Psychiatry, Coventry and Warwickshire Partnership Trust

19. Dr Lesley Stockton, Lecturer, School of Health Sciences, University of Liverpool

20. Dr Simon Wright, GP Partner, Walkden Medical Centre, Manchester

1. Dr Kay Pattison, Senior NIHR Programme Manager, Department of Health

2. Dr Morven Roberts, Clinical Trials Manager, Health Services and Public Health Services Board, Medical Research Council

3. Professor Tom Walley, CBE, Director, NIHR HTA programme, Professor of Clinical Pharmacology, University of Liverpool

4. Dr Ursula Wells, Principal Research Officer, Policy Research Programme, Department of Health